Skip to content

Fix in-place gem rebuild after a one-shot build script (#2106) - #2161

Open
Mikola Lysenko (mikolalysenko) wants to merge 4 commits into
mainfrom
agent/fix-gem-in-place-consumed-build-files
Open

Mikola Lysenko (mikolalysenko) wants to merge 4 commits into
mainfrom
agent/fix-gem-in-place-consumed-build-files

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-fable-5-2

Fixes #2106

Summary

Agent-mode apply could never rebuild a gem whose build script uses up a file the gem ships. argon2 is the reported case: its extconf.rb renames Makefile.real to Makefile, so the installed gem has no Makefile.real, and the rebuild in the installed tree failed in the script with native_extension_build_failed.

apply now puts the files the gem's first build used up back for that rebuild, from the source gem it has already unpacked and proven to be the installed gem's own source. A failed build leaves the gem exactly as installed, and a finished one leaves nothing in the tree the build did not make. Nothing changes for a gem whose installed tree is complete, or when no source gem is at hand.

Root cause

A gem that loads its library from its own tree (an FFI gem) cannot take a build staged elsewhere, so it is rebuilt where it is installed (rebuild in crates/socket-patch-core/src/patch/gem_ext.rs). That build assumed the installed tree still held everything the build scripts read. The tree is what the install-time build left, and a build may rename or delete what the gem shipped.

The change

  • build_in_stage reads the files the pristine source gem ships and the installed copy does not hold (used_up_files), before the staged build uses them up too. At most 64 MiB, all or none, and only for a copy RubyGems built.
  • StageFailure::InTree carries them to rebuild with the list of products it already carried (SeenInStage).
  • rebuild writes them into the gem's tree after the tree snapshot is taken (give_back), so the existing undo of a failed or interrupted build removes them. It never writes over a file the tree holds, a file of the patch record, or through a symlink in the gem dir. Each file keeps the time the staged copy was written, which is older than the patched sources, so a build that derives one file from another does not take a shipped file for current.
  • After the build, the files it left untouched are removed again (take_away), with the directories made for them. rollback therefore still gives the installed tree back byte for byte.
  • A rollback that has lost its backup rebuilds through the same path.
  • docs/ecosystems.md ("Ruby gems: native extensions") names argon2 again and describes this.

Evidence

Real gem, Ruby 3.3.6, Bundler 4.0.18, the repro from the issue:

Step origin/main this branch
socket-patch apply on argon2 2.3.0 native_extension_build_failed (extconf.rb: Makefile.real missing) 1 of 1 applied
the patched symbol in lib/libargon2_wrap.so absent present, Argon2::Password works
apply --check not reached in sync
rollback, then diff -r against the fresh install not reached no difference

Tests:

  • a_build_script_that_uses_up_a_shipped_file_is_given_it_back (e2e_gem_native_ext_build) covers Fixes #2106: a gem built with the real toolchain whose script renames a shipped file. It fails on origin/main with the issue's error (first commit of the branch) and passes with the fix. It asserts that only the patched source and the library changed, that apply --check and vex accept the build, and that rollback restores the tree with and without the backup.
  • a_failed_build_takes_the_shipped_file_it_was_given_away_again: a build that fails in the installed tree after using the file up leaves the gem exactly as installed.
  • Three unit tests in gem_ext.rs cover reading the files, giving them back and taking them away, and the paths that are never written (existing file, record file, unsafe path, path under a symlink).

Local runs: cargo clippy --locked --workspace --all-features -- -D warnings clean; socket-patch-core patch:: unit tests pass (1364); e2e_gem_native_ext_build (41), e2e_redirect_gem_build (49), e2e_vendor_gem_build (18) and in_process_gem_apply (3) pass with --include-ignored. e2e_gem needs the live patch API, which this sandbox cannot reach; CI runs it.

CI on the head commit: ci-ok and clippy are green, and Bugbot found no issues. The nokogiri compatibility run was cancelled at creation (the push and the ready-for-review event raced) and was re-run. Its Ruby 2.3 / nokogiri 1.6.8.1 leg then failed once in bundle install, before socket-patch ran (Bundler 1.17 resolved a minitest release that needs Ruby 3.1), and passed on its one re-run.

Review

/code-review high on the diff gave six findings. Fixed: the failed-build test never had a file given back; given-back files were newer than the patched sources; a file was read before its size was checked; a write failure while giving files back failed the rebuild with a misleading message (the rebuild now runs over the tree as it is, as before this change).

Not fixed here:

  • The used-up files are only carried out of a staged build that ends because the gem builds into its own tree. A staged build that fails for another reason (a build dependency that cannot be staged, a build error) still rebuilds in place without them, as on main. That is the same behavior as before, not a regression, and widening it touches every StageFailure variant; it is better as its own change if a real gem needs it.
  • A run killed during the in-place build can leave empty directories that were made for given-back files. The next run's recovery removes the files; like the directories a build itself makes, empty ones are not removed. No file of the gem differs.

https://claude.ai/code/session_011f9aeRso7kGGWRcMwXQpkz


Generated by Claude Code

Assisted-by: Claude Code:claude-fable-5-2

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aeRso7kGGWRcMwXQpkz
A build script can use up a file the gem ships: argon2's extconf.rb
renames Makefile.real to Makefile. The installed tree then lacks the
file, and the in-place rebuild of agent mode fails in the script.

The first test fails on main (the apply ends in
native_extension_build_failed). The second guards the fix: a build
that fails leaves the gem exactly as installed.

Refs #2106

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aeRso7kGGWRcMwXQpkz
Assisted-by: Claude Code:claude-fable-5-2
A gem that loads its library from its own tree is rebuilt in agent
mode where it is installed. That build ran over the tree as the
install left it, and a build script may use up a file the gem ships:
argon2's extconf.rb renames Makefile.real to Makefile. Run again, the
script fails, so every agent-mode patch to an argon2 build input
ended in native_extension_build_failed.

The staged build of the source gem already says the gem builds into
its own tree. It now also keeps the files the source gem ships that
the installed copy no longer holds, read before the staged build uses
them up as well. The in-place build gets them back after the tree
snapshot is taken, so the undo of a failed build removes them. After
a build that finishes, those it left untouched are removed again: the
tree then differs from the installed one by what the build made, and
rollback gives the installed tree back byte for byte.

A rollback that has lost its backup rebuilds the same way.

Fixes #2106

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aeRso7kGGWRcMwXQpkz
Assisted-by: Claude Code:claude-fable-5-2
Review fixes for the in-place build of a gem with a one-shot build
script.

A file given back kept the time it was written, which is after the
patched sources were. A build that makes one from the other (a parser
the gem ships generated) would take the shipped file for current. It
now keeps the time the stage's copy was written, which is before the
record's files are written anywhere.

The size of a file is checked against the limit before it is read.

Files that cannot be given back (a full disk) no longer fail the
rebuild: it runs over the tree as it is, as it did before.

The end-to-end test of a failed build broke the patched source, which
fails the staged build first, so nothing was ever given back in it.
Its build now fails only in the installed tree, after it used the
shipped file up.

Refs #2106

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aeRso7kGGWRcMwXQpkz
Assisted-by: Claude Code:claude-fable-5-2
@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review November 19, 2026 14:11
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor

cursor Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes gem native rebuild preparation and filesystem undo semantics on failure; limited to in-tree FFI-style gems with source gem available, but touches critical apply/rollback paths.

Overview
Fixes in-place native extension rebuilds when the gem’s install-time build consumes shipped files (e.g. argon2’s extconf.rb renaming Makefile.real to Makefile), which left the installed tree unable to run the script again and caused native_extension_build_failed.

rebuild now learns from a staged source-gem build which files the copy is missing (used_up_files) and which in-tree artifacts a fresh build would recreate. Before the in-place build it writes those shipped files back from the pristine source (give_back), after snapshotting so failure undo removes them; on success or failure it strips untouched “given back” files (take_away) so the tree matches a normal install. StageFailure::InTree carries this as SeenInStage (created products + used-up files) instead of only a stale-file list. Rollback paths without a backup use the same logic.

Tests: e2e sp_wrap fixtures mimicking argon2’s one-shot rename (success + failed build leaving the tree unchanged); unit tests for read/give-back/take-away and safe write rules. docs/ecosystems.md documents argon2-style scripts and this behavior.

Reviewed by Cursor Bugbot for commit 48c82f8. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 48c82f8. Configure here.

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] PR shepherd: review brief for the approver

What it does. Fixes #2106. An agent-mode apply rebuilds an FFI gem such as argon2 in its installed tree, and that rebuild failed because the gem's first build had used up a file it ships: argon2's extconf.rb renames Makefile.real. The rebuild now gives those files back from the gem's own verified source, then takes away again whatever the build didn't touch, so a failed build or a rollback still leaves the tree exactly as installed.

Risk: low. It only writes paths that don't exist yet, inside the gem dir, never through a symlink or junction, and after the tree snapshot, so the existing undo removes them on a failed build. A gem whose installed tree is complete takes the same path as before.

Look here

  • gem_ext.rs:3325, :3387, :3438: rebuild gives the files back after the snapshot, and takes them away on success and after restore_tree on failure
  • gem_ext.rs:3736: used_up_files, with the 64 MiB cap checked before any read, all or none
  • gem_ext.rs:3785-3868: give_back / write_used_up, which skip an existing file, a record path or a linked directory, and undo a partial write
  • gem_ext.rs:4321: build_in_stage reads the files after prove_own_source and before the patch touches the stage
  • [e2e_gem_native_ext_build.rs:3878](https://github.com/SocketDev/socket-patch/blob/48c82f84c1a001cde065d39a0c516cc40ce37119/crates/socket-patch-cli/tests/e2e_gem_native_ext_build.rs#L3878), [:3951](https://github.com/SocketDev/socket-patch/blob/48c82f84c1a001cde065d39a0c516cc40ce37119/crates/socket-patch-cli/tests/e2e_gem_native_ext_build.rs#L3951): the two new real-toolchain tests

Verified. I reviewed the whole diff for path safety, partial failures, ordering, Windows (bytes are copied raw) and debug leftovers. CHANGELOG.md is untouched and the description matches the code.

  • cargo test -p socket-patch-core --lib gem_ext: 66 passed.
  • The two new e2e tests pass with real Ruby 3.3.6 and cc.
  • With gem_ext.rs reverted to main, both e2e tests fail, so they catch the bug.
  • CI on 48c82f8: ci-ok is green, clippy was skipped because the PR was still a draft when that run started (the merge queue runs it again), and Bugbot found no issues.

Changes I made: none.

Open questions: none that block a merge. Two nits for a follow-up:

  • A run killed during the in-place build can leave empty directories that were made for given-back files. No file differs, and the description lists this.
  • The comment on ShippedFile::modified says the stage copy predates any record write. That doesn't hold on the rollback path, which is harmless there.

Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

2 participants