Repository navigation
Vendored mode: vendor gems locked as precompiled platform gems (nokogiri), on MRI and JRuby - #2154
Mikola Lysenko (mikolalysenko) wants to merge 10 commits into
Conversation
The patch service is gaining patched platform gems ("prebuilds") for gem
patches. A package reference for a gem patch then lists, beside the ruby
gem's `tarball` and stub, one `gem-platform` artifact and one
`gem-platform-stub-gemspec` per platform it built, each naming its
`platform` (`x86_64-linux-gnu`, `arm64-darwin`: the token Gemfile.lock
spells).
`PackageVendorArtifact` reads the new optional `platform`. A patch with
no prebuild serves neither kind, and nothing changes for it.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Bundler installs a native gem such as nokogiri as precompiled platform gems: Gemfile.lock holds one entry per platform (`nokogiri (1.18.10-x86_64-linux-gnu)`, `nokogiri (1.18.10-arm64-darwin)`) and the installed copy is `gems/nokogiri-1.18.10-arm64-darwin`. Vendored mode refused every such gem with `platform_gem_unsupported`: Bundler never builds a `path:` gem's extensions, so the gem's own tree cannot be vendored, and one platform's build would break every other platform. Such a gem is now vendored from the patch service's prebuilds, one per platform the lock pins. Verified against real Bundler 2.3.27, 2.5.23, 2.7.2 and 4.0.15: a `path:` directory holding one unpacked platform gem per platform, each in its own subdirectory with a gemspec that sets its platform, is locked as one `PATH` section with a spec per platform, and Ruby loads the tree of the platform it runs on with no build. - The lock decides. Platform-suffixed entries of the gem at the patched version select the new path, whether the gem is installed as a platform gem, as a ruby gem, or not at all. A plain entry alone keeps today's behaviour, including the refusal of a platform-gem install. - The copy is `.socket/vendor/gem/<uuid>/<name>-<version>/<platform>/` per platform: the prebuild's files and the stub gemspec the service serves for it (the client now carries each served artifact's platform to the backends). Each prebuild and stub is verified against its served sha512 like the ruby gem and its stub, and a patched file the prebuild ships must be the patched one. A patched file it does not ship (the source of a compiled extension) is accepted: the binary is the service's build. - The Gemfile is wired as before. In the lock every platform block moves, verbatim and in lock order, into one `PATH` section; `DEPENDENCIES` is pinned with `!`; every `CHECKSUMS` line of the gem becomes bare. - The ledger entry records the platforms (`gemPlatforms`), every moved block, and the registry `CHECKSUMS` line of each platform entry it made bare (an array per record, paired by entry on revert), so revert restores the Gemfile and the lock byte for byte, also after a patch update moved the wiring to a new uuid. - `vendor --check`, `vex` and `repair` verify every platform's tree: the patched files each ships and, through the file inventory, every other byte. An entry read off a lock without the ledger finds the trees in the vendored directory. `repair` restores a damaged copy from the prebuilds of the recorded platforms. - A second run is a no-op that asks the service nothing; a copy that lost a platform's tree is rebuilt in full. - The trees hold compiled extensions (`*.so`, `*.bundle`, a JRuby gem's `*.jar`), which stock ignore templates name. The uuid dir gets a `.gitignore` of `!*`, as the NuGet and npm backends write, and a tree git would still not commit in full is refused with `vendor_artifact_gitignored` before either project file is written: a gem wired without its binary installs everywhere and fails to load. - Lockfile discovery reads the platform specs of a vendored `PATH` section as one vendored reference of the gem. Refused with `platform_gem_unsupported`, before anything is written: a locked platform the service lists no prebuild or stub for (the message names the platforms that have none and the ones that do), and a lock that also holds the plain `name (version)` entry, which is the ruby platform and has no prebuilt tree (decided from the lock, before the service is asked and before a hosted pin of the gem is taken over; the message names the entry and `bundle lock --remove-platform ruby`). A native gem locked only for the ruby platform is refused as before. The ruby gem that comes with the package reference is still downloaded and verified on this path, and not used. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The capstone builds, with RubyGems, a local gem source that publishes a fixture gem as platform gems only (this machine's platform and two foreign ones; a pure-Ruby gem whose gemspec sets `platform` is a platform gem) and the artifacts a patch service serves for its patch. Real Bundler locks the gem for the three platforms. Then, for a project that declares the gem and for one that only reaches it through another gem: - a service without one platform's prebuild refuses the gem by that platform's name, with nothing written; - `vendor` unpacks a tree per platform and moves every block into one PATH section; `vex` attests it, with the manifest, without it, and from the lock and the patch API alone; `vendor --check` fails on an unpatched tree of a foreign platform; - a fresh checkout of the committable files installs frozen, with no request to the patch service and a byte-stable lock, and Ruby loads the patched file from the tree of the running platform; an unfrozen install and `bundle lock` regenerate the same lock; - a second run changes nothing and downloads nothing; - `vendor --revert` restores the Gemfile and the lock byte for byte. It runs on Bundler 2.3 and later, the versions the layout was checked against. CLI_CONTRACT.md and docs/ecosystems.md describe the new behaviour where they described the refusal, including how to add a platform to the lock of a gem that is already vendored. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ittable Fixes from the code review of the vendored platform-gem support. A locked platform with no prebuild yet - The patch service lists a prebuild only once it is built, so "not built yet" and "never" look the same to the client. That was the failure `platform_gem_unsupported` (exit 1). It is now the skip `vendor_prebuilt_platform_pending`: the run does not fail, nothing is downloaded or written, and the detail names the gem, the platforms with no prebuild and the ones that have one. A patch that is already vendored stays in force and the skip names both uuids, like the other unserved codes; unlike them it is a skip with nothing to keep too. A hosted to vendored takeover previews the same skip in a dry run. A copy the lock is ALREADY wired to that is missing or damaged and cannot be rebuilt stays a failure (`vendor_prebuilt_required`), in a re-run and in `repair`. - Which prebuilds exist is now read from the package reference alone (`ApiClient::fetch_vendor_reference`, `fetch_served_artifacts`). The ruby gem that reference also names was downloaded and hashed only to read that list, and is no longer downloaded, so no archive download is planned for such a gem either. The prebuilds are fetched a few platforms at a time and unpacked in lock order. What git commits - The uuid dir gets its `.gitignore` and `.gitattributes` from the writer the npm and PyPI backends share, instead of a `.gitignore` of its own: without the attributes a project rule such as `*.so filter=lfs` committed a prebuilt binary as an LFS pointer. - `repair`'s restore wrote that metadata into its temporary stage and moved only the copy into place, so a restored copy had none. It is written in the project's uuid dir after the move, and an in-sync re-run puts back metadata that went missing. - A rule that ignores the uuid dir itself is refused (`vendor_artifact_gitignored`) before anything is downloaded, in a dry run as in a wet one. It was found only after every prebuild had been downloaded, and never by a dry run. - A `.gitignore` a prebuild ships inside its own tree wins over the uuid dir's `!*` for the files below it, and the project had no rule to remove. Each one is emptied while the tree is staged (`vendor_gem_gitignore_neutralized`), as the Composer backend does for a copy's filters, so the ledger's inventory and a restore see the same bytes. One the patch itself rewrites keeps the patch's bytes. One rule for a prebuild tree - The download gate, the in-sync test of a re-run and `vendor --check` / `vex` / `repair` judged a tree with two implementations that disagreed about a missing patched file. All three now call `verify::verify_platform_tree`: a patched file the tree holds must be the patched one; one it lacks is accepted (a platform gem need not ship the sources of its extension) unless the ledger's inventory lists it, which means it was vendored and then lost; and a tree that holds none of the patched files never passes. A re-run reads the ledger's inventory for this, so a copy that lost a patched file is rebuilt instead of reported in sync. For the check this accepts an unshipped file of an entry with no inventory, where it used to fail, and refuses a prebuild that ships none of the patch's files. The lock - The layout is wired only when the lock's `BUNDLED WITH` names Bundler 2.3 or later, the oldest the real-Bundler tests run it on. An older or unnamed one is refused `platform_gem_unsupported`, by the version needed, from the lock alone. - The refusals decided from the lock alone read and parse `Gemfile.lock` once for every purl (`gem_lock_refusals`), not once per purl. - Ledger recovery reads the checksum record of a gem vendored as platform gems, an array of lines, as well as the single line of a plain gem. Tests: the unit tests of the platform-gem module cover each change, and the real-Bundler capstone now runs the skip (with and without a vendored patch to keep) and checks that only the prebuilds are downloaded. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
CLI_CONTRACT.md and docs/ecosystems.md describe what changed for a gem vendored as prebuilt platform gems: - `vendor_prebuilt_platform_pending`, the skip for a locked platform the patch service has no prebuild for yet, in place of the `platform_gem_unsupported` failure; the one case that stays a failure; - the prebuilds are read from the package reference, and the ruby gem is not downloaded; - the uuid dir's `.gitignore` and `.gitattributes`, which a re-run and `repair` put back; the `.gitignore` files of a prebuild that are emptied (`vendor_gem_gitignore_neutralized`); and `vendor_artifact_gitignored` before any download, dry run included; - the one rule every check of a prebuild tree applies; - the refusal of a lock written by a Bundler older than 2.3. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The usual nokogiri patch adds a file under `patches/` or changes `dependencies.yml`. A precompiled platform gem ships neither: the patch lives entirely in the binaries the patch service rebuilt. The shared rule for a prebuild tree refused a tree that holds none of the patched files, so such a patch could not be vendored at all. `verify::verify_platform_tree` now takes what vouches for the tree (`TreeVoucher`) and judges it like this, at the download gate, in a re-run's in-sync test and in `vendor --check` / `vex` / `repair`: - a patched file the tree holds must hash to its `afterHash`, as before; - a patched file the tree does not hold is no mismatch, also when it holds none of them; - while the tree is staged, the integrity check of the prebuild against the digest the service states for that platform vouches for it. That check is mandatory: an artifact with no digest is not listed, and no unverified archive is unpacked; - once vendored, the ledger's file inventory vouches for it: every file of the tree must match it, so a file that is gone, changed or added is a mismatch. A re-run now applies this too and rebuilds a copy whose binary was tampered with, instead of reporting it in sync; - an entry with no inventory has nothing to vouch for the tree, so the tree must then hold at least one patched file, and it must be correct. A ledger this code writes always has an inventory for platform trees. A tree that fails is therefore one of: a patched file with the wrong content, a tree that does not match its inventory, or a tree with no inventory and no patched file. The test that expected the refusal now expects the opposite: such a prebuild vendors, a dry run previews it, the check and a second run pass by the inventory, both catch a tampered binary, and without the inventory the check fails. CLI_CONTRACT.md states the new rule. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The nokogiri script gains a vendored scenario. It scans a real install whose lock pins nokogiri as platform gems against a stand-in for the patch service, which serves a patched build of each real platform gem and its stub gemspec. It checks that every platform is vendored under .socket/vendor/gem, installs from a fresh checkout with the lock frozen and the service unreachable, checks that Ruby loads the vendored build, runs vex, and rolls back to a byte-identical lock with nothing vendored left behind. Four legs run it: Ruby 3.4 and 3.1 on Linux, Ruby 3.4 on macOS, and JRuby 9.4, where the platform is java. The script also carries the hosted scenario; its legs come with the hosted change. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR SummaryMedium Risk Overview Behavior changes: missing prebuilds for any locked platform become a benign skip ( CI & tests: extends nokogiri-compatibility with Reviewed by Cursor Bugbot for commit 3a41ba0. Configure here. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 3a41ba0. Configure here.
|
One more run, beyond what the description lists, on a local build of this branch merged with the other two open gem changes (#2150, #2154, #2155). The stand-in patch service in Ruby 3.4.10, Bundler 4.0.15, macOS arm64: 45 checks pass across the hosted and vendored scenarios.
The option that serves a prebuilt gem ( |
The in-sync test of a re-run checked only the tree of each locked platform against the ledger inventory. A file outside those trees (at the copy root, or the tree of a platform the lock does not name, which bundler's glob would load once that platform is added) passed it, so a re-run reported the copy as already vendored while `vendor --check` and VEX, which match the whole artifact to the inventory, rejected it. With an inventory, the in-sync test now also matches the whole copy to it, so such a copy is rebuilt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011f9aeSJKmBzyKc5MwjJdri Assisted-by: Claude Code:claude-opus-5-5
|
[agent] PR shepherd: final review of Question: is it acceptable that a Also worth knowing before this lands:
Verified:
Generated by Claude Code |
…tform-prebuilds # Conflicts: # .github/workflows/nokogiri-compatibility.yml # crates/socket-patch-core/src/api/types.rs # crates/socket-patch-core/src/vex/discover/gem.rs # docs/ecosystems.md
|
[agent] PR shepherd: review brief for the approver What it does. In vendored mode, a gem that Risk: medium. It touches the gem vendoring pipeline, the ledger and
Look here
Verified
Changes I made: none. Open questions: none blocking. A follow-up adds the test script's Generated by Claude Code |

Summary
Vendored mode now vendors a gem that
Gemfile.lockpins as precompiledplatform gems. nokogiri is the gem this is for: its lock has one entry per
platform (
nokogiri (1.18.10-x86_64-linux-gnu), …) and Bundler never buildsan extension for a
path:gem, so until nowvendorrefused it. On JRubythe platform is
java, and the same path covers it.It needs Socket's backend to serve a patched build of each platform gem (a
"prebuild") and a stub gemspec for it. Where it serves none yet, the gem is
skipped with a warning and the run does not fail.
How it works
platform, an artifact of kind
gem-platform(the patched platform gem)and one of kind
gem-platform-stub-gemspec, each with aplatformfield.They are read from the reference alone: the ruby-platform
.gemis notdownloaded to learn that a patch has no prebuilds.
.socket/vendor/gem/<uuid>/<name>-<version>/<platform>/with its stub as<platform>/<name>.gemspec. One Bundlerpath:source points at<name>-<version>/; Bundler's glob finds one gemspec per platform, locksone spec per platform under a single
PATHsection, and loads theprebuilt binary of the platform it runs on. Nothing is compiled.
prebuild. A platform without one makes the gem a skip
(
vendor_prebuilt_platform_pending, which names the platforms): the runsucceeds, and an older vendored patch of the gem is kept. A lock that also
holds the plain ruby-platform entry beside the platform entries is refused
(
platform_gem_unsupported), and so is a lock from a Bundler older than2.3, the oldest the layout was run against.
<uuid>/.gitignore(!*) and<uuid>/.gitattributeskeep the prebuilt binaries committable under a project's
*.soor LFSrules, through the helper npm and PyPI already use. A
.gitignoreaplatform gem ships inside its own tree is emptied in the copy (warning
vendor_gem_gitignore_neutralized), because a deeper ignore file wouldwin. The git-ignored check runs before anything is downloaded, in dry and
wet runs.
service states for that platform before it is unpacked; a mismatch is
never vendored. Afterwards the ledger inventory vouches for every file of
every platform tree, in the in-sync fast path,
vendor --check,repairand restore alike (one shared rule). A patched file that is in a tree must
have its patched content. A patch whose files a platform gem does not
ship at all (a fix to the vendored libxml2 lives only in the rebuilt
binary) vendors and verifies by the inventory.
CLI_CONTRACT.mdanddocs/ecosystems.mddescribe it.What a reviewer should weigh
no older vendored patch to keep. Rebuilding a copy the lock is already
wired to,
repairof one, and the all-or-nothing eject plan still failwhen a prebuild is missing.
To add one later: revert the gem,
bundle lock --add-platform, vendoragain. The docs say so.
.gitignoreis neutralised. A.gitattributesinside aprebuild is left alone.
exact kind) and keep refusing the gem.
Code review
/code-review highraised ten findings, all fixed:restorenot putting theuuid-level
.gitignoreback; a hand-rolled copy of the uuid metadata writerthat left out
.gitattributes; the in-sync fast path not consulting theledger inventory, and a second copy of the per-tree rule with a different
policy; a shipped
.gitignoremaking the gem unvendorable with no remedy;the git-ignored check running after the downloads and not at all in a dry
run; a missing prebuild failing the run; no Bundler floor; the lock-entry
recovery not reading the array form of a checksum record; and the ruby gem
downloaded only to read the artifact list.
Cursor Bugbot's "in-sync skips full-tree inventory" finding is fixed in
aef0614: with an inventory, the in-sync test of a re-run also matches the
whole copy to it, as
vendor --checkdoes, so a file outside the lockedplatform trees forces a rebuild.
Testing
tests/nokogiri/nokogiri_e2e.rb, against a stand-in for the patchservice that repacks the real platform gems:
platform is vendored; the lock takes nokogiri from the vendored path; a
fresh checkout installs with the lock frozen, compiles nothing and asks
the patch service for nothing; Ruby loads the patched gem from the
vendored tree, there and in the scanned project;
vexattests;rollbackleaves the lock and the Gemfile byte for byte as they wereand nothing vendored behind.
same sequence on the
javaplatform gem.JRuby 9.4.
e2e_vendor_gem_build(19 pass, Bundler4.0.15): vendor, frozen install, an unbuilt platform skipped, the takeover
of an older vendored patch, repair, revert.
main:vendor::3,163,in_process_vendor157,scan_rollout_e2e20,commands::vendor81, the two contract suites. After it:cargo fmt --checkand CI's clippy command pass, and the two real-nokogiri runsabove; the suites run again in CI.
floor is tested with synthetic locks); a real patch service (the backend
change that serves prebuilds is in review). The takeover dry-run skip and
the eject-plan mapping in
commands/vendor.rshave no dedicated test.The nokogiri script also carries a hosted scenario, which this branch's CLI
does not pass; its legs come with the hosted change (#2150). The two PRs
touch the same test script and workflow, so the second to land needs a
rebase.
🤖 Generated with Claude Code
Generated by Claude Code