unified: Add Swift parser based on `swift-syntax` by tausbn · Pull Request #22195 · github/codeql

tausbn · 2026-07-15T11:32:13Z

Adds an alternative parser to tree-sitter-swift, using the Swift-native swift-syntax package instead.

The current implementation exposes this parser through a Rust wrapper, using the C FFI to communicate with Swift. The Rust component passes the source file contents to the Swift component, which in turn responds with a JSON string containing the AST.

Note that at present, the AST is quite verbose (though some effort has been made to make it less so). In particular, it contains detailed location information about every single token in the input, which is almost certainly overkill for our purposes.

This PR does not attempt to replace the existing yeast rules to be based on the swift-syntax AST. That is left as follow-up work.

Can be reviewed commit-by-commit.

Adds an initial prototype of an interface from Rust to Swift, which enables us to use the `swift-syntax` package for parsing. At present, the parsed AST is passed between Swift and Rust as a JSON string.

These were taking up roughly 20% of the JSON payload.

Adds a preliminary mapping that decodes the JSON into the yeast AST. The JSON AST itself is still very verbose, but it would be useful to see if it actually contains all of the things we want it to contain.

Introduces new yeast types for `Point`s and `Range`s (corresponding to the tree-sitter equivalents), since it seemed silly to have `swift-syntax-rs` pull in `tree-sitter` just to have those types available. This _does_ mean there is a slight overhead in the shared extractor when converting between these types, but I think this is negligible.

Also gathers these into a separate side-channel during the initial AST construction. That way, we don't encounter these as weird extra nodes while running yeast.

At this point it's only a proof-of-concept translation -- it translates `sourceFile` nodes, but everything else gets mapped to 'unsupported_node`.

Copilot

Pull request overview

Adds a Swift-native swift-syntax parser exposed through Rust and prepares unified extraction to consume its JSON AST.

Changes:

Adds Swift/Rust FFI parser and Bazel/Cargo builds.
Adds JSON-to-yeast AST adaptation and minimal Swift rules.
Extends yeast for externally constructed ASTs and owned locations.

Show a summary per file

File	Description
`MODULE.bazel`	Registers Swift dependencies and toolchains.
`Cargo.toml`	Adds the parser crate to the workspace.
`Cargo.lock`	Locks the new crate.
`shared/yeast-schema/src/schema.rs`	Supports merging schema names.
`shared/yeast-macros/src/parse.rs`	Uses yeast-owned ranges.
`shared/yeast/src/build.rs`	Migrates build contexts to owned ranges.
`shared/yeast/src/lib.rs`	Adds external AST construction/desugaring APIs.
`shared/yeast/src/range.rs`	Defines owned point and range types.
`shared/yeast/tests/test.rs`	Tests desugaring hand-built ASTs.
`shared/tree-sitter-extractor/src/extractor/mod.rs`	Converts yeast points for extraction.
`unified/extractor/src/languages/mod.rs`	Exposes the Swift JSON adapter.
`unified/extractor/src/languages/swift/adapter.rs`	Converts JSON into a yeast AST.
`unified/extractor/src/languages/swift/swift.rs`	Adds minimal swift-syntax rules.
`unified/extractor/tests/fixtures/let_x.swiftsyntax.json`	Provides parser output fixture.
`unified/extractor/tests/swift_syntax_pipeline.rs`	Tests the adapter/desugarer pipeline.
`unified/swift-syntax-rs/.gitignore`	Ignores generated outputs.
`unified/swift-syntax-rs/.swift-version`	Pins Swift 6.3.2.
`unified/swift-syntax-rs/BUILD.bazel`	Defines Bazel Swift/Rust targets.
`unified/swift-syntax-rs/Cargo.toml`	Defines the Rust crate and CLI.
`unified/swift-syntax-rs/README.md`	Documents building and usage.
`unified/swift-syntax-rs/build.rs`	Builds and links the Swift shim for Cargo.
`unified/swift-syntax-rs/src/lib.rs`	Implements safe Rust FFI bindings.
`unified/swift-syntax-rs/src/main.rs`	Adds the parser CLI.
`unified/swift-syntax-rs/swift/Package.resolved`	Locks swift-syntax.
`unified/swift-syntax-rs/swift/Package.swift`	Defines the Swift FFI package.
`unified/swift-syntax-rs/swift/Sources/SwiftSyntaxFFI/SwiftSyntaxFFI.swift`	Serializes SwiftSyntax ASTs as JSON.
`unified/swift-syntax-rs/xcode_transition.bzl`	Isolates macOS Xcode configuration.

Review details

Files reviewed: 26/27 changed files
Comments generated: 9
Review effort level: Medium

+    println!("cargo:rustc-link-search=native={}", build_dir.display());
+    println!("cargo:rustc-link-lib=dylib=SwiftSyntaxFFI");
+    println!("cargo:rustc-link-arg=-Wl,-rpath,{}", build_dir.display());


- BUILD.bazel: require x86_64 on the Linux branch of the compatibility select. The only registered standalone Linux Swift toolchain is x86_64-only, so without the CPU constraint Linux/aarch64 targets stayed "compatible" and then failed toolchain resolution instead of being skipped cleanly. - build.rs: emitting `rerun-if-changed` disables Cargo's default whole-package scan, so list every input to `swift build` — the package manifests, `Package.resolved`, `.swift-version`, and the whole Sources tree — so stale shim/toolchain output can't linger. (`.build/` is left unwatched, as it is this build's own output.) - src/lib.rs: a null result from the shim is not a parse failure — SwiftParser recovers from invalid syntax and always yields a tree — so it means the shim failed to serialize/allocate the JSON. Fix the error message and the variant doc accordingly. - README.md: `.swift-version` is not honored by the macOS Bazel build (which uses the host Xcode toolchain); document that it pins the Linux and local builds only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The doc claimed the schema passed to `Ast::with_schema` must already have every node kind and field name registered, contradicting the registration methods just below and the swift-syntax adapter, which starts from `Schema::new()` and registers names on demand during construction. Document that up-front registration is optional. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

aCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

It seems that using swift_version_file has some issues when `codeql` is consumed as a dependency module. I'm hoping hardcoding the version instead will fix this, but ideally we should find a more robust solution.

Registers the `unified/swift-syntax-rs` workspace member (added in "unified: Add swift-syntax-rs") in the tree-sitter extractors crate universe. It has no external Rust dependencies, so its dependency entries are empty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jketema

I mostly skimmed the Rust changes.

For the Swift code I wonder whether we really need to write the complete serializer ourselves, or whether we could leverage things that are already there in swift-syntax. This is not blocking though.

Do we need someone with more Bazel knowledge to review those parts?

jketema · 2026-07-16T12:45:23Z

+If your `swift`/`swiftc` are not on `PATH`, point the build at them explicitly:
+
+```sh
+SWIFT=/path/to/swift SWIFTC=/path/to/swiftc cargo build
+```


Is it common to have something like this in rust builds that also compile code in other languages? Note that I'm not suggesting to change this.

I don't think we do it anywhere else, so it's a valid concern. Originally I had swift-syntax-rs much more tightly integrated with the extractor, in which case it made sense to build both at the same time. However, now that we have (for the time being) moved to a model where we call an external binary for the parsing, we could also decouple the builds themselves.

jketema · 2026-07-16T13:27:16Z

+//!   swift-syntax JSON -> `swift_adapter::json_to_ast` -> yeast `Ast`
+//!   -> `Desugarer::run_from_ast` (the real Swift translation rules) -> dump.
+//!
+//! This exercises the whole chain *without* the Swift toolchain: the JSON


This seems potentially fragile if swift-syntax changes?

Agreed. This was a "proof of concept" test that was useful at the time, but it is brittle. We can probably just get rid of it once the whole extraction is using swift-syntax-rs.

jketema

I think we should do the following (apologies for this being split in two comments).

Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

redsun82

Hey, thanks for this, seems like it was quite some work to get this working! I've looked at the bazel and build side of things, and didn't look at the actual compiled code at all. I don't have major comments on the overall direction, but I do have some comments I'd like answers to before we go ahead with this.

redsun82 · 2026-07-20T14:26:41Z

+    if attr.os != "macos":
+        # Preserve inputs so the configuration is identical to the incoming one.
+        return {
+            _XCODE_VERSION_CONFIG: settings[_XCODE_VERSION_CONFIG],
+            _EXTRA_TOOLCHAINS: settings[_EXTRA_TOOLCHAINS],
+        }


This branch can just be return {} — on the Bazel version we pin, omitted transition outputs keep their incoming value, so it's equivalent to echoing the two settings back.

If you do that, _XCODE_VERSION_CONFIG is no longer read anywhere (the macOS branch sets it to a constant without reading the incoming value), so you can also drop it from the transition's inputs and keep only _EXTRA_TOOLCHAINS.

Fixed in bf57586

redsun82 · 2026-07-20T14:26:41Z

+    # `xcode_swift_toolchain`).
+    data = select({
+        "@platforms//os:macos": [],
+        "@platforms//os:linux": ["@swift_toolchain_ubuntu24.04//:files"],


so @swift_toolchain_ubuntu24.04//:files is glob(["**/*"], exclude=["BUILD.bazel", "*.pkg"]), which is the entire swift.org toolchain, ~4.3 GB uncompressed. That's 3.3 GB of usr/ (compiler + all the static/dynamic libs), plus the 1 GB swift-6.3.2-RELEASE-ubuntu24.04.tar.gz download archive itself (the glob only excludes *.pkg, not *.tar.gz, so the tarball ships too).

At runtime though, the binary only actually needs the Swift runtime shared libs it rpaths into, i.e. usr/lib/swift/linux/, which is 142 MB total (~91 MB of .sos). So as it stands this pulls in 30-45x more than we need as runfiles, and I'm a bit worried it'll push us over the total codeql bundle size limit once we try to release it.

Could we point data at a narrow filegroup of just the runtime libs instead of //:files (something like glob(["usr/lib/swift/linux/**"]))? That'd get it down to ~90-140 MB and drop the tarball entirely.

Couple of related things, mostly for @tausbn:

the legacy Swift extractor already bundles a runtime this way (@swift-resource-dir-{os} → resource-dir/{CODEQL_PLATFORM}), so I wonder if we could reuse that rather than ship a second one. The catch is it's built for the legacy compiler fork, and the Linux Swift runtime ABI isn't stable across versions, so that only works if both end up pinned to the exact same Swift release.

it also depends on the rollout plan: if the new extractor replaces legacy in the same release, only one runtime ships either way and just narrowing is enough. But if the two coexist in the bundle for a while, we'd be shipping two runtimes unless we share one.

I have this as follow-up at the moment:

$ cat unified/swift-syntax-rs/swift_runtime.bzl """Filter the Swift toolchain down to just its Linux runtime shared objects. The standalone toolchain's `:files` bundles the compiler, host libraries, clang runtime, plugins, etc. For *running* a Swift-linked binary we only need the runtime shared objects in `usr/lib/swift/linux/`. Flattened packaging (codeql_pkg_runfiles) also can't tolerate the whole toolchain, since duplicate basenames (e.g. clang's inttypes.h, or lib_InternalSwiftScan.so appearing in multiple host dirs) collide once directory structure is stripped. """ def _swift_runtime_libs_impl(ctx): libs = [ f for f in ctx.files.toolchain if "/usr/lib/swift/linux/" in f.path and f.path.endswith(".so") ] return [DefaultInfo(files = depset(libs))] swift_runtime_libs = rule( implementation = _swift_runtime_libs_impl, attrs = {"toolchain": attr.label(allow_files = True)}, )

and:

--- a/unified/swift-syntax-rs/BUILD.bazel +++ b/unified/swift-syntax-rs/BUILD.bazel @@ -1,37 +1,22 @@ load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_test") load(":xcode_transition.bzl", "xcode_transition_swift_library") +load(":swift_runtime.bzl", "swift_runtime_libs") package(default_visibility = ["//visibility:public"]) +swift_runtime_libs( + name = "swift_runtime_libs", + toolchain = "@swift_toolchain_ubuntu24.04//:files", + )

and then do:

- "@platforms//os:linux": ["@swift_toolchain_ubuntu24.04//:files"], + "@platforms//os:linux": [":swift_runtime_libs"],

Does the above make sense?

Regarding the follow-ups: note that we do not ship any Swift on Linux at the moment so this does look like a real problem. Also I we would ship this, this would currently depend on the runtime libraries already being present on any macOS system on which this get run. I don't know if that's a valid assumption?

yeah, that looks like a good solution! and yes, you're right about Linux not being in in any case.

Fixed in cf8eeb8

redsun82 · 2026-07-20T14:26:41Z

+)
+
+register_toolchains(
+    "@swift_toolchain//:swift_toolchain_exec_ubuntu24.04",


could we use a 22.04 toolchain, for a broader glibc compatibility range?

Yes, assuming it exists.

Fixed in 38c9c84

redsun82 · 2026-07-20T14:26:41Z

+        .current_dir(&swift_dir);
+    apply_bare_repository_workaround(&mut command);
+    let status = command.status().unwrap_or_else(|e| {
+        panic!(


could we make this degrade gracefully when there's no working swift toolchain, instead of hard-panicking? our actual builds go through bazel, and cargo day-to-day is mostly just fmt/check for us, but right now build.rs runs swift build unconditionally, so with no swift on PATH even those blow up. and since the crate is a workspace member now, a plain cargo check at the repo root fails for anyone who doesn't have swift installed, not just in here.

check/clippy don't link, so they don't actually need the swift lib to exist. so I think if swift build isn't runnable we could just emit a cargo:warning=... and skip the link directives rather than panic. that keeps the cargo fmt/check loop we rely on working swift-free, and only cargo build/cargo test would fail after that (at link, which is fair enough: those genuinely need swift, and we get them through bazel anyway).

(speaking from a Linux box perspective that doesn't have Swift preinstalled 😉)

I'll leave this one for @tausbn to fix.

tausbn and others added 8 commits July 10, 2026 11:35

unified: Add swift-syntax-rs

89e11e5

Adds an initial prototype of an interface from Rust to Swift, which enables us to use the `swift-syntax` package for parsing. At present, the parsed AST is passed between Swift and Rust as a JSON string.

unified: Don't emit empty trivia

87c8173

These were taking up roughly 20% of the JSON payload.

unified: Add mapping from swift-syntax JSON to yeast AST

446ff7c

Adds a preliminary mapping that decodes the JSON into the yeast AST. The JSON AST itself is still very verbose, but it would be useful to see if it actually contains all of the things we want it to contain.

unified: Emit locations for comments

ddab6cc

Also gathers these into a separate side-channel during the initial AST construction. That way, we don't encounter these as weird extra nodes while running yeast.

unified: Hook up swift-syntax AST

58ddeac

At this point it's only a proof-of-concept translation -- it translates `sourceFile` nodes, but everything else gets mapped to 'unsupported_node`.

Support building swift-syntax-rs on macOS

3e2daf2

Upgrade swift-syntax-rs to swift-syntax 603.0.2 / Swift 6.3.2

1577d82

github-actions Bot added the documentation label Jul 15, 2026

tausbn requested a review from Copilot July 15, 2026 11:32

Copilot started reviewing on behalf of tausbn July 15, 2026 11:33 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

tausbn and others added 5 commits July 15, 2026 12:14

swift-syntax-rs: Fix bazel formatting

205c9a9

aCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

unified: Hardcode swift_version

7474a33

It seems that using swift_version_file has some issues when `codeql` is consumed as a dependency module. I'm hoping hardcoding the version instead will fix this, but ideally we should find a more robust solution.

tausbn force-pushed the tausbn/swift-syntax-rs branch from 4ba73bd to da1bbb7 Compare July 15, 2026 13:15

tausbn marked this pull request as ready for review July 15, 2026 13:58

tausbn requested review from a team as code owners July 15, 2026 13:58

jketema previously approved these changes Jul 16, 2026

View reviewed changes

jketema reviewed Jul 17, 2026

View reviewed changes

Comment thread unified/swift-syntax-rs/BUILD.bazel Outdated

Comment thread unified/swift-syntax-rs/BUILD.bazel Outdated

unified: Clean up build

f384e29

Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

tausbn dismissed jketema’s stale review via f384e29 July 20, 2026 11:10

redsun82 reviewed Jul 20, 2026

View reviewed changes

jketema added 3 commits July 21, 2026 10:49

Fix Bazel formatting

0a1139c

Simplify Xcode transition per review comments

bf57586

Only depend on the runtime libraries from the Swift toolchain

cf8eeb8

Use 22.04 Swift toolchain

38c9c84

Uh oh!

Conversation

tausbn commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

redsun82 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jketema Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tausbn commented Jul 15, 2026 •

edited

Loading

jketema Jul 20, 2026 •

edited

Loading