Repository navigation
Import JavaScript modules from Python and call their exports by name - #7950
FarhanAliRaza wants to merge 27 commits into
Conversation
`from my_app.scripts.party import celebrate` now finds my_app/scripts/party.js (or .ts) when no Python module of that name exists. Each named export becomes a JsFunction; calling it returns a `_call_js` event naming the module and the function, with the arguments as JSON. It works in event triggers, returned or yielded from handlers, and uncalled as a trigger (it takes the trigger's arguments, like a handler). An event handler argument arrives as its full name, for `env.addEvents([env.event(name, payload)])`. The compiler copies each imported module, with the .js/.ts files beside it so relative imports resolve, under utils/script_modules, and utils/scripts.js re-exports it under a key `$` makes unique. The frontend imports that index on the first call, so nothing is evaluated and an app that never calls a module does not fetch the chunk: it works under a CSP without unsafe-eval. npm packages the modules import are installed with the app. A stub (party.pyi) with untyped arguments is written next to the module, so type checkers resolve the import; typed stubs come next. The finder is stdlib-only and reflex_base.js is lazy, so `import reflex`, which installs it, still does not load the event system. The function's return value is not sent back: results will come from awaiting the event, a separate design. Part of ENG-9127. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
…arwin-dx76ek-js-modules
[Medium impact] The changes affect apps that import JavaScript modules and the shared event-trigger argument path. Update PR #7951’s compiler and stub code before merging this change through the stack. Findings
SummaryAdds Python imports for named JavaScript functions and sends calls to the browser without evaluating code.
Reviews (10) · Last reviewed commit: "Merge the *args fix from its own branch" · Reviewed by Greptile |
Merging this PR will not alter performance1 warning
Comparing Footnotes
|
…t reflex cheap
Review fixes for the JavaScript module import:
- Exports and imports are read from tokens (reflex_base/js/source.py) instead
of regular expressions over the source. Text inside strings, template
literals, regular expressions and comments is never read as code, so a
string such as "/api/*" no longer hides the exports after it. Every name of
`export const a = ..., b = ...` is found, as are destructured exports, TS
annotations with commas, and `"string" as name` export lists. Imports are
found wherever line breaks fall, including dynamic `import("x")`.
- The compiler copies the module and the local files it imports, followed
transitively and resolved as the bundler does (extensionless, index files,
`.js` naming a `.ts`), at their paths from the directory of the module's
package, instead of the script files beside it. Nested helpers and imported
CSS now resolve, and unused siblings are no longer bundled or installed. A
file outside the package's directory is a clear error.
- RegistrationContext.fork() copies script_modules, so an app built in a fork
still compiles the modules imported before it.
- The finder no longer imports pathlib, importlib.abc or typing, which
`import reflex` does not load otherwise (CodSpeed: test_import_reflex
+1.6ms); a test pins it.
- Test docstrings.
Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
…arwin-dx76ek-js-modules
Review fixes: - Files a module imports that are not scripts (stylesheets, images, fonts) were read as UTF-8 into the compile output, so `import logo from "./logo.png"` stopped the compile with UnicodeDecodeError. compile_script_modules now returns them as (source, target) copies, which compile_app copies byte for byte after the dry-run return, skipping targets that are up to date. - A copied stylesheet's `@import` and `url(...)` references are followed too, so `./theme.css` and `./bg.png` used by an imported `party.css` are bundled. Absolute paths, URLs, data URIs and fragments are left alone. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
CodSpeed still showed test_import_reflex about 15% slower on c0ebd9c. The benchmark clears the reflex modules and imports `reflex` again in the same process, so it measures module execution: importing the lazy reflex_base.js package and its finder module on every `import reflex` cost about 200us of about 1300us. It also showed a bug: install() matched the finder with isinstance, and a re-imported finder module is a new class, so every fresh import appended another finder to sys.meta_path (205 after 200 rounds). reflex_base/__init__.py, which every Reflex import loads first, now registers a small finder class that loads reflex_base.js.finder only on the first import no other finder found, matched by name so a fresh import never adds a second. `import reflex` loads no module for it (about 12us of execution), and reflex/__init__.py is back to its previous form, so the generated reflex/__init__.pyi stays free of runtime code. Tests pin one finder after a re-import and that `import reflex` does not load reflex_base.js. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
Review fixes: - call_event_fn counted `*args` as one required parameter and passed only as many arguments as the callable declared, so an uncalled JS module function failed to compile on a trigger with no arguments (`on_mount=initialize`) and got only the first argument of one with several. The same broke any `lambda *args: ...`. Variadic parameters are no longer required, and a `*args` callable gets every argument. - A relative import with a query or fragment (Vite's `./message.txt?raw`, `./logo.png?url`) resolves to its file again, so the file is copied with the module, which keeps the import as written. Package specifiers drop the query before install. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
…arwin-dx76ek-js-modules Keep reflex_base.js a package: the base's js.py becomes js/api.py, whose ref, add_events and library the package re-exports lazily, since the module finder `import reflex` registers lives in the package too. The obsolete environment.py goes, test_js.py moves to js/test_api.py, and the JavaScript module tests, docs and demo use `reflex` from $/utils/reflex. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
A callable taking `**kwargs` or a keyword-only parameter was called with one argument per parameter, so `lambda key, **kw: ...` on on_key_down got two positional arguments and raised TypeError. Count only positional parameters, both to call the function and to check that the trigger provides enough arguments; `*args` still takes all of them. The varargs test moves next to the new one in the event module's test file. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
Scope: the finder only claims modules of the app's package, the package rxconfig's app_name names, so a third-party package's .js file is never imported or registered. The config loads only once a file is found, and the finder is now a function the stub in reflex_base calls. Parsing: one scanner serves tokenize, template expressions and JSX, and each file is read and tokenized once per compile (module_files returns each script's text, exports and imports). `=>`, `++` and `--` are single tokens, so annotations like `() => Map<K, V>` and `i++ / 2` read right; `.jsx`/`.tsx` files skip JSX text; type items of export lists, generic arrow functions, `import type` and `export type` are handled. A `const` bound to a literal is a value, not a function. Every new function stays within a complexity of 8. Compile: files must be inside the app package's directory, not merely the project's. utils/scripts.js now maps each module's Python name, as a JSON string, to a loader, so any module name or path is safe in the generated code. Specifiers with a scheme, `#` or `@/` are not installed, and Node.js built-ins fail the compile with what to do instead. Errors: importing a missing function names the file and its functions; importing a value says only functions can be called; the browser error for a missing function gives both causes and both fixes. Frontend: state.js loads the app's modules with the page and caches them, so a `_call_js` runs its function before awaiting anything and the next event, even one processEvent starts concurrently, sees its effect. An app without modules loads nothing. Errors of event code are reported through one helper. Public API: the module helpers are private (_CALL_JS, _JsFunction, _load_module, _render_stub, _package_imports), module_key is gone, and RegistrationContext.script_modules is _script_modules. Tests follow the module layout: test_finder.py, test_modules.py, test_source.py, and reflex_base/test_init.py. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
A handler yields a module function and then a script that reads what it wrote; each arrives in a message of its own. With the module loaded on demand, the script ran first and read the old text. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
The section leads with a runnable app, then the rules (the app package is the scope, files must be inside it, what is installed, arguments as JSON, how to get a value back, JSX only in .jsx/.tsx) and each error with its fix. News fragments say the same for downstream users. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
The helper that reports errors of event code now serves four blocks, so a test pins what each logs and reports, with and without window.onerror. A string after `from` always ends an import or export clause in valid code, so finding the clause's keyword needs no bail-out branches. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
Loading the modules at page load does not guarantee they finish before events arrive. While a _call_js waits for its module, processEvent now holds the events after it, in the order they arrived, until the call has run, so they see what its function did. Nothing waits once the module has loaded. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
The tokenizer skipped the expressions of templates and JSX without keeping
their tokens, so an import("./x.js") in ${...} or {...} was never copied.
A literal's token now holds the tokens of its expressions, and the import
search walks them; the export search still sees one token per literal.
Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
An event that waited for one pending module call could run ahead of the next one: the event before it may be a module call that only starts waiting for its module once the first has run. processEvent now waits until no call is pending. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
|
CodSpeed reports
So the code under that benchmark is byte-for-byte the same in both runs. Generated by Claude Code |
A callable given to an event trigger, such as `lambda key, **kw: ...` or `lambda key, *, flag=1: ...`, raised TypeError because every parameter, keyword-only and `**kwargs` included, was counted as a slot for the trigger's arguments. `*args` now takes all of them, even none. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
Counting only positional parameters let `def handle(self, value, *, required)` bind to a one-argument trigger and fail later in the backend. Only `*args` and `**kwargs` are left out of the count now. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
A function given to an event trigger gets the trigger's arguments by position, so a required keyword-only parameter can never be filled. It failed with a TypeError when the event fired; it now fails when the page is built, with an error that names the parameter and the fix. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
script-modules.js parses each module the app imports from Python with rolldown's parser, which every app installs with vite, follows its local imports and stylesheet references, and prints each module's files and exports, each script's imports and the first syntax error of each file. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
Importing a JavaScript module no longer reads it: each identifier looked up on the module is a function, recorded for the compiler. At compile, after the frontend packages (and with them the parser) are installed, script-modules.js reads the modules; the compiler checks that each name Python used is an exported function, reports syntax errors at their file:line:column, installs the packages the modules import and copies their files. An app that imports no module runs no script, and --dry never does. The hand-written tokenizer (reflex_base/js/source.py) and its tests are removed. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
It returns when nothing is queued; it does not keep a second drain from starting while one is running. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
Usage first: any syntax the bundler reads, no star import, names used only in event handlers checked in the browser, and each compile error with its fix. The frontend README lists script-modules.js and how it finds rolldown. Claude-Session: https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf
Type of change
Description
Second of the ENG-9127 stack, on top of #7949.
Problem. More than a line of JavaScript ends up as strings passed to
rx.call_script. Those strings get no editor support or lint, the browser evals them, and they break under a Content Security Policy withoutunsafe-eval.Change. Write the JavaScript in a
.js,.ts,.jsxor.tsxfile in your app package, and import its exported functions from Python:Each named export becomes a Python function that returns an event. Use it in an event trigger, or return or yield it from a handler. Passed to a trigger without being called, it receives the trigger's arguments. The browser receives the module, the function name and the arguments as JSON, and evaluates nothing.
Rules.
.jsfiles in other installed packages are never claimed../helpers.js, CSS with its@import/url()references, images) are bundled with it. They must be inside the app package's directory.virtual:,#…,@/…and type-only imports. A Node built-in with no browser package (fs,node:*) stops the compile with an error.reflex.addEvents.from module import *is not supported: Python does not read the file, so it cannot list its exports.Errors. Each one states the problem, its cause and the fix, and is pinned by an exact test. All are raised when the app compiles:
nope): names the file, lists its functions, and shows theexport function nope() { ... }to add.export const notAFunction = 5used from Python: it is a value, not a function, which Python cannot call.file:line:columnand the parser's message.Design.
__getattr__returns a function for any name and records the name. The backend needs no parser and no Node.js:--backend-onlydeployments are unaffected.script-modules.jsruns in.webwith the oxc parser that ships with rolldown, which Vite 8 installs in every app. It reports each module's function and value exports, its local files and its npm imports, so JS, TS, JSX and TSX are parsed exactly. This replaces an earlier hand-written tokenizer (~890 lines). The compile then checks every name Python used. The node call takes ~40 ms, and only for apps that import a module. The first compile of such an app installs the frontend packages before reading the modules, because the parser comes with them.import reflexdoesn't load the event system. It claims a module only if a matching script exists under the app package, and reads the config only then.utils/scripts.jsmaps module names to() => import(...)with JSON-escaped strings, so each module is its own lazy chunk in the production build. Loading starts at page load and is cached, so a module function runs before anything awaited after it.processEventintentionally lets several drains run at once (state_connection.test.jspins that). So events wait only behind a pending module call rather than behind a global in-flight guard. The misleading comment above its empty-queue check now says what the check does.*args/**kwargsfix for callables passed to triggers comes from there, merged in until it lands.Tests
test_finder.py,test_modules.py,test_compiler_utils.py,test_compiler.py): the import scope, name recording, the compile-time checks and their exact messages, the folder limit, generated module names, npm detection, the script runner (real Node), and install ordering.script-modules.json fixtures (TS, JSX/TSX, CSS, assets, dynamic imports,export *, type-only exports),_call_js, and the ordering of a module function before the next event.test_call_script.py: module functions from a trigger and from a handler, an async function, an npm package, CSS, an uncalledon_change, and the ordering.npm run checkare clean.https://claude.ai/code/session_01LwJUCVxPh3ihrnaFn6CWnf