Feature/use forked blue quickjs #134
Open
+5,011
−1,488
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4baef7094
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
317
to
159
| if (!result.ok) { | ||
| throw mapEvaluateError(result); | ||
| } | ||
| context.setProp(context.global, 'console', consoleHandle); | ||
| }); | ||
| } | ||
|
|
||
| private installDeterministicGlobals(context: QuickJSContext): void { | ||
| // Hide non-deterministic and host-specific globals from user code | ||
| // Ensure typeof returns 'undefined' for these symbols | ||
| context.setProp(context.global, 'Date', context.undefined.dup()); | ||
| } | ||
|
|
||
| private installCanonNamespace(context: QuickJSContext): void { | ||
| context.newObject().consume((canonHandle) => { | ||
| context | ||
| .newFunction('unwrap', (...args: readonly QuickJSHandle[]) => { | ||
| try { | ||
| const [targetHandle, deepHandle] = args; | ||
| const target = | ||
| targetHandle !== undefined | ||
| ? context.dump(targetHandle) | ||
| : undefined; | ||
| const deep = | ||
| deepHandle === undefined | ||
| ? true | ||
| : Boolean(context.dump(deepHandle)); | ||
| const value = unwrapCanonicalValue(target, deep); | ||
| return this.createReturnHandle(context, value); | ||
| } catch (error) { | ||
| const message = | ||
| error instanceof Error ? error.message : String(error); | ||
| return context.newError(message); | ||
| } | ||
| }) | ||
| .consume((handle) => context.setProp(canonHandle, 'unwrap', handle)); | ||
|
|
||
| context | ||
| .newFunction('at', (...args: readonly QuickJSHandle[]) => { | ||
| try { | ||
| const [targetHandle, pointerHandle] = args; | ||
| const target = | ||
| targetHandle !== undefined | ||
| ? context.dump(targetHandle) | ||
| : undefined; | ||
| if (pointerHandle === undefined) { | ||
| throw new TypeError( | ||
| 'canon.at(target, pointer) requires a pointer argument', | ||
| ); | ||
| } | ||
| const pointerValue = context.dump(pointerHandle); | ||
| if (typeof pointerValue !== 'string') { | ||
| throw new TypeError('canon.at pointer must be a string'); | ||
| } | ||
| const resolved = getCanonicalPointerValue(target, pointerValue); | ||
| return this.createReturnHandle(context, resolved); | ||
| } catch (error) { | ||
| const message = | ||
| error instanceof Error ? error.message : String(error); | ||
| return context.newError(message); | ||
| } | ||
| }) | ||
| .consume((handle) => context.setProp(canonHandle, 'at', handle)); | ||
|
|
||
| context.setProp(context.global, 'canon', canonHandle); | ||
| }); | ||
| } | ||
|
|
||
| private installBindings( | ||
| context: QuickJSContext, | ||
| bindings: QuickJSBindings | undefined, | ||
| ): void { | ||
| if (!bindings) { | ||
| return; | ||
| } | ||
|
|
||
| for (const [key, value] of Object.entries(bindings)) { | ||
| if (typeof value === 'function') { | ||
| const fnHandle = this.createFunctionHandle( | ||
| context, | ||
| key, | ||
| value as (...args: unknown[]) => unknown, | ||
| ); | ||
| fnHandle.consume((handle) => | ||
| context.setProp(context.global, key, handle), | ||
| ); | ||
| } else if (value !== undefined) { | ||
| const valueHandle = this.createOwnedHandle(context, value); | ||
| valueHandle.consume((handle) => | ||
| context.setProp(context.global, key, handle), | ||
| ); | ||
| if (wasmGasLimit !== undefined && onWasmGasUsed) { |
There was a problem hiding this comment.
In QuickJSEvaluator.evaluateOnce, errors are thrown before onWasmGasUsed is invoked. That means any evaluation that fails (e.g., syntax error or OutOfGas) skips gas accounting, so callers like JavaScriptCodeStepExecutor never charge the meter even though wasm fuel was consumed. This under-reports gas for failed or runaway scripts and can be exploited by repeatedly triggering out-of-gas errors to avoid being billed. Consider invoking the callback with result.gasUsed before throwing, or in a finally block when result is available.
Useful? React with 👍 / 👎.
mjwebblue
force-pushed
the
feature/use-forked-blue-quickjs
branch
4 times, most recently
from
5a0d06f to
29f33d6
Compare
mjwebblue
force-pushed
the
feature/use-forked-blue-quickjs
branch
from
29f33d6 to
093b92c
Compare
feat(document-processor): add gas metering support for QuickJS Wasm
- Introduced gas metering capabilities in the QuickJSEvaluator to limit execution based on a gas budget.
- Added a new `quickjs-gas-loader` module to handle loading instrumented QuickJS Wasm.
- Created a `gas-schedule.md` to document operation costs for gas metering.
- Updated CI workflow to install Rust and instrument QuickJS Wasm.
- Added tests to validate gas metering functionality, ensuring proper handling of out-of-gas scenarios.
This enhancement allows for more controlled execution of scripts, preventing infinite loops and excessive resource consumption.
**feat(document-processor): integrate wasm gas metering for QuickJS execution**
- Charge QuickJS wasm gas usage into `GasMeter` via `JavaScriptCodeStepExecutor` with a default wasm gas limit and metered QuickJS module loading.
- Add wasm gas accounting tests for JavaScript code steps and adjust QuickJS evaluator gas test limits and loader path.
- Refactor Rust `quickjs-gas-instrument` into a clap-based CLI using mutable global metering, EVM-like cost rules, and updated CI instrumentation script.
refactor: rely on wasm gas_left global for QuickJS gas metering
- Remove custom `gas_charge` import and JS-side `currentGasBudget` tracking in `quickjs-gas-loader`.
- Assume wasm is instrumented with the `mutable_global` backend exposing `gas_left` as the fuel counter.
- Update reset logic to set `__gasGlobal.value` only, aligning gas metering with the new deterministic instrumentation.
refactor: extract gas metering logic into QuickJSGasController
Extract gas-related functionality from QuickJSEvaluator into a dedicated
QuickJSGasController class to improve separation of concerns and testability.
Key changes:
- Move gas limit setting, reading, and error normalization into controller
- Simplify findGasGlobal to rely on loader invariants (__gasGlobal and
__gasExports.gas_left) instead of checking multiple nested paths
- Remove complex path traversal logic that checked 7+ different module
structure variations
- Remove debug console.log from test file
This refactoring maintains the same functionality while making the code
more maintainable and easier to test in isolation.
refactor: simplify resource disposal in QuickJSEvaluator
Removed the `safeDispose` flag and streamlined the resource disposal logic in the QuickJSEvaluator class. This change enhances code clarity by directly disposing of resources without conditional checks, while maintaining the same error handling for disposal issues during runtime aborts.
feat: add WASM gas metering to expression evaluation
- Extract DEFAULT_WASM_GAS_LIMIT to shared quickjs-config module
- Add DEFAULT_EXPRESSION_WASM_GAS_LIMIT constant for expression evaluation
- Wire WASM gas tracking through expression evaluation pipeline:
- Add wasmGasLimit and onWasmGasUsed parameters to evaluateQuickJSExpression
- Propagate gas tracking through resolveTemplateString
- Charge WASM gas usage via gas meter in resolveNodeExpressions
- Add test coverage for WASM gas charging during expression evaluation
This enables deterministic gas metering for embedded JavaScript execution
by tracking and charging WASM gas consumption for each evaluated expression.
refactor(gas): remove redundant static gas charges for JS execution
Replace static size-based charges (jsCodeBaseAmount, expressionAmount,
templateAmount) with actual WASM fuel consumption via chargeWasmGas.
This aligns with Ethereum-style deterministic gas metering where cost
reflects actual VM work, not code size heuristics.
refactor: enhance QuickJS evaluator with metered WASM loading
- Introduced `getMeteredQuickJS` function to manage caching and loading of metered QuickJS modules.
- Simplified the `QuickJSEvaluator` class by removing redundant module promises and instances.
- Updated gas controller initialization to always use metered execution.
- Refactored WASM loading logic in `quickjs-gas-loader` to improve clarity and error handling.
These changes streamline the integration of WASM gas metering, ensuring efficient module management and improved code maintainability.
feat: introduce quickjs-wasm-gas library
refactor(quickjs-wasm-gas): extract gas metering into standalone variant package
Move the gas-instrumented QuickJS WASM module from document-processor into
@blue-labs/quickjs-wasm-gas as a proper quickjs-emscripten variant.
Key changes:
- Implement official quickjs-emscripten variant pattern with default export
- Add createGasVariant() factory for custom WASM URLs
- Export gas control helpers: setGasBudget(), getGasRemaining(), getGasGlobal()
- Move quickjs.release.gas.wasm to quickjs-wasm-gas package
- Simplify document-processor by consuming the new package
- Update CI instrumentation script output path
This makes the gas-metered variant independently publishable and reusable,
following the same pattern as official quickjs-emscripten variant packages.
feat: change name to more like emscripten variant
refactor(quickjs-wasm-gas): make package self-contained with integrated build tooling
- Move Rust instrumentation tool into libs/quickjs-wasmfile-release-sync-gas/tools/
- Move gas-schedule.md docs into the package
- Replace root-level ci-instrument-quickjs.sh with package-local instrument-wasm.sh
- Add Nx instrument-wasm target with proper inputs/outputs caching
- Make build target depend on instrument-wasm for automatic WASM generation
- Remove generated quickjs.release.gas.wasm from git (now built during build)
- Add .gitignore for Cargo artifacts and generated WASM
- Update README with comprehensive usage docs and build instructions
- Remove manual WASM instrumentation step from publish workflow
The package is now fully self-contained and builds the instrumented WASM
as part of the standard Nx build process, with proper caching based on
tool sources and upstream WASM changes.
refactor(quickjs-wasm-gas): rename output wasm to emscripten-module-gas.wasm
Rename the gas-instrumented WASM file from quickjs.release.gas.wasm to
emscripten-module-gas.wasm for consistency with the upstream source file
naming (emscripten-module.wasm).
Updated references in:
- Test files (quickjs-evaluator-gas, quickjs-fuel-calibration)
- Package exports and build scripts
- Nx project outputs
- Instrumentation tool paths
refactor(quickjs-gas): simplify gas variant to always use bundled WASM
Remove support for custom WASM URLs in favor of the bundled gas-instrumented
WASM file. This simplifies the API and removes unnecessary configuration.
Changes:
- Remove `instrumentedWasmUrl` option from QuickJSEvaluator.evaluate()
- Export `gasVariant` directly instead of `createGasVariant()` factory
- Make `defaultWasmUrl()` private (renamed to `getBundledWasmUrl()`)
- Simplify module caching from Map-based multi-key to single instance
- Remove conditional test execution and WASM path detection from tests
chore: update configuration and scripts for QuickJS WASM instrumentation
- Added new ignore pattern for ESLint to exclude the target directory of the QuickJS gas instrumentation tool.
- Updated Vite configuration to allow passing with no tests.
- Modified the instrument-wasm.sh script to set the target directory for output files to the new tools path.
These changes enhance the build process and ensure proper configuration for the QuickJS WASM project.
chore: add dependency for QuickJS WASM file synchronization
- Included `@blue-labs/quickjs-wasmfile-release-sync-gas` as a dependency in the document-processor package.
This addition supports the integration of the QuickJS WASM file synchronization functionality.
chore: update .gitignore files and add Cargo.lock for QuickJS gas instrumentation
- Removed unnecessary `target` entries from the main and QuickJS-specific .gitignore files.
- Added `Cargo.lock` for the QuickJS gas instrumentation tool to ensure consistent dependency resolution.
These changes streamline the build process and improve dependency management for the QuickJS project.
chore: add Rust toolchain configuration for QuickJS gas instrumentation
- Introduced a new `rust-toolchain.toml` file specifying the Rust toolchain version (1.91.1) for the QuickJS gas instrumentation tool.
This addition ensures consistent Rust environment setup for development and builds.
chore: add Rust installation step to CI workflow
- Included a step to install the Rust toolchain version 1.91.1 in the CI workflow configuration.
This addition ensures that the necessary Rust environment is set up for the build process.
chore: update CI and publish workflows to use Docker for deterministic builds
- Removed Rust installation steps from CI and publish workflows, as they are no longer needed.
- Added environment variable `USE_DOCKER=true` to ensure consistent WASM instrumentation using a pinned Rust Docker image.
- Updated the `instrument-wasm.sh` script to support Docker-based builds, enhancing cross-platform consistency.
These changes streamline the build process and improve reliability across different environments.
chore: remove USE_DOCKER environment variable from CI and publish workflows
- Eliminated the USE_DOCKER environment variable from the CI and publish workflows, as it is no longer necessary for deterministic builds.
- Updated the instrument-wasm.sh script to default to using Docker for builds, enhancing clarity and reliability.
These changes simplify the workflows and ensure consistent behavior across different environments.
chore: update Node.js version in CI and publish workflows to 22
- Changed the Node.js version from 20 to 22 in both CI and publish workflows to ensure compatibility with the latest features and improvements.
- Updated the .gitignore file to include a new target directory for Docker builds, preventing conflicts with native builds.
- Modified the instrument-wasm.sh script to specify the Docker platform as linux/amd64 and adjusted the target directory for Docker builds.
These changes enhance the build process and maintain consistency across different environments.
chore: enhance CI workflow to build and verify WASM checksums
- Added a build step for the QuickJS WASM file to generate the output before verification.
- Implemented a checksum verification step to ensure the integrity and determinism of the generated WASM files.
These changes improve the reliability of the build process by ensuring that the output matches expected checksums.
fix: reorder gas limit setting in QuickJSEvaluator
- Moved the gas limit setting for the WASM controller to occur after the context setup in the QuickJSEvaluator class. This ensures that the gas limit is applied correctly before evaluating the wrapped code.
- Updated ESLint configuration to include a new ignore pattern for the QuickJS gas instrumentation tool's Docker target directory, improving the project's linting process.
These changes enhance the functionality and maintainability of the QuickJS evaluator and its associated tooling.
test(quickjs): add gas metering non-determinism proof tests and simplify evaluator
Add comprehensive test suite proving WASM-level gas metering is not
deterministic across runtimes, contexts, and platforms:
- Runtime/context creation produces different gas values for identical code
- Same-context evaluations still show variance
- Gas values differ between module instances
- First evaluation has significantly more overhead than subsequent ones
- Platform-specific variance tests (local vs CI)
Also adds hostFuel conversion output to baseline fuel samples.
Simplify QuickJSEvaluator promise handling:
- Remove unused executed job count variable
- Remove timeout check during pending jobs (gas limits handle this)
- Clean up promise rejection error extraction code
refactor(document-processor): migrate QuickJS eval to @blue-quickjs runtime
- Replace `quickjs-emscripten` + gas-instrumented WASM with `@blue-quickjs/{quickjs-runtime,dv,abi-manifest}`
- Remove `quickjs-wasmfile-release-sync-gas` project and CI WASM build/checksum steps
- Enforce deterministic execution: disallow `async/await` and function-valued bindings; normalize `undefined` → `null`
- Make `JavaScriptCodeStepExecutor` gas limit configurable; update gas/determinism tests and snapshots
style(document-processor): apply prettier formatting to processors and types
- Reformat class declarations to single lines across multiple processor files.
- Adjust indentation and multiline formatting for generic interfaces in registry types.
feat(document-processor): upgrade quickjs to 0.2.2 and support emit binding
- bump `@blue-quickjs/*` to `0.2.2` (root + document-processor)
- wire `emit` to `Host.v1.emit` (with tests + validation)
- update fuel calibration snapshots for new metering/runtime behavior
style(processors): reformat class declarations to single lines for consistency
- Updated class declarations in `InferBasicTypesForUntypedValues.ts` and `ReplaceInlineValuesForTypeAttributesWithImports.ts` to single-line format for improved readability and consistency across processor files.
fix(document-processor): bump quickjs to 0.2.3 and update fuel snapshots
- upgrade `@blue-quickjs/*` deps to `0.2.3` (root + document-processor)
- drop `emit` prelude injection (rely on host binding directly)
- refresh fuel calibration snapshots for new runtime metering
mjwebblue
force-pushed
the
feature/use-forked-blue-quickjs
branch
from
093b92c to
959197e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.