Repository navigation
fix: compile validators on first use - #333
rami-hatoum wants to merge 2 commits into
Conversation
The validation module created its AJV instance and compiled a validator
for each of the 171 types in validationPointers as soon as the SDK was
imported: 217 functions generated from strings, about 110 ms on
Node.js 26 and 130 ms on Node.js 22, before anything was validated.
Deserializing with `{ validate: false }`, building a graph or writing
Mermaid code paid for it all the same, and where generating code from
strings is refused, such as a browser whose Content Security Policy
leaves out 'unsafe-eval', importing the SDK threw an EvalError.
The AJV instance is now created, and a type's validator compiled, the
first time that type is validated, and the compiled validator is kept
for the calls that follow. The errors for an unknown type and a missing
schema are unchanged, and so is the public API.
Signed-off-by: Rami <5664927+rami-hatoum@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t use, not when imported (#308) ## What this changes `@openworkflowspec/sdk` builds an Ajv instance over its schema and compiles all 171 of its validators the moment it is imported, generating code from strings, though the runtime never asks it to validate: the coordination capability checks a document with an interpreting validator and uses the SDK for its classes and its graph alone. A library that compiles what nobody asks for, at import, is a defect of the library, and the fix is the library's, compiling a validator the first time `validate` is called for a type; it is offered upstream as `open-workflow-specification/sdk-typescript#333`. Until the library ships it, the runtime carries it as a dependency patch, byte-identical to the one Studio carries, named in the workspace and copied into the container image before its install. Measured on a quiet machine: importing the SDK took 119 ms of wall time and 174 ms of CPU as published, 19 ms and 28 ms patched, generating 217 functions as published and none patched; the first workflow save, which now loads the SDK, takes 82 ms against 182 ms with the SDK as published; the second save is unchanged. The coordination and tool-servers READMEs carry the figures with their dates; the minified browser bundle is left as published. ## Tests - A child Node process imports the SDK, hydrates a workflow, builds its graph and writes its Mermaid with code generation from strings disallowed, and only `validate` throws; without the patch the import itself fails. - `pnpm check` passes. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
LGTM |
JBBianchi
left a comment
There was a problem hiding this comment.
Thanks for the PR and the detailed explanation!
I reproduced the issue on main: importing dist/index.cjs takes around 430 ms, and fails with an EvalError when running with --disallow-code-generation-from-strings.
The proposed approach looks good overall. In particular:
- Making the AJV instance lazy is important, since even its constructor generates code.
- Using a
Mapavoids issues with special property names. - Not caching compilation failures seems reasonable.
I have a few suggestions, mostly around testing and error handling.
1. Add a test to check all validation pointers (requested)
Previously, importing the SDK compiled all 171 validators. This meant that an invalid or missing schema pointer would immediately cause a test failure.
With lazy compilation, we lose that automatic check. Since the tests only validate a handful of types, a broken pointer could go unnoticed until someone actually uses it.
I'd suggest adding a test that checks whether every pointer resolves to a valid schema:
it('should resolve the schema of every type', () => {
const ajv = new Ajv({ schemas: [workflowSchema], strict: false });
addFormats(ajv);
const unresolved = Object.entries(validationPointers)
.filter(([, pointer]) => !pointer || !ajv.getSchema(pointer))
.map(([typeName]) => typeName);
expect(unresolved).toEqual([]);
});An alternative would be to move the lazy validation helpers into an internal module and test them directly. This would avoid duplicating the AJV configuration and keep those helpers out of the public SDK API.
2. Should compilation errors be treated as validation errors? (question)
Currently, getValidator runs inside withValidationContext. As a result, errors caused by the environment (such as blocked code generation) are converted into DslValidationError.
The problem is that these errors don't necessarily mean the user's workflow is invalid.
For example, an editor catching WorkflowValidationError might tell users their document is incorrect, when the real issue is that their environment doesn't allow code generation.
One possible improvement would be to resolve the validator before entering withValidationContext in validate().
This would make the distinction clearer:
- Invalid workflow: a
WorkflowValidationError. - Compilation or environment issue: the original error.
The unknown-type error could still be wrapped as it is today.
Not a blocker, but I think it's worth deciding how these errors should be handled, especially since the new test establishes the current behaviour.
3. Finish configuring AJV before storing it (minor)
In validation.ts (L39–43), ajv is assigned before calling addFormats().
This works today, but I'd suggest completing the initialization first, so we never store a partially configured instance:
const instance = new Ajv({
schemas: [workflowSchema],
strict: false,
});
addFormats(instance);
ajv = instance;4. Simplify the no-codegen test (minor)
In validator-compilation.spec.ts (L107–123), the test checks exact node IDs and the complete Mermaid output.
These details are already covered by other tests, and they aren't really related to the purpose of this test.
I'd keep this test focused on verifying that the SDK works without code generation when validation is disabled.
For example, we could simply check that the relevant operations succeed, rather than asserting their exact formatted output.
Also, if the child process crashes, several tests currently fail with a JSON parsing error. Including run.stderr in the error reported by readOutput would make these failures much easier to debug.
Optional improvements (can be follow-ups)
A few additional ideas that don't need to block this PR:
- Bundle tests:
scripts/validate-tarball.mjsalready tests the UMD bundle in a VM. We could disable string-based code generation there and add equivalent checks for CJS/ESM. This would turn the manual verification into an automated regression test. - README: Add a short note explaining that validation requires runtime code generation (
unsafe-evalunder CSP), and that{ validate: false }allows using the SDK without it. This is particularly relevant because builders anddeserializeenable validation by default. - Repeated compilation failures: When CSP blocks code generation, each validation attempt retries compilation, and AJV logs roughly 8 KB of generated code to
console.error. Not a blocker, but something to keep in mind. - CSP-compatible validation: AJV's standalone validators could eventually allow validation without runtime code generation. That's a larger change and probably deserves a separate discussion.
Overall, the approach makes sense to me. The main thing I'd like to see before merging is the test covering all validation pointers. The error-handling question is worth discussing; the remaining points are relatively minor.
…inter The helpers that create the AJV instance and compile a type's validator on first use move from validation.ts into validators.ts, an internal module the SDK does not export. A new test compiles the validator of every type in validationPointers through that module, with the SDK's own AJV configuration, so a pointer that is missing or resolves to no schema fails the tests, as it did when every validator was compiled at import. validate() now gets the validator before it enters the validation context. An invalid workflow still throws a WorkflowValidationError, and an unknown type still throws the same DslValidationError. An error raised while compiling a validator, such as the EvalError thrown where code generation from strings is disallowed, is now thrown unchanged, since it does not mean that the workflow is invalid. The AJV instance is stored only once addFormats has configured it. The test without code generation now checks that deserializing, building the graph and writing the Mermaid code succeed, not their exact output. If the child process crashes, the tests report its exit status and its stderr, quoted line by line so that its stack frames are not taken for frames of the test's own error. Signed-off-by: Rami <5664927+rami-hatoum@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for the careful review, and for reproducing the issue. The new commit addresses the four points.
If they would help, I'm happy to follow up with separate pull requests for the README note on runtime code generation and |
Many thanks for submitting your Pull Request ❤️!
What this PR does / why we need it:
src/lib/validation.tscreated its AJV instance and compiled a validator for every entry ofvalidationPointers, 171 types, while the module was being evaluated. In other words, it did this as soon as the SDK was imported. That has two costs:index.cjsgenerated 217 functions from strings. The import, with its dependencies, took about 106 ms on Node.js 26.10 and 124 ms on Node.js 22.23. With this change it takes about 15 ms and 18 ms. Of that, the SDK's own code went from about 96 ms and 112 ms to about 3 ms; loading its dependencies takes the remaining 14 to 17 ms either way. Every caller that deserializes with{ validate: false }, builds a graph or writes Mermaid code paid that cost each time it started.new Function. Undernode --disallow-code-generation-from-strings, in avmcontext created withcodeGeneration: { strings: false }, or in a browser whose Content Security Policy leaves out'unsafe-eval', importing the SDK threwEvalError: Code generation from strings disallowed for this context. Nothing in the SDK could be used, not even the parts that never validate.With this change, the AJV instance is created, and a type's validator is compiled, the first time
validate(typeName, …)is called for that type. Thevalidatorsmap keeps the compiled validator for the calls that follow. The firstvalidate('Workflow', …)compiles 47 functions, which takes about 55 ms on Node.js 26 and 65 ms on Node.js 22. The second call takes about 0.1 ms, as it did before.The public API is unchanged.
validatekeeps its signature and throws the same errors. An unknown type still fails withUnable to find a validation function for '<type>', wrapped in aDslValidationErroras before. I compared the output before and after forUnknown,toStringand__proto__, and it is identical. TheNo JSON pointer provided…andUnable to find schema…guards keep their exact wording. They are now raised the first time a type is validated, not at import.Special notes for reviewers:
validatenow throws, and the rest of the SDK works. The error is aDslValidationErrorat<root>whosecauseis theEvalError, which is howvalidatealready reports any other error raised inside it. AJV still logs the generated source toconsole.errorwhen that compile fails, just as it did at import before.jsonPointersis aMapbuilt fromvalidationPointersat import. Building it compiles nothing. Using aMapmeans a name such astoStringor__proto__is reported as an unknown type, not looked up on the object's prototype.tests/validation/validator-compilation.spec.tsstarts Node.js with--disallow-code-generation-from-stringsand--import tsx(already a dev dependency), then requiressrc/index.ts. The flag can only be set for a whole process, which is why the test needs a child process. The test deserializes a workflow with{ validate: false }, builds its graph, writes its Mermaid code and only then validates it. It asserts that every step beforevalidatesucceeds and thatvalidatethrows, with anEvalErroras the cause.Additional information (if needed):
Verification.
npm run lint,npm test(215 tests across 22 files, up from 210 across 21),npm run validate:packageandnpm run codegen:checkall pass on Node.js 22.23. With the unchangedvalidation.ts, all five new tests fail with theEvalErrorat import. Outside the suite, I checked the built bundles in two ways: the builtindex.cjsundernode --disallow-code-generation-from-strings, and the minified UMD bundle in avmcontext withcodeGeneration: { strings: false }. Both import, deserialize with{ validate: false }, build a graph and write Mermaid code, and throw only fromvalidate.No version bump. The version is set at release time, as step 1 of
RELEASING.mddescribes.