Skip to content

fix: compile validators on first use - #333

Open
rami-hatoum wants to merge 2 commits into
open-workflow-specification:mainfrom
rami-hatoum:validators-compile-on-first-use
Open

rami-hatoum wants to merge 2 commits into
open-workflow-specification:mainfrom
rami-hatoum:validators-compile-on-first-use

Conversation

@rami-hatoum

@rami-hatoum rami-hatoum commented Oct 23, 2026 •

Copy link
Copy Markdown

Many thanks for submitting your Pull Request ❤️!

What this PR does / why we need it:

src/lib/validation.ts created its AJV instance and compiled a validator for every entry of validationPointers, 171 types, while the module was being evaluated. In other words, it did this as soon as the SDK was imported. That has two costs:

  • Time, whether or not anything is validated. Importing the built index.cjs generated 217 functions from strings. The import, with its dependencies, took about 106 ms on Node.js 26.10 and 124 ms on Node.js 22.23. With this change it takes about 15 ms and 18 ms. Of that, the SDK's own code went from about 96 ms and 112 ms to about 3 ms; loading its dependencies takes the remaining 14 to 17 ms either way. Every caller that deserializes with { validate: false }, builds a graph or writes Mermaid code paid that cost each time it started.
  • The import fails where generating code from strings is refused. AJV compiles a schema with new Function. Under node --disallow-code-generation-from-strings, in a vm context created with codeGeneration: { strings: false }, or in a browser whose Content Security Policy leaves out 'unsafe-eval', importing the SDK threw EvalError: Code generation from strings disallowed for this context. Nothing in the SDK could be used, not even the parts that never validate.

With this change, the AJV instance is created, and a type's validator is compiled, the first time validate(typeName, …) is called for that type. The validators map keeps the compiled validator for the calls that follow. The first validate('Workflow', …) compiles 47 functions, which takes about 55 ms on Node.js 26 and 65 ms on Node.js 22. The second call takes about 0.1 ms, as it did before.

The public API is unchanged. validate keeps its signature and throws the same errors. An unknown type still fails with Unable to find a validation function for '<type>', wrapped in a DslValidationError as before. I compared the output before and after for Unknown, toString and __proto__, and it is identical. The No JSON pointer provided… and Unable to find schema… guards keep their exact wording. They are now raised the first time a type is validated, not at import.

Special notes for reviewers:

  • Where code generation is refused, validate now throws, and the rest of the SDK works. The error is a DslValidationError at <root> whose cause is the EvalError, which is how validate already reports any other error raised inside it. AJV still logs the generated source to console.error when that compile fails, just as it did at import before.
  • A failed compile is not cached, so a later call tries it again.
  • jsonPointers is a Map built from validationPointers at import. Building it compiles nothing. Using a Map means a name such as toString or __proto__ is reported as an unknown type, not looked up on the object's prototype.
  • The new test runs in a child Node.js process. tests/validation/validator-compilation.spec.ts starts Node.js with --disallow-code-generation-from-strings and --import tsx (already a dev dependency), then requires src/index.ts. The flag can only be set for a whole process, which is why the test needs a child process. The test deserializes a workflow with { validate: false }, builds its graph, writes its Mermaid code and only then validates it. It asserts that every step before validate succeeds and that validate throws, with an EvalError as the cause.

Additional information (if needed):

Verification. npm run lint, npm test (215 tests across 22 files, up from 210 across 21), npm run validate:package and npm run codegen:check all pass on Node.js 22.23. With the unchanged validation.ts, all five new tests fail with the EvalError at import. Outside the suite, I checked the built bundles in two ways: the built index.cjs under node --disallow-code-generation-from-strings, and the minified UMD bundle in a vm context with codeGeneration: { strings: false }. Both import, deserialize with { validate: false }, build a graph and write Mermaid code, and throw only from validate.

No version bump. The version is set at release time, as step 1 of RELEASING.md describes.

The validation module created its AJV instance and compiled a validator
for each of the 171 types in validationPointers as soon as the SDK was
imported: 217 functions generated from strings, about 110 ms on
Node.js 26 and 130 ms on Node.js 22, before anything was validated.
Deserializing with `{ validate: false }`, building a graph or writing
Mermaid code paid for it all the same, and where generating code from
strings is refused, such as a browser whose Content Security Policy
leaves out 'unsafe-eval', importing the SDK threw an EvalError.

The AJV instance is now created, and a type's validator compiled, the
first time that type is validated, and the compiled validator is kept
for the calls that follow. The errors for an unknown type and a missing
schema are unchanged, and so is the public API.

Signed-off-by: Rami <5664927+rami-hatoum@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

rami-hatoum added a commit to BeOnAuto/runtime that referenced this pull request Oct 23, 2026
…t use, not when imported (#308)

## What this changes

`@openworkflowspec/sdk` builds an Ajv instance over its schema and
compiles all 171 of its validators the moment it is imported, generating
code from strings, though the runtime never asks it to validate: the
coordination capability checks a document with an interpreting validator
and uses the SDK for its classes and its graph alone. A library that
compiles what nobody asks for, at import, is a defect of the library,
and the fix is the library's, compiling a validator the first time
`validate` is called for a type; it is offered upstream as
`open-workflow-specification/sdk-typescript#333`. Until the library
ships it, the runtime carries it as a dependency patch, byte-identical
to the one Studio carries, named in the workspace and copied into the
container image before its install.

Measured on a quiet machine: importing the SDK took 119 ms of wall time
and 174 ms of CPU as published, 19 ms and 28 ms patched, generating 217
functions as published and none patched; the first workflow save, which
now loads the SDK, takes 82 ms against 182 ms with the SDK as published;
the second save is unchanged. The coordination and tool-servers READMEs
carry the figures with their dates; the minified browser bundle is left
as published.

## Tests

- A child Node process imports the SDK, hydrates a workflow, builds its
graph and writes its Mermaid with code generation from strings
disallowed, and only `validate` throws; without the patch the import
itself fails.
- `pnpm check` passes.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
@JBBianchi
JBBianchi requested a lite review from Copilot October 24, 2026 19:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@lornakelly

Copy link
Copy Markdown
Contributor

LGTM

@JBBianchi JBBianchi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and the detailed explanation!

I reproduced the issue on main: importing dist/index.cjs takes around 430 ms, and fails with an EvalError when running with --disallow-code-generation-from-strings.

The proposed approach looks good overall. In particular:

  • Making the AJV instance lazy is important, since even its constructor generates code.
  • Using a Map avoids issues with special property names.
  • Not caching compilation failures seems reasonable.

I have a few suggestions, mostly around testing and error handling.

1. Add a test to check all validation pointers (requested)

Previously, importing the SDK compiled all 171 validators. This meant that an invalid or missing schema pointer would immediately cause a test failure.

With lazy compilation, we lose that automatic check. Since the tests only validate a handful of types, a broken pointer could go unnoticed until someone actually uses it.

I'd suggest adding a test that checks whether every pointer resolves to a valid schema:

it('should resolve the schema of every type', () => {
  const ajv = new Ajv({ schemas: [workflowSchema], strict: false });
  addFormats(ajv);

  const unresolved = Object.entries(validationPointers)
    .filter(([, pointer]) => !pointer || !ajv.getSchema(pointer))
    .map(([typeName]) => typeName);

  expect(unresolved).toEqual([]);
});

An alternative would be to move the lazy validation helpers into an internal module and test them directly. This would avoid duplicating the AJV configuration and keep those helpers out of the public SDK API.

2. Should compilation errors be treated as validation errors? (question)

Currently, getValidator runs inside withValidationContext. As a result, errors caused by the environment (such as blocked code generation) are converted into DslValidationError.

The problem is that these errors don't necessarily mean the user's workflow is invalid.

For example, an editor catching WorkflowValidationError might tell users their document is incorrect, when the real issue is that their environment doesn't allow code generation.

One possible improvement would be to resolve the validator before entering withValidationContext in validate().

This would make the distinction clearer:

  • Invalid workflow: a WorkflowValidationError.
  • Compilation or environment issue: the original error.

The unknown-type error could still be wrapped as it is today.

Not a blocker, but I think it's worth deciding how these errors should be handled, especially since the new test establishes the current behaviour.

3. Finish configuring AJV before storing it (minor)

In validation.ts (L39–43), ajv is assigned before calling addFormats().

This works today, but I'd suggest completing the initialization first, so we never store a partially configured instance:

const instance = new Ajv({
  schemas: [workflowSchema],
  strict: false,
});

addFormats(instance);
ajv = instance;

4. Simplify the no-codegen test (minor)

In validator-compilation.spec.ts (L107–123), the test checks exact node IDs and the complete Mermaid output.

These details are already covered by other tests, and they aren't really related to the purpose of this test.

I'd keep this test focused on verifying that the SDK works without code generation when validation is disabled.

For example, we could simply check that the relevant operations succeed, rather than asserting their exact formatted output.

Also, if the child process crashes, several tests currently fail with a JSON parsing error. Including run.stderr in the error reported by readOutput would make these failures much easier to debug.

Optional improvements (can be follow-ups)

A few additional ideas that don't need to block this PR:

  • Bundle tests: scripts/validate-tarball.mjs already tests the UMD bundle in a VM. We could disable string-based code generation there and add equivalent checks for CJS/ESM. This would turn the manual verification into an automated regression test.
  • README: Add a short note explaining that validation requires runtime code generation (unsafe-eval under CSP), and that { validate: false } allows using the SDK without it. This is particularly relevant because builders and deserialize enable validation by default.
  • Repeated compilation failures: When CSP blocks code generation, each validation attempt retries compilation, and AJV logs roughly 8 KB of generated code to console.error. Not a blocker, but something to keep in mind.
  • CSP-compatible validation: AJV's standalone validators could eventually allow validation without runtime code generation. That's a larger change and probably deserves a separate discussion.

Overall, the approach makes sense to me. The main thing I'd like to see before merging is the test covering all validation pointers. The error-handling question is worth discussing; the remaining points are relatively minor.

…inter

The helpers that create the AJV instance and compile a type's validator
on first use move from validation.ts into validators.ts, an internal
module the SDK does not export. A new test compiles the validator of
every type in validationPointers through that module, with the SDK's own
AJV configuration, so a pointer that is missing or resolves to no schema
fails the tests, as it did when every validator was compiled at import.

validate() now gets the validator before it enters the validation
context. An invalid workflow still throws a WorkflowValidationError, and
an unknown type still throws the same DslValidationError. An error
raised while compiling a validator, such as the EvalError thrown where
code generation from strings is disallowed, is now thrown unchanged,
since it does not mean that the workflow is invalid.

The AJV instance is stored only once addFormats has configured it.

The test without code generation now checks that deserializing, building
the graph and writing the Mermaid code succeed, not their exact output.
If the child process crashes, the tests report its exit status and its
stderr, quoted line by line so that its stack frames are not taken for
frames of the test's own error.

Signed-off-by: Rami <5664927+rami-hatoum@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rami-hatoum

Copy link
Copy Markdown
Author

Thank you for the careful review, and for reproducing the issue. The new commit addresses the four points.

  1. Validation pointers. I went with the alternative: the lazy helpers now live in an internal module, src/lib/validators.ts, which the SDK does not export, and tests/validation/validators.spec.ts compiles every type in validationPointers through it, with the SDK's own AJV instance, so there is no second configuration to drift. It fails, naming each type and the reason, when a pointer is emptied or points to a path that does not exist.

  2. Compilation errors. Agreed. validate() now gets the validator before entering withValidationContext: an invalid workflow still throws a WorkflowValidationError, an unknown type the same DslValidationError as before (now tested, including toString and __proto__), and a compilation error, such as the EvalError where code generation is disallowed, is thrown as is, which the no-codegen test now expects. One consequence: in that environment the validator is compiled before the lifecycle hooks run, so even a workflow with an unsupported DSL version gets the EvalError.

  3. AJV set-up. The instance is stored only after addFormats() has run.

  4. No-codegen test. It now only checks that deserializing, building the graph and writing the Mermaid code succeed, and readOutput reports the child's exit status and stderr. Testing a crash showed that Vitest reads the stack lines in the child's stderr as frames of the test's own error and fails on their source maps, hiding every result (the existing expect(run.status, run.stderr) did this too), so the stderr is now quoted line by line in both places.

If they would help, I'm happy to follow up with separate pull requests for the README note on runtime code generation and { validate: false }, and for running the bundle checks in scripts/validate-tarball.mjs with code generation disabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants