Skip to content

arazzo-executor: prepare checked workflows before execution - #285

Merged
SVilgelm merged 2 commits into
mainfrom
feat/arazzo-preparation
Sep 8, 2026
Merged

arazzo-executor: prepare checked workflows before execution#285
SVilgelm merged 2 commits into
mainfrom
feat/arazzo-preparation

Conversation

@SVilgelm

@SVilgelm SVilgelm commented Sep 8, 2026

Copy link
Copy Markdown
Member

Add IO-free prepare and required_sources APIs and an immutable, reusable
PreparedWorkflow. Preparation aggregates deterministic diagnostics with field
paths, workflow/step context and byte offsets before a checked run can send a
request. It composes structural validation with static expression/reference,
capability, effective-parameter, operation and dependency checks across the
selected workflow's potential calls and recovery branches.

Prepared runs reuse condition/runtime-expression syntax, interpolation templates,
constant regex/JSONPath programs, endpoints and ordering while keeping inputs,
attempts and outputs independent. A named condition profile and optional
portability warnings make implementation policy explicit. Retained token offsets
keep diagnostics accurate with quoted expression lookalikes and cached ASTs.

The CLI now prepares before execution and prints available partial history on
terminal runtime errors unless quiet. Source discovery avoids loading unrelated
documents for qualified operations while preserving bare operation-ID uniqueness
checks. Existing validation-ignore options remain honored.

This intentionally makes the CLI stricter: a criterion such as
{"condition":"true || $steps.typo.outputs.value"}, a typed criterion without
its required context, or a constant malformed regex fails before requests,
including on a potential recovery branch. Runtime-dependent criterion failures
retain the recovery/reporting behavior from #284. Existing execute,
execute_async, execute_v1_0 and Run::start signatures and lazy validation
remain unchanged; strict preparation is opt-in for library callers.

The unreleased 0.2 API also adds CriterionError::Syntax::offset and removes the
location prefix from that variant's message. Downstream code constructing or
destructuring all fields must adapt; consumers should read the typed byte offset
instead of parsing display text. The README migration notes cover this breaking
change. Preparation uses the runtime's typed missing-reference errors, displays
syntax offsets once, and sorts bracketed diagnostic indices numerically. Cached
condition validation remains per use site, including shared reusable actions.

XPath, AsyncAPI, external workflow calls, non-RFC9535 JSONPath and nested workflow
calls that require their own dependency scheduling are explicit checked-path
capability errors. Input schemas remain available for caller validation; fetching,
cross-document resolution and JSON Schema execution stay in later stages.

Assisted-by: Codex
Signed-off-by: Sergey Vilgelm <sergey@vilgelm.com>
Copilot AI lite review requested due to automatic review settings September 8, 2026 14:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@SVilgelm

SVilgelm commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

Reviewed at 37baf58, based on current main (8e6b84a). cargo fmt --all --check, cargo clippy --workspace --all-features --all-targets -- -D warnings, 9 doctests (including the new PreparedWorkflow example) and cargo nextest run --workspace --all-features (3315 tests) are green. For the new enumset runtime dependency: cargo machete finds nothing unused and cargo deny check reports advisories/bans/licenses/sources ok — the yanked chacha20 warning is pre-existing, via salvo_coreroas-http-validator, and Cargo.lock is untouched here.

The central risk in this design is that a prepared run and a lazy run could disagree, since Compiled is consulted by string key from evaluate, interpolate, simple, apply, place and regex. I checked it rather than assumed it: across eight documents — simple conditions, regex, JSONPath, interpolated parameters, selector outputs, bare-value truthiness, cross-step reads and a failing step — execute and plan.execute produced identical reports, and that held for goto-workflow, retry-workflow and workflow-call steps too. Sharing order_steps between the two paths rather than reimplementing the ordering is the right call. Site::component keeping the invocation workflow while re-pointing path at the component is also right: a reusable action whose criterion reads $steps.only_in_a prepares cleanly for the workflow that declares that step and is rejected for the one that does not.

Confirmed Issues

1. Expression references are reported with goto-target errors — Medium

prepare.rs:1073-1086 reports a $steps.… / $workflows.… reference that names nothing with ExecutionError::UnknownStep / UnknownWorkflow. Those variants are for action targets — report.rs:250 documents the field as "The workflow the goto was in", and report.rs:449 pins the wording. So the PR's headline example produces:

#.workflows[0].steps[0].successCriteria[0].condition at byte 8: workflow `w` has no step `typo` to go to

for {"condition": "true || $steps.typo.outputs.value"}. Nothing goes anywhere; the condition reads a step that does not exist. The same defect at runtime, through expression::check_reference, already says the right thing:

`$steps.typo.outputs.value` names step `typo`, which this workflow has not got

It matters most where preparation is most valuable — the shared-component case reports workflow bhas no steponly_in_a to go to for a criterion in #.components.successActions.shared.

Fix: emit ExpressionError::Missing from Compiler::reference, matching the runtime wording for the same defect, and keep UnknownStep/UnknownWorkflow in Compiler::action_target (prepare.rs:845), where they are accurate.

2. The byte offset is printed twice — Low

prepare.rs:134-140 appends at byte {offset} to every diagnostic that has one, but both syntax error types already carry the offset in their own Display. The result, for the two most common preparation errors:

…successCriteria[0].condition at byte 14: `$statusCode ==` is not a valid condition: at byte 14: expected a value
…outputs.v at byte 9: `$steps.a.bogus` is not a valid runtime expression at byte 9: unknown exchange field

Fix: add the location prefix only for issues that do not describe their own position — PreparationIssue::Model today — or drop it from the wrapped messages.

Possible Risks

3. The condition offset is recovered by parsing an error message — Medium, and this release is the window

prepare.rs:499-505:

ExecutionError::Criterion(CriterionError::Syntax { message, .. }) => message
    .strip_prefix("at byte ")
    .and_then(|message| message.split_once(':'))
    .and_then(|(offset, _)| offset.parse().ok()),

This depends on criterion::syntax (criterion.rs:388) formatting "at byte {offset}: {message}". The coupling is invisible from criterion.rs, nothing fails loudly if that format changes, and the failure mode is silent: every condition diagnostic loses its offset. The comment acknowledges it ("preserve that variant's field shape for downstream users"), but the constraint it is preserving does not apply yet — roas-arazzo-executor is still tagged at v0.1.2, and 0.2.0 is unreleased, so adding offset: usize to CriterionError::Syntax costs nothing today and is breaking after release. Same situation as #[non_exhaustive] in #282. Doing it also removes half of finding 2.

4. The cached branch of simple() skips check_reference — Low

criterion.rs:395-401 returns truthy(tree.evaluate(scope)) directly when the condition is in Compiled, while the uncached path first runs expression::check_reference over every operand including short-circuited ones. That is sound today, because Compiler::criteria visits reference at every use site — I confirmed the per-site behaviour with a component shared by two workflows — so a plan cannot contain a condition with an undeclared reference. But nothing in the code states that invariant, and it is the only thing keeping the two paths equivalent; a future cache on the lazy path, or a Compiled populated from anywhere but prepare, would drop the check silently. A comment naming the invariant plus a test asserting that lazy and prepared runs agree on a short-circuited undeclared reference would pin it.

Nice-to-Have Improvements

5. Diagnostics sort lexicographically, not by location — Low

prepare.rs:371-379 sorts on diagnostic.path as a string, so a workflow with ten or more steps reads:

#.workflows[0].steps[0].successCriteria[0].condition
#.workflows[0].steps[10].successCriteria[0].condition
#.workflows[0].steps[11].successCriteria[0].condition
#.workflows[0].steps[1].successCriteria[0].condition
…

PreparationError is documented as "sorted deterministically by location" — it is deterministic, but this is not location order. Sorting on a key that keeps the indices numeric would match the doc.

6. Instrumentation indices are unnamed — Low

instrumentation::compiled(0..3) is called from runtime_syntax.rs:43, criterion.rs:413, criterion.rs:151 and select.rs:187. At the call sites the number says nothing; the test at prepare.rs:59 is what defines them. Named constants or a small enum would make each call site self-explanatory.

7. Module layout and docs — Low

#[cfg(test)] mod tests sits at prepare.rs:57-107, between PreparationIssue and PreparationDiagnostic, where every other module in the crate keeps it at the end. And PreparationError::diagnostics, PreparationDiagnostic::issue and PreparedWorkflow::condition_profile are the only new public items without doc comments, next to neighbours that have them.

— Reviewed by Claude Opus 5

Assisted-by: Codex
Signed-off-by: Sergey Vilgelm <sergey@vilgelm.com>
Copilot AI review requested due to automatic review settings September 8, 2026 14:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@SVilgelm

SVilgelm commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

Re-reviewed at 5bf39b0. All seven findings are addressed, each with a test. cargo fmt --all --check, cargo clippy --workspace --all-features --all-targets -- -D warnings, 9 doctests and cargo nextest run --workspace --all-features (3320 tests) are green. Crate coverage is 98.50% lines / 96.93% regions, with the new prepare.rs at 99.11% lines / 98.88% regions.

No major issues detected.

1. Expression references now use expression::undeclared_step / undeclared_workflow, shared with check_reference, so preparation and runtime say the same thing for the same defect:

…successCriteria[0].condition at byte 8: `$steps.typo.outputs.value` names step `typo`, which this workflow has not got

and the goto wording is kept where it is accurate rather than flattened away — …onSuccess[0].stepId: workflow whas no stepnope to go to. The component case reads correctly too: #.components.successActions.shared.criteria[0].condition at byte 0: $steps.only_in_a.outputs.vnames steponly_in_a, which this workflow has not got, for workflow b only.

2 and 3. Offsets are now a real CriterionError::Syntax { offset } field, the message-parsing in Compiler::error is gone, and PreparationDiagnostic's Display prints the location once for both syntax shapes:

…condition at byte 14: `$statusCode ==` is not a valid condition: expected a value
…outputs.v at byte 9: `$steps.a.bogus` is not a valid runtime expression: unknown exchange field

The nested case is the one I'd have expected to regress, and it does not — 1 == $steps.a.bogus reports byte 14, which is where bogus starts in the whole condition (operand at 5, inner offset 9), with the runtime-expression reason nested and no second location:

…condition at byte 14: `1 == $steps.a.bogus` is not a valid condition: `$steps.a.bogus` is not a valid runtime expression: unknown exchange field

The README migration section covers the field change, including that the standalone Display still carries the location while a diagnostic prints it once.

4. The cached-simple() invariant is now stated at the call site and on Compiled, and pinned by shared_cached_conditions_are_checked_in_each_workflow_scope. I re-ran the agreement matrix independently: eight documents — simple conditions, regex, JSONPath, interpolated parameters, selector outputs, bare-value truthiness, a failing step, and a short-circuited reference to a declared-but-not-yet-run step — all produce identical reports through execute and plan.execute.

5. path_order sorts bracketed indices numerically; twelve steps now come back steps[0]steps[11] in order. path_order_only_treats_representable_bracketed_digits_as_indices covers the fallbacks ([name], [], [+1], unterminated, and an index too large for usize), which is the right set — the full path string remains in the sort tuple, so ordering stays total when a bracket is not an index.

6 and 7. The instrumentation hooks now name their parser (Parser::RuntimeExpression and friends) with a named Counts struct instead of a four-element array, instrumentation and mod tests moved to the end of prepare.rs, and PreparationError::diagnostics, PreparationDiagnostic::issue, is_error and condition_profile all carry doc comments now.

— Reviewed by Claude Opus 5

@SVilgelm
SVilgelm merged commit dbdbcbb into main Sep 8, 2026
126 checks passed
@SVilgelm
SVilgelm deleted the feat/arazzo-preparation branch September 8, 2026 15:04
SVilgelm added a commit that referenced this pull request Sep 10, 2026
)

Add the optional `source-graph` executor feature, following the checked
preparation
work in #285. Complete documents now retain opaque registry handles,
retrieval
locations, resolved Arazzo `$self` identities, effective reference bases
and written
versions. Supplied Arazzo documents are fully parsed and indexed before
resolution;
aliases and source/API-base overrides are scoped to the owning document.

Source traversal uses caller-configured `roas::Loader` fetchers, with
shared
documents, explicit back edges, separate document/depth budgets,
root-source selection,
and located errors that preserve readable parts of a graph. Later
identity discovery
can repair an earlier unresolved reference, including a newly discovered
shorter
path. A qualified operation may prepare with an unrelated source
missing; a bare
operation ID still requires enough sources to prove uniqueness.

The loader adds unchanged-document/retrieval-metadata APIs with default
methods for
existing sync/async fetchers. HTTP fetchers expose final redirect URLs
and retain
caller redirect/timeout policy. YAML sniffing respects explicit content
types and,
when absent or generic, either the requested or final URL's extension.
Thus
`/source.yaml` redirecting to `/blob` retains legacy parsing behavior.
Legacy reference readers retain requested-URI rewriting and store only
one full
document plus changed-reference strings. A raw view is materialized only
on demand,
without another fetch. Shared-document APIs let the loader, graph and
cloned Options
retain the same immutable raw value; both full views are retained only
if both are
requested. All loader types are also exported at the roas crate root.
`Options::source` remains compatible, and explicit source/base overrides
take precedence.

The CLI keeps selected-workflow source discovery by default. New flags
expose
`--source-document`, `--load-all-sources`, `--source-max-documents`,
`--source-max-depth`, and `--allow-source-retrieval-aliases`; none
enables IO beyond
the existing `--load` policy. Canonical Arazzo identities are strict by
default;
retrieval aliases are an explicit compatibility extension. Fetched
unrelated failures
are reported without unnecessarily blocking a qualified checked run.
Each diagnostic
is rendered once with its owning document identity, location and source
alias, without
opaque registry IDs; quiet mode suppresses optional warnings but retains
failure details.
Relative sources under a non-hierarchical `$self` name the two remedies:
an absolute
source URL or a hierarchical `$self`.

`crates/roas-cli/tests/fixtures/source-graph/root.json` links JSON/YAML
branches
sharing `shared.yaml` and `api.json`, with `shared.yaml` pointing back
to the root.
For identity resolution, a document retrieved from
`https://example.test/cache/child.json`
with `"$self":"../identity/child.json"` is identified by
`https://example.test/identity/child.json`, and resolves its relative
sources there.

Recognized families are Arazzo 1.0/1.1, OpenAPI 2.0/3.0/3.1/3.2, and
AsyncAPI
2.6/3.0/3.1. API documents retain complete raw values with model-checked
versions,
not structural/schema validation. Broker execution, external workflow
calls,
referenced OpenAPI Path Items and relative API-server resolution remain
deferred.

The additive loader/fetcher APIs bump roas to 0.20.1 and
roas-http-fetcher to 0.2.5;
the executor remains on unreleased 0.2.0. Those dependency minimums
ensure downstream
builds receive the metadata APIs. Consumers inherit the workspace
dependency versions
while explicitly preserving their existing feature selections. No
existing execution
signatures are changed.

---------

Signed-off-by: Sergey Vilgelm <sergey@vilgelm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants