Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/concepts/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,8 @@ Or use the saved result as a model in another query:

A consumed query-backed model is spliced into the consuming query as stages whose names stay private to it, and one that references itself, directly or through other query-backed models, is rejected.

Joins declared on a saved query-backed model, keyed on its output columns, stay usable wherever the model is consumed; the backing source model's joins are not inherited.

### Variables in model SQL

`{variable}` substitution is not limited to query-level filters — the same mechanism injects caller-supplied values into a model's **raw-SQL (Mode A) surfaces**. This is the primitive for parameterizing hand-written model SQL (for example, representing a Cube `FILTER_PARAMS` cube as an `sql`-mode model whose filter bounds arrive as variables).
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-10-21
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
## Context

The public save path validates and persists joins after populating a query-backed model's output columns. Execution strips those joins from its placeholder, then creates a flat synthetic model from the final StageSchema. Product-hop support can turn the missing edge into a successful uncorrelated filter rather than a refusal.

## Decisions

1. Repair the saved-relation boundary. Preserve only the query-backed model's own declared joins; never inherit its backing source's join graph.
2. Reuse `model_from_stage_schema` to carry optional declared joins for saved final relation names in source, extension and sibling/target paths. Keep StageSchema unchanged. Validate joins against the saved relation's real identity so a legal target named `_stage` is not mistaken for a self-join.
3. Select ModelScope for a materialized saved relation by its identity in the query-backed model map, including relations with incoming reverse edges and no outgoing joins. Ordinary/private query stages retain flat StageSchema binding.
4. Keep demand-based splicing. Merely declaring or reaching an unread query-backed model must not plan it or manufacture a cycle.
5. Reuse the existing logical join-key and extension semantics. Test distinct/named extension joins. An extension join equal to one of its base model's stored joins in all but metadata and pair order is that stored edge, applied once, for every model kind; the runtime overlay and the schema-drift stage graph share this one rule. A join differing in target, name, join type, cardinality or pairs stays a separate edge.

## Validation

Record failing segmented-total tests before the implementation. Execute the repaired paths on SQLite and DuckDB using renamed physical/logical keys. Check root and target consumption, aggregate attribution, extension parity, private-stage scopes and cycles. Run the required unit, lint, type and architecture gates and the documented CI integration selection, then inspect current-head CI and review results.

## Risks

An incomplete root-only fix leaves target/sibling reads broken. Passing saved relationships to ordinary stages breaks stage isolation. Eager traversal of retained edges can introduce false cycles. Preserve those distinctions and use existing fixtures and regression suites.
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
## Why

A saved query-backed model's declared joins disappear when its final stage becomes a consuming relation. A customer-segment filter consequently returns all-customer revenue (150 instead of 120), because the missing relationship is treated as a product hop. The older outbound traversal gap is tracked as DEV-1918; this change fixes its wrong-result consequence reported in GitHub issue #507.

## What Changes

- Retain the saved relation's own joins through query-backed placeholder and relation materialization.
- Bind saved relations as model scopes, keeping ordinary query-stage outputs flat.
- Add executable SQLite/DuckDB tests for filtered totals, traversal and aggregate attribution.
- Preserve existing demand splicing, private stage scopes, cycle errors and extension behavior.
- Treat an extension join repeating a stored join of its base as that edge, for every model kind.

## Capabilities

### Modified Capabilities

- `queries/query-backed-inline`: consuming a saved relation retains its own declared relationships.
- `models/join-traversal`: an extension join repeating a stored edge is that edge.

## Impact

A focused engine/IR fix with regression tests and a clarification in the models guide. No new dependency, persisted schema, public request field or normative architecture change.
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
## ADDED Requirements

### Requirement: An extension join repeating a stored edge is that edge

A query-level model extension's join that equals one of its base model's stored joins in every field except `description`, `meta` and pair order SHALL be that stored edge, applied once, for every model kind; an extension join differing in target, name, join type, cardinality or join pairs SHALL remain a separate edge.

#### Scenario: Repeating a stored join in an extension
- **WHEN** an extension over a table-backed model repeats a stored join and a query reads through it
- **THEN** the query resolves over that single edge instead of failing as an ambiguous hop

#### Scenario: A differing extension join stays parallel
- **WHEN** an extension adds an unnamed join to the same target with different join pairs
- **THEN** a bare hop to that target fails closed as ambiguous
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## ADDED Requirements

### Requirement: Saved relation relationships survive splicing

A stored query-backed model consumed as a relation SHALL retain that model's own declared joins, with keys resolved against its final-stage output columns. Those relationships SHALL remain available when the relation is a source, a join target or an extended source. Ordinary query-stage outputs SHALL remain flat and SHALL NOT inherit joins from their backing source.

#### Scenario: Joined segment filter on a saved relation
- **WHEN** a saved customer-total relation declares a customer-key join and a query filters to gold customers
- **THEN** only the related gold customers contribute, with the same values as an equivalent relationship declared solely on a query extension, on SQLite and DuckDB

#### Scenario: Joined dimensions and aggregate attribution
- **WHEN** a query reads a dimension or aggregate through a saved relation's declared join
- **THEN** the relationship determines the rows and attribution instead of being treated as an unrelated dataset

#### Scenario: A saved relation consumed through another model
- **WHEN** another model joins a saved query-backed relation and a query traverses onward through that relation's own join
- **THEN** the declared relationships remain available with logical key spelling and correct values

#### Scenario: Extending a saved relation
- **WHEN** a query extension adds a distinct named join to a saved relation
- **THEN** both its stored joins and the extension's joins remain available

#### Scenario: Repeating an existing saved relationship
- **WHEN** an extension over a saved query-backed relation repeats an exactly identical declared join
- **THEN** that edge is applied once, without introducing an ambiguous parallel path

#### Scenario: Legal join targets do not collide with temporary identities
- **WHEN** a saved relation declares a join to a legal model named `_stage`
- **THEN** both unjoined and joined consumers execute without a false self-join error

#### Scenario: Ordinary stage isolation
- **WHEN** an ordinary query stage is consumed by a later stage
- **THEN** only its projected output schema is visible and its backing source's joins are not inherited
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
## 1. Plan and reproduce

- [x] 1.1 Refresh current upstream and public issue/PR history; report the wrong-result behavior in #507, distinguishing the known DEV-1918 traversal gap.
- [x] 1.2 Review the relation-boundary design independently, preserving flat ordinary stages and demand splicing without a normative architecture edit.
- [x] 1.3 Record failing SQLite/DuckDB segmented-total regressions before production changes.

## 2. Implement and document

- [x] 2.1 Preserve declared joins through saved-relation placeholder, root/extension and sibling/target materialization.
- [x] 2.2 Verify logical-key traversal, aggregate attribution, extension compatibility, ordinary-stage isolation and cycle behavior.
- [x] 2.3 Clarify saved-relation joins in the user models guide.
- [x] 2.4 Treat an extension join repeating a stored join as that edge for every model kind, in the runtime overlay and the schema-drift stage graph.

## 3. Validate and review

- [x] 3.1 Pass the focused regressions and related query-backed scope/cycle tests.
- [x] 3.2 Run the full documented unit selection, retest environment/timing failures, and pass lint, type and architecture gates.
- [x] 3.3 Independently review the implementation and verify the final tree.

Validation note: the local Python 3.14 unit run reported 31,377 passed, 99 skipped, 7 xfailed and three failures. All three passed isolated retries; the Poetry PATH was corrected for the affected subprocess test, and the timing-sensitive failure also reproduced on unchanged upstream. The documented integration selection passed with 656 passed, 36 skipped and 1 xfailed. CI validates the full suites on Python 3.11 and 3.14.
12 changes: 12 additions & 0 deletions openspec/specs/models/join-traversal/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,3 +323,15 @@ directed cycle SHALL fail closed when every candidate root is blocked by a fanni
#### Scenario: Rootless query across a directed 3-cycle fails closed
- **WHEN** a query without `source_model` selects `x.v`, `y.v` and `z.v` over `x → y → z → x`
- **THEN** it fails with the no-viable-candidate error, and succeeds once `source_model` is given

### Requirement: An extension join repeating a stored edge is that edge

A query-level model extension's join that equals one of its base model's stored joins in every field except `description`, `meta` and pair order SHALL be that stored edge, applied once, for every model kind; an extension join differing in target, name, join type, cardinality or join pairs SHALL remain a separate edge.

#### Scenario: Repeating a stored join in an extension
- **WHEN** an extension over a table-backed model repeats a stored join and a query reads through it
- **THEN** the query resolves over that single edge instead of failing as an ambiguous hop

#### Scenario: A differing extension join stays parallel
- **WHEN** an extension adds an unnamed join to the same target with different join pairs
- **THEN** a bare hop to that target fails closed as ambiguous
32 changes: 32 additions & 0 deletions openspec/specs/queries/query-backed-inline/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,3 +123,35 @@ cycle. A planning failure unrelated to a cycle SHALL keep its own error.
#### Scenario: Direct and transitive cycles
- **WHEN** `A` consumes `A`, or `A` consumes `B` which consumes `A`, and a consumer of `A` executes, `A` runs by name, `A` is saved, or `A`'s column types are requested
- **THEN** each raises the same cycle error naming the path

### Requirement: Saved relation relationships survive splicing

A stored query-backed model consumed as a relation SHALL retain that model's own declared joins, with keys resolved against its final-stage output columns. Those relationships SHALL remain available when the relation is a source, a join target or an extended source. Ordinary query-stage outputs SHALL remain flat and SHALL NOT inherit joins from their backing source.

#### Scenario: Joined segment filter on a saved relation
- **WHEN** a saved customer-total relation declares a customer-key join and a query filters to gold customers
- **THEN** only the related gold customers contribute, with the same values as an equivalent relationship declared solely on a query extension, on SQLite and DuckDB

#### Scenario: Joined dimensions and aggregate attribution
- **WHEN** a query reads a dimension or aggregate through a saved relation's declared join
- **THEN** the relationship determines the rows and attribution instead of being treated as an unrelated dataset

#### Scenario: A saved relation consumed through another model
- **WHEN** another model joins a saved query-backed relation and a query traverses onward through that relation's own join
- **THEN** the declared relationships remain available with logical key spelling and correct values

#### Scenario: Extending a saved relation
- **WHEN** a query extension adds a distinct named join to a saved relation
- **THEN** both its stored joins and the extension's joins remain available

#### Scenario: Repeating an existing saved relationship
- **WHEN** an extension over a saved query-backed relation repeats an exactly identical declared join
- **THEN** that edge is applied once, without introducing an ambiguous parallel path

#### Scenario: Legal join targets do not collide with temporary identities
- **WHEN** a saved relation declares a join to a legal model named `_stage`
- **THEN** both unjoined and joined consumers execute without a false self-join error

#### Scenario: Ordinary stage isolation
- **WHEN** an ordinary query stage is consumed by a later stage
- **THEN** only its projected output schema is visible and its backing source's joins are not inherited
10 changes: 10 additions & 0 deletions slayer/core/join_edges.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"relationship_signature",
"resolve_join_ref",
"resolve_join_refs",
"same_edge",
"without_joins",
]

Expand Down Expand Up @@ -58,6 +59,15 @@ def _pair_set(pairs: list[list[str]]) -> frozenset[tuple[str, str]]:
return frozenset((p[0], p[1]) for p in pairs)


def same_edge(a: ModelJoin, b: ModelJoin) -> bool:
"""Whether two declarations on one model are one edge: equal in all but metadata and pair order."""
return (
(a.target_model, a.name, a.join_type, a.cardinality)
== (b.target_model, b.name, b.join_type, b.cardinality)
and _pair_set(a.join_pairs) == _pair_set(b.join_pairs)
)


def _joins_to(model: SlayerModel, target: str) -> list[ModelJoin]:
return [j for j in model.joins if j.target_model == target]

Expand Down
4 changes: 2 additions & 2 deletions slayer/engine/bundle_builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ async def _split_query_backed(
) -> "Tuple[List[SlayerModel], Dict[str, SlayerModel]]":
"""Split ``models`` into referenced models and query-backed placeholders, closing
over each placeholder's stage bases and each model's definition-named models (and
their components). Placeholders keep their stored form with joins dropped; on-chain
their components). Placeholders keep their stored form, including declared joins; on-chain
names are never loaded."""
referenced: Dict[str, SlayerModel] = {}
query_backed: Dict[str, SlayerModel] = {}
Expand All @@ -279,7 +279,7 @@ async def _split_query_backed(
storage=storage, data_source=data_source, failures=failures,
) if not is_spine(m))
continue
query_backed[model.name] = model.model_copy(update={"joins": []})
query_backed[model.name] = model
pending.extend(await _stage_base_components(
model=model, skip=[*chain, *query_backed], known={*referenced, *query_backed},
storage=storage, data_source=data_source, failures=failures,
Expand Down
4 changes: 4 additions & 0 deletions slayer/engine/plan.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ def _stage_scope_and_bundle_unframed(
if sib is not None and isinstance(src, ModelExtension):
base = model_from_stage_schema(
name=sib, schema=stage_schemas[sib], data_source=data_source,
joins=bundle.saved_joins(sib),
)
overlaid = apply_extension_overlay(base, src)
others = {n: s for n, s in stage_schemas.items() if n != sib}
Expand All @@ -144,12 +145,15 @@ def _stage_scope_and_bundle_unframed(
if isinstance(src, str) and src in stage_schemas:
synth = model_from_stage_schema(
name=src, schema=stage_schemas[src], data_source=data_source,
joins=bundle.saved_joins(src),
)
others = {n: s for n, s in stage_schemas.items() if n != src}
sb = stage_bundle_with_siblings(
bundle=bundle, source_model=synth,
sibling_schemas=others, data_source=data_source,
)
if src in bundle.query_backed:
return ModelScope(source_model=synth), sb
return stage_schemas[src], sb

# 3. Model-scoped: the stage's own resolved source model.
Expand Down
3 changes: 3 additions & 0 deletions slayer/engine/schema_drift.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
from slayer.sql.client import SlayerSQLClient, build_sql_model_trial_query
from slayer.sql.dialects import dialect_for_ds_type
from slayer.sql.engine_factory import EngineCacheKey, _sql_client_cache_key
from slayer.ir.source_bundle import added_joins

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -770,6 +771,8 @@ def _build_stage_graph(
extension_joins = _stage_source_joins(stage)
extension_targets = {j.target_model for j in extension_joins}
base = models_by_name.get(stage_source_name) if stage_source_name else None
if base is not None:
extension_joins = added_joins(base=base, joins=extension_joins)
source = base.model_copy(update={"joins": [*base.joins, *extension_joins]}) if base is not None else None
models = {**models_by_name, source.name: source} if source is not None else models_by_name
roots = [source] if source is not None else [models[t] for t in sorted(extension_targets) if t in models]
Expand Down
Loading
Loading