Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
e480d24
Plan the dev-2101 Mode-B grammar OpenSpec change
ZmeiGorynych Oct 21, 2026
abe10ba
Add the failing test suite for the Mode-B SQL-compatible grammar
ZmeiGorynych Oct 21, 2026
9188602
Merge branch 'origin/main' into egor/dev-2101-mode-b-as-a-sql-compati…
ZmeiGorynych Oct 21, 2026
3a271ba
Move the Mode-B parser to slayer/core/syntax.py and the transform tab…
ZmeiGorynych Oct 21, 2026
22b6b54
Mode-B grammar: pyparsing parser, SQL forms in every position, casts,…
ZmeiGorynych Oct 21, 2026
1737b01
NULL ordering, variables as literals, stored-text migration, one impo…
ZmeiGorynych Oct 21, 2026
665fd51
Docs and agent surfaces for the Mode-B SQL grammar: operators, FILTER…
ZmeiGorynych Oct 21, 2026
9adce32
Retarget kwarg-in-scalar tests: '=' inside a scalar call is a comparison
ZmeiGorynych Oct 21, 2026
5701653
Hint how name=value reads when it lands in the wrong call kind
ZmeiGorynych Oct 21, 2026
b4dd784
Grouped ORDER BY over an off-grain row expression sorts by its group …
ZmeiGorynych Oct 21, 2026
ba7ac5b
Order expressions resolve query measure aliases, as filters do
ZmeiGorynych Oct 21, 2026
02a76d1
Align remaining tests with the Mode-B grammar spec
ZmeiGorynych Oct 21, 2026
d8ad3f2
Mode-B grammar on pyparsing infix_notation, packrat off
ZmeiGorynych Oct 22, 2026
7370fa1
Postgres integration tests follow the cast spec; % scenarios on an in…
ZmeiGorynych Oct 22, 2026
a1992eb
Exact decimal literals, overflow-safe SQLite TRY_CAST, Sonar complexi…
ZmeiGorynych Oct 22, 2026
913808e
Expanded null-safe equality is two-valued, so IS DISTINCT FROM keeps …
ZmeiGorynych Oct 22, 2026
2c90453
Archive the Mode-B SQL grammar OpenSpec change
ZmeiGorynych Oct 22, 2026
322a892
Usage-reporting disable-race test gates the send on an explicit hold,…
ZmeiGorynych Oct 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
360 changes: 0 additions & 360 deletions .basedpyright/baseline.json

Large diffs are not rendered by default.

14 changes: 9 additions & 5 deletions architecture/core.arc42.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
family, and the typed error/warning vocabulary. It is the bottom layer: it is
meant to import no other SLayer node; the remaining crossings are the
grandfathered child-level `#legacy` doors (`core.query`/`core.models` →
`engine.syntax`, `sql.*`, `storage.migrations`), slated to die (system
`sql.*`, `storage.migrations`), slated to die (system
principle 2).

## 2. Building blocks
Expand All @@ -20,14 +20,16 @@ flowchart TD
subgraph core["Core domain models"]
core__query["Query"]
core__models["Models"]
core__syntax["Syntax"]
end
subgraph sql["SQL generation"]
sql__dialects["Dialects"]
sql__sql_predicate["SQL predicate"]
sql__window_detect["Window detect"]
end
subgraph engine["Query engine"]
engine__syntax["Syntax"]
engine__binding["Binding"]
engine__bind_inputs["Bind inputs"]
end
ir["Intermediate representation"]
subgraph storage["Storage backends"]
Expand All @@ -41,7 +43,6 @@ flowchart TD
usage_reporting__features["Features"]
end
surfaces("User-facing surfaces")
core__query -.-> engine__syntax
core__models -.-> sql__dialects
core__models -.-> sql__sql_predicate
core__models -.-> sql__window_detect
Expand All @@ -50,6 +51,9 @@ flowchart TD
core__query -.-> storage__migrations
core__models -.-> core__query
core__query --> core__models
core__query --> core__syntax
engine__binding --> core__syntax
engine__bind_inputs --> core__syntax
engine --> core
importers --> core
ir --> core
Expand All @@ -61,14 +65,14 @@ flowchart TD
surfaces --> core
usage_reporting__features --> core
classDef leaf fill:none;
class core__query,core__models,sql__dialects,sql__sql_predicate,sql__window_detect,engine__syntax,ir,storage__migrations,importers,search,memories,protocols,usage_reporting__features,surfaces leaf;
class core__query,core__models,core__syntax,sql__dialects,sql__sql_predicate,sql__window_detect,engine__binding,engine__bind_inputs,ir,storage__migrations,importers,search,memories,protocols,usage_reporting__features,surfaces leaf;
```
*Dashed arrows: legacy edges slated to die.*
<!-- /likec4:core_focus -->

Identity: `keys.py`; scopes:
`scope.py`; errors/warnings: `errors.py`, `warnings.py`; Pydantic domain
models alongside.
models alongside; the Mode B grammar: `syntax.py`.

## 3. Principles

Expand Down
23 changes: 10 additions & 13 deletions architecture/engine.arc42.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ flowchart TD
subgraph core["Core domain models"]
core__query["Query"]
core__models["Models"]
core__syntax["Syntax"]
end
subgraph sql["SQL generation"]
sql__render["Render"]
Expand All @@ -26,7 +27,6 @@ flowchart TD
sql__window_detect["Window detect"]
end
subgraph engine["Query engine"]
engine__syntax["Syntax"]
engine__binding["Binding"]
engine__bind_inputs["Bind inputs"]
engine__elaborate_env["Checker"]
Expand All @@ -38,7 +38,6 @@ flowchart TD
subgraph storage["Storage backends"]
storage__migrations["Migrations"]
end
core__query -.-> engine__syntax
core__models -.-> sql__dialects
core__models -.-> sql__sql_predicate
core__models -.-> sql__window_detect
Expand All @@ -47,12 +46,13 @@ flowchart TD
core__query -.-> storage__migrations
core__models -.-> core__query
core__query --> core__models
core__query --> core__syntax
sql__render --> sql__dialects
sql__sql_predicate --> sql__window_detect
engine__binding --> engine__syntax
engine__binding --> core__syntax
engine__bind_inputs --> engine__binding
engine__bind_inputs --> engine__elaborate_env
engine__bind_inputs --> engine__syntax
engine__bind_inputs --> core__syntax
engine__compile --> engine__elaborate
engine__compile --> engine__elaborate_env
engine__elaborate --> engine__bind_inputs
Expand All @@ -70,7 +70,7 @@ flowchart TD
storage --> engine
storage --> sql
classDef leaf fill:none;
class core__query,core__models,sql__render,sql__dialects,sql__sql_predicate,sql__window_detect,engine__syntax,engine__binding,engine__bind_inputs,engine__elaborate_env,engine__elaborate,engine__compile,engine__plan,ir,storage__migrations leaf;
class core__query,core__models,core__syntax,sql__render,sql__dialects,sql__sql_predicate,sql__window_detect,engine__binding,engine__bind_inputs,engine__elaborate_env,engine__elaborate,engine__compile,engine__plan,ir,storage__migrations leaf;
```
*Dashed arrows: legacy edges slated to die.*
<!-- /likec4:query_pipeline -->
Expand All @@ -83,31 +83,28 @@ only permitted arrows among them (DEV-1897 child-level model-truth):
flowchart TD
%% engine_focus: Engine seam
subgraph engine["Query engine"]
engine__syntax["Syntax"]
engine__binding["Binding"]
engine__bind_inputs["Bind inputs"]
engine__elaborate_env["Checker"]
engine__elaborate["Elaborate"]
engine__compile["Compile"]
engine__plan["Plan"]
end
engine__binding --> engine__syntax
engine__bind_inputs --> engine__binding
engine__bind_inputs --> engine__elaborate_env
engine__bind_inputs --> engine__syntax
engine__compile --> engine__elaborate
engine__compile --> engine__elaborate_env
engine__elaborate --> engine__bind_inputs
engine__elaborate --> engine__elaborate_env
engine__plan --> engine__compile
engine__plan --> engine__elaborate
classDef leaf fill:none;
class engine__syntax,engine__binding,engine__bind_inputs,engine__elaborate_env,engine__elaborate,engine__compile,engine__plan leaf;
class engine__binding,engine__bind_inputs,engine__elaborate_env,engine__elaborate,engine__compile,engine__plan leaf;
```
<!-- /likec4:engine_focus -->

Stages:
`normalization.py` → `syntax.py` (parse) → `binding.py` + `bind_inputs.py`
`normalization.py` → `core/syntax.py` (parse) → `binding.py` + `bind_inputs.py`
(bind) → `elaborate.py` / `elaborate_env.py` (type) → `compile/` (plan) →
`slayer/ir/planned.py` (the typed hand-off to `sql`), fed by
`bundle_builder.py` (builds the `ir` source bundle); `plan.py` composes
Expand Down Expand Up @@ -155,11 +152,11 @@ elaborate→compile and `query_engine.py` orchestrates.
unchanged; normalization applies to queries at execute time only. The one
exception: a saved query stores each stage's inferred population as its
`source_model`. [review]
9. **The term-interface boundary**: `syntax`, `binding`, `bind_inputs`,
9. **The term-interface boundary**: `binding`, `bind_inputs`,
`elaborate`, `elaborate_env` (THE checker), `compile` and `plan` are
declared children with only the modeled arrows among them — the compiler
consults the checker, never syntax or binding; every user-facing algebra
type error raises in the checker, as a `QueryTypeError`.
consults the checker, never `core.syntax` or binding; every user-facing
algebra type error raises in the checker, as a `QueryTypeError`.
[enforced: arch_check:model-truth]
[enforced: test:tests/test_dev1871_raise_parity.py]
10. **Dependencies are closures**: every predicate that classifies a column
Expand Down
2 changes: 1 addition & 1 deletion architecture/index.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ python:

# Count of `#legacy` arrows in the model; arch_check fails if the model's count
# differs. Only ever lower this — every legacy arrow is a door slated to die.
legacy_arrows: {baseline: 8}
legacy_arrows: {baseline: 7}

# Fail-closed deferral-site count (tests/test_law_guard_ratchet.py). Only ever lower.
x-guards: {baseline: 0}
Expand Down
8 changes: 4 additions & 4 deletions architecture/model.c4
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ model {
}
query = node 'Query'
models = node 'Models'
syntax = node 'Syntax'
}
sql = node 'SQL generation' {
metadata {
Expand All @@ -40,7 +41,6 @@ model {
package 'slayer.engine'
arc42 'architecture/engine.arc42.md'
}
syntax = node 'Syntax'
binding = node 'Binding'
bind_inputs = node 'Bind inputs'
elaborate_env = node 'Checker'
Expand Down Expand Up @@ -98,7 +98,6 @@ model {
}
}

core.query -> engine.syntax #legacy
core.models -> sql.dialects #legacy
core.models -> sql.sql_predicate #legacy
core.models -> sql.window_detect #legacy
Expand All @@ -107,12 +106,13 @@ model {
core.query -> storage.migrations #legacy
core.models -> core.query #legacy
core.query -> core.models
core.query -> core.syntax
sql.render -> sql.dialects
sql.sql_predicate -> sql.window_detect
engine.binding -> engine.syntax
engine.binding -> core.syntax
engine.bind_inputs -> engine.binding
engine.bind_inputs -> engine.elaborate_env
engine.bind_inputs -> engine.syntax
engine.bind_inputs -> core.syntax
engine.compile -> engine.elaborate
engine.compile -> engine.elaborate_env
engine.elaborate -> engine.bind_inputs
Expand Down
8 changes: 4 additions & 4 deletions architecture/system.arc42.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ flowchart TD
protocols("BI wire protocols")
usage_reporting["Usage reporting"]
surfaces("User-facing surfaces")
core -.-> engine
core -.-> sql
core -.-> storage
engine --> core
Expand Down Expand Up @@ -91,7 +90,7 @@ All code, old and new, MUST obey these.
[enforced: arch_check:model-truth]
2. **`core` imports no other SLayer node.** The remaining crossings are the
grandfathered child-level `#legacy` doors (`core.query`/`core.models` →
`engine.syntax`, `sql.*`, `storage.migrations`), slated to die.
`sql.*`, `storage.migrations`), slated to die.
[enforced: arch_check:model-truth]
3. **Every top-level `slayer.*` package/module belongs to exactly one node.**
A new top-level package must be claimed in a node's model metadata in the same change.
Expand Down Expand Up @@ -125,8 +124,9 @@ All code, old and new, MUST obey these.
[enforced: test:tests/test_dev1743_resolution.py]
[enforced: test:tests/test_dev1954_canonical_paths.py]
10. **Two expression layers**: Mode A free SQL (`Column.sql`, model `filters`)
vs Mode B Python-AST DSL (formulas, query fields, scalar allowlist only) —
one canonical scalar allowlist, extended never forked. [review]
vs Mode B, a SQL-compatible grammar (formulas, query fields) — one
canonical function library, extended never forked. [review] Every Mode B
construct means the same on every dialect. [target: DEV-2120]
11. **Versioned persistence**: models/queries/datasource configs carry
`version`; migrations run automatically on load. [review]
12. **Pydantic v2 for all models; never dataclasses.** [review]
Expand Down
20 changes: 8 additions & 12 deletions docs/concepts/formulas.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ new derived quantity owned by the query author. Global validation still
applies: the aggregation name must be known, and numeric-only aggregations
reject a confidently non-numeric expression (`sum(lower(name))`).

### Filtered aggregation (`FILTER (WHERE …)`)

Any aggregation takes a SQL `FILTER (WHERE c)` clause with a row-level condition `c` and aggregates only the rows passing it, every row-level input included (`count(*) FILTER (WHERE status = 'paid')`, `last(amount) FILTER (WHERE amount > 0)` picks the newest passing row).

### Windowed aggregations

Every aggregation — built-in or custom — accepts an optional `window` parameter for
Expand Down Expand Up @@ -265,7 +269,8 @@ When a measure is renamed via `name`, query filters and ORDER BY entries in the
| `-` | `"sum(revenue) - sum(cost)"` | `SUM(revenue) - SUM(cost)` |
| `*` | `"avg(price) * sum(quantity)"` | `AVG(price) * SUM(quantity)` |
| `/` | `"sum(revenue) / count(*)"` | `SUM(revenue) / NULLIF(COUNT(*), 0)` |
| `**` | `"sum(value) ** 2"` | `SUM(value) ** 2` |
| `**` | `"sum(value) ** 2"` | `POWER(SUM(value), 2)` |
| `%` | `"sum(quantity) % 12"` | `SUM(quantity) % 12` |

Parentheses work as expected: `"(sum(revenue) - sum(cost)) / count(*)"`. Division by zero is NULL on every backend.

Expand Down Expand Up @@ -449,7 +454,7 @@ Multiple partition columns: `partition_by=[region, channel]`. Cross-model dotted

The rank family's `partition_by=` may name a computed dimension, as an aggregation's can (one that itself wraps an aggregation is for now accepted only in filters and order). Each key must be a member of the transform's operand grain — the union of its inner aggregates' grains (an ungrained inner contributes the query's dimensions) — so `rank(sum(revenue, partition_by=[city, region]), partition_by=product, direction='desc')` is an error naming the remedy.

> **Note:** SLayer's formula parser is Python-AST-based and rejects raw `OVER (...)` SQL in `ModelMeasure.formula` and filter strings. Use the rank-family transforms (`rank`, `percent_rank`, `dense_rank`, `ntile`) for ranking instead of `row_number() over (...) <= N`. If you need a non-standard window expression, define it on a `Column.sql` (e.g., `{"name": "rn", "sql": "row_number() over (order by mass desc)", "type": "NUMBER"}`) and filter on the column — SLayer auto-promotes the predicate to a post-aggregation outer `WHERE`.
> **Note:** SLayer's formula grammar rejects raw `OVER (...)` SQL in `ModelMeasure.formula` and filter strings. Use the rank-family transforms (`rank`, `percent_rank`, `dense_rank`, `ntile`) for ranking instead of `row_number() over (...) <= N`. If you need a non-standard window expression, define it on a `Column.sql` (e.g., `{"name": "rn", "sql": "row_number() over (order by mass desc)", "type": "NUMBER"}`) and filter on the column — SLayer auto-promotes the predicate to a post-aggregation outer `WHERE`.

### First and Last Functions

Expand Down Expand Up @@ -543,13 +548,4 @@ Any formula, filter, or field expression can branch with SQL `CASE`:

## Parsing Internals

Both field and filter formulas are parsed by `slayer/core/formula.py` using Python's `ast` module.

**Field formulas** are classified into:

- **AggregatedMeasureRef** — an aggregation over a measure (`"sum(revenue)"`, `"count(*)"`)
- **ArithmeticField** — arithmetic on aggregated measures (`"sum(revenue) / count(*)"`)
- **TransformField** — function call, possibly nested (`"cumsum(sum(revenue))"`)
- **MixedArithmeticField** — arithmetic containing function calls. Covers both transform calls (`"cumsum(sum(revenue)) / count(*)"`) and non-transform SQL function calls wrapping aggregated refs, e.g. `"count(*) / nullif(max(revenue), 0)"` or `"coalesce(sum(revenue), 0) + avg(amount)"`. Aggregated refs nested inside non-transform calls are resolved as their own measure aliases; the call passes through to emitted SQL unchanged.

The query engine binds and expands field formulas into ordered planned stages, and the SQL generator translates them into stacked CTEs.
Every Mode-B field is parsed by one SQL-compatible grammar (`slayer/core/syntax.py`) into an expression tree that the query engine binds, checks and expands into planned stages, and the SQL generator translates into stacked CTEs.
Loading
Loading