Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c00338e
Merge branch 'egor/dev-2101-mode-b-as-a-sql-compatible-grammar-custom…
ZmeiGorynych Oct 21, 2026
3892d2d
Preserve declared joins on saved query-backed models
mattfaltyn Oct 21, 2026
45cacec
Record query-backed join validation results
mattfaltyn Oct 21, 2026
96ef3da
Merge branch 'egor/dev-2101-mode-b-as-a-sql-compatible-grammar-custom…
ZmeiGorynych Oct 22, 2026
67e7fcb
Plan the dev-2120 portable expression library OpenSpec change
ZmeiGorynych Oct 22, 2026
f88cd80
Treat a repeated stored join in any extension as that edge
ZmeiGorynych Oct 22, 2026
e6be7f7
Specify extension-join reuse of stored edges
ZmeiGorynych Oct 22, 2026
57e6a8b
Skip the DuckDB httpfs notebook when the CDN file changes mid-read
ZmeiGorynych Oct 22, 2026
87c635b
Failing tests for the dev-2120 portable expression library; digit-and…
ZmeiGorynych Oct 22, 2026
a728f77
Merge branch 'origin/egor/dev-2101-mode-b-as-a-sql-compatible-grammar…
ZmeiGorynych Oct 22, 2026
535efd3
Archive fix-query-backed-declared-joins OpenSpec change
ZmeiGorynych Oct 22, 2026
54beea3
Merge pull request #508 from mattfaltyn/fix/query-backed-declared-joins
ZmeiGorynych Oct 22, 2026
2717651
chore: bump version to 1.4.1
ZmeiGorynych Oct 22, 2026
516539d
Merge pull request #510 from MotleyAI/egor/bump-version-1.4.1
ZmeiGorynych Oct 22, 2026
1e7fe26
Merge branch 'origin/main' into egor/dev-2120-portable-expression-lib…
ZmeiGorynych Oct 22, 2026
429a8e2
Portable Mode-B library: strings, math, regex, JSON, TEXT-vs-number c…
ZmeiGorynych Oct 22, 2026
2e76ee6
A regex group that did not take part extracts as '' on every dialect
ZmeiGorynych Oct 22, 2026
2047618
Tests pinning superseded semantics follow the portable library; drop …
ZmeiGorynych Oct 22, 2026
cc626b2
date_trunc time-dimension test compares rows regardless of DuckDB's G…
ZmeiGorynych Oct 22, 2026
a2619f9
Principle 10's one-meaning clause is enforced by the library matrix
ZmeiGorynych Oct 22, 2026
704eb3b
SQLite library functions register as deterministic; conventions clean…
ZmeiGorynych Oct 22, 2026
c8e28c9
Merge branch 'origin/egor/dev-2101-mode-b-as-a-sql-compatible-grammar…
ZmeiGorynych Oct 22, 2026
0f599ae
chr(0) is rejected at plan time like every other out-of-domain literal
ZmeiGorynych Oct 22, 2026
c30fb45
One-argument log is base 10, trim strips spaces only, and text slots …
ZmeiGorynych Oct 23, 2026
d71c9f8
BigQuery timestamps are DATETIME, text matching ignores collation, Sn…
ZmeiGorynych Oct 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,9 @@ jobs:
# `secrets.SNOWFLAKE_CONNECTIONS_TOML`.
env:
SLAYER_SNOWFLAKE_CONNECTION: slayer_test
run: poetry run pytest tests/integration/test_integration_snowflake.py -v -m integration --timeout=300
# The secret is present (gate above), so an unreachable Snowflake fails rather than skips.
SLAYER_SNOWFLAKE_REQUIRED: "1"
run: poetry run pytest tests/integration/test_integration_snowflake.py -v -rs -m integration --timeout=300

- name: Wipe Snowflake credentials
if: always()
Expand Down
2 changes: 1 addition & 1 deletion architecture/system.arc42.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ All code, old and new, MUST obey these.
10. **Two expression layers**: Mode A free SQL (`Column.sql`, model `filters`)
vs Mode B, a SQL-compatible grammar (formulas, query fields) — one
canonical function library, extended never forked. [review] Every Mode B
construct means the same on every dialect. [target: DEV-2120]
construct means the same on every dialect. [enforced: test:tests/test_dev2120_library_matrix.py]
11. **Versioned persistence**: models/queries/datasource configs carry
`version`; migrations run automatically on load. [review]
12. **Pydantic v2 for all models; never dataclasses.** [review]
Expand Down
2 changes: 2 additions & 0 deletions docs/concepts/formulas.md
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,8 @@ Inside `Column.sql`, `ModelMeasure.formula`, or any `Aggregation.formula`, you c
| `round(x[, ndigits])` | 1–2 | Round to `ndigits` decimal places (default 0). `ndigits` must be an integer literal. |
| `abs(x)` | 1 | Absolute value. |

Query and measure formulas also take `div`, `pi`, `sin` … `atan2` and `cbrt` from the [portable function library](references.md#the-portable-function-library).

Unlike the other scalar functions above — which pass through only when embedded in a larger `Column.sql` or arithmetic expression — `round` and `abs` are also valid as the **top-level** form of a query measure or `ModelMeasure.formula`:

```python
Expand Down
2 changes: 2 additions & 0 deletions docs/concepts/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,8 @@ Or use the saved result as a model in another query:

A consumed query-backed model is spliced into the consuming query as stages whose names stay private to it, and one that references itself, directly or through other query-backed models, is rejected.

Joins declared on a saved query-backed model, keyed on its output columns, stay usable wherever the model is consumed; the backing source model's joins are not inherited.

### Variables in model SQL

`{variable}` substitution is not limited to query-level filters — the same mechanism injects caller-supplied values into a model's **raw-SQL (Mode A) surfaces**. This is the primitive for parameterizing hand-written model SQL (for example, representing a Cube `FILTER_PARAMS` cube as an `sql`-mode model whose filter bounds arrive as variables).
Expand Down
30 changes: 13 additions & 17 deletions docs/concepts/queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -357,16 +357,13 @@ by.

### Scalar Functions in Filters

Filters in `SlayerQuery.filters` accept the closed Mode-B scalar
allowlist: string hygiene (`lower`, `upper`, `trim`, `ltrim`,
`rtrim`, `replace`, `substr`, `substring`, `instr`, `length`, `concat`),
null handling (`coalesce`, `nullif`, `ifnull`), math (`round`, `abs`,
`ceil`, `floor`, `sign`, `trunc`, `mod`, `log10`, …), scalar min/max
(`greatest`, `least`), and [date functions](references.md#date-and-time-functions)
(`date_part`, `date_diff`, `date_add`, `current_date`, `now`). The SQL `||` concat operator is
rewritten to `concat(...)` automatically. See
[references](references.md#scalar-functions-and-dialect-semantics) for the
full list and per-dialect semantics.
Filters in `SlayerQuery.filters` accept the
[portable function library](references.md#the-portable-function-library) — strings
(`lower`, `left`, `split_part`, `concat_ws`, …), math (`round`, `div`, `atan2`, …), null
handling, `greatest` / `least`, regex (`regexp_like`, `x ~ 'p'`), JSON (`json_value`,
`payload ->> 'key'`) and [date functions](references.md#date-and-time-functions) — each with
one meaning on every database. The SQL `||` concat operator is rewritten to `concat(...)`
automatically.

```json
"filters": [
Expand All @@ -375,16 +372,15 @@ full list and per-dialect semantics.
"replace(category, ',', '') = 'books'",
"substr(s, 1, instr(s, ',') - 1) = 'first_token'",
"coalesce(nickname, name) = 'Ada'",
"first || ' ' || last = 'jane doe'"
"first || ' ' || last = 'jane doe'",
"regexp_like(sku, '^[A-Z]{3}-[0-9]+$')",
"json_value(payload, '$.plan') = 'pro'"
]
```

Names are matched case-insensitively — `LOWER(...)` and `lower(...)` both bind.
sqlglot translates each call to the target dialect's preferred spelling at SQL-generation
time (`instr` → `POSITION` / `LOCATE` / `STRPOS`, `substr` →
`SUBSTRING`, `concat` → `||` on SQLite). Raw SQL functions outside the
allowlist (`json_extract`, `date_trunc`, `CASE WHEN`, …) belong in
`Column.sql` / `Column.filter` / `SlayerModel.filters` (Mode A SQL).
Names are matched case-insensitively — `LOWER(...)` and `lower(...)` both bind — and raw SQL
functions outside the library (`initcap`, `to_date`, …) belong in `Column.sql` /
`Column.filter` / `SlayerModel.filters` (Mode A SQL).

### Filtering on Computed Columns

Expand Down
45 changes: 30 additions & 15 deletions docs/concepts/references.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ SLayer has two distinct expression layers and the rules for what each one accept
| Mode | Fields | Parser | Accepts | Rejects |
|---|---|---|---|---|
| **A — SQL** | `Column.sql`, `Column.filter`, each entry of `SlayerModel.filters` | sqlglot | Any valid SQL expression for the underlying dialect — function calls (`json_extract`, `coalesce`, `nullif`, `lower`, `length`, …), arithmetic, `CASE WHEN`, string literals, comparison and boolean operators in SQL spelling (`=`, `<>`, `IS NULL`, `AND`, `OR`, `NOT`, `IN`, `LIKE`). Bare names and dotted join paths (`customers.regions.name`). | Aggregations (`sum(revenue)`); SLayer transform calls (`cumsum`, `change`, `rank`, …); references to `ModelMeasure` formulas; raw `OVER (...)` window functions inside `Column.filter` / `SlayerModel.filters` (allowed only in `Column.sql`); the legacy `__`-delimited split-alias qualifier (`customers__regions.name`) — now a hard error (`LegacyDunderAliasError`). |
| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | SQL-compatible grammar | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `sum(customers.revenue)`); aggregations (`sum(revenue)`, `count(*)`, `percentile(price, p=0.9)`, parametric forms), including same-model expression sources (`sum(amount - cost)`); transform calls (`cumsum(sum(revenue))`, `rank(sum(revenue), partition_by=region, direction='desc')`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); SQL forms — `CASE`, `CAST` / `TRY_CAST` / `::`, `BETWEEN`, `IS [NOT] DISTINCT FROM`, `[NOT] LIKE` / `ILIKE`, `EXTRACT`, `POSITION`, `SUBSTRING … FROM … FOR`, `INTERVAL` / `DATE` / `TIMESTAMP` literals, `CURRENT_DATE`, `FILTER (WHERE …)`, plus the `like(value, pattern)` scalar; a closed allowlist of scalar functions (matched case-insensitively) — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `ceiling`, `round`, `sign`, `trunc`, `mod`), scalar min/max (`greatest`, `least`), string hygiene (`lower`, `upper`, `trim`, `ltrim`, `rtrim`, `replace`, `substr`, `substring`, `instr`, `length`, `concat`), `like`, and [date functions](#date-and-time-functions) (`date_part`, `date_diff`, `date_add`, `current_date`, `now`), each with a declared argument count that is validated (`coalesce`, `concat`, `greatest` and `least` are variadic; `trunc` takes exactly one argument); `{variable}` placeholders supplying literal values. | identifiers using the reserved `__slayer_` prefix; raw SQL function calls outside that allowlist (`json_extract`, `date_trunc`, …), and any allowlisted call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below); `\|` (use `\|\|`), `//`, `SIMILAR TO`, `REGEXP` / `RLIKE`, `TIME '…'` literals. |
| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | SQL-compatible grammar | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `sum(customers.revenue)`); aggregations (`sum(revenue)`, `count(*)`, `percentile(price, p=0.9)`, parametric forms), including same-model expression sources (`sum(amount - cost)`); transform calls (`cumsum(sum(revenue))`, `rank(sum(revenue), partition_by=region, direction='desc')`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); SQL forms — `CASE`, `CAST` / `TRY_CAST` / `::`, `BETWEEN`, `IS [NOT] DISTINCT FROM`, `[NOT] LIKE` / `ILIKE`, the regex operators `~` / `~*` / `!~` / `!~*`, the JSON arrows `->` / `->>`, `EXTRACT`, `POSITION`, `SUBSTRING … FROM … FOR`, `INTERVAL` / `DATE` / `TIMESTAMP` literals, `CURRENT_DATE`, `FILTER (WHERE …)`, plus the `like(value, pattern)` scalar; the [portable function library](#the-portable-function-library) (matched case-insensitively) — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `ceiling`, `round`, `sign`, `trunc`, `mod`, `div`, `pi`, `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `atan2`, `cbrt`), scalar min/max (`greatest`, `least`), strings (`lower`, `upper`, `trim`, `ltrim`, `rtrim`, `replace`, `substr`, `substring`, `instr` / `strpos`, `length`, `concat`, `left`, `right`, `starts_with`, `ends_with`, `contains`, `lpad`, `rpad`, `reverse`, `repeat`, `split_part`, `concat_ws`, `chr`, `ascii`), regex (`regexp_like`, `regexp_extract`, `regexp_replace`), JSON (`json_value` and its aliases `json_extract`, `json_extract_scalar`, `json_extract_path_text`, `jsonb_extract_path_text`), `like`, and [date functions](#date-and-time-functions) (`date_part`, `date_diff`, `date_add`, `date_trunc`, `current_date`, `now`), each with a declared argument count that is validated (`coalesce`, `concat`, `concat_ws`, `greatest` and `least` are variadic; `trunc` takes exactly one argument); `{variable}` placeholders supplying literal values. | identifiers using the reserved `__slayer_` prefix; raw SQL function calls outside that library (`initcap`, `to_date`, `strftime`, JSON builders, …), and any library call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below); `\|` (use `\|\|`), `//` (use `div(a, b)`), `SIMILAR TO`, `REGEXP` / `RLIKE` (use `~`), a bare `->` (end the chain with `->>`), `TIME '…'` literals. |

## Identifier resolution

Expand Down Expand Up @@ -135,8 +135,8 @@ Rejected at `SlayerQuery` construction:
Rejected at enrichment:

```json
{"source_model": "orders", "dimensions": ["id"], "filters": ["json_extract(data, '$.x') > 5"]}
// ↑ ReferenceError: raw SQL function calls in DSL mode
{"source_model": "orders", "dimensions": ["id"], "filters": ["initcap(name) = 'Ada'"]}
// ↑ UnknownFunctionError: raw SQL function calls outside the library in DSL mode

{"source_model": "orders", "dimensions": ["id"], "filters": ["unknown_col > 0"]}
// ↑ ReferenceError: 'unknown_col' is not a Column / ModelMeasure on 'orders'
Expand All @@ -160,7 +160,7 @@ Accepted at construction:
Rejected at enrichment (when the formula is evaluated against a model):

```json
{"name": "bad", "formula": "json_extract(data, '$.x')"} // raw SQL fn
{"name": "bad", "formula": "initcap(name)"} // raw SQL fn
```

A bare `aov` or dotted `customers.aov` reuses a saved measure by expanding its formula inline, but only inside a measure formula or computed-dimension expression — see [Reusing another model's saved measure](models.md#reusing-another-models-saved-measure-customersaov).
Expand All @@ -186,17 +186,7 @@ Five consequences worth knowing:
{"filters": ["concat(ifnull(first_name, ''), ifnull(last_name, '')) = 'AdaLovelace'"]}
```

* **`greatest` / `least` NULL handling is backend-specific.** They pass through
to each backend's native form, and those forms disagree on `NULL`: Postgres,
DuckDB, SQL Server, ClickHouse 24.12+ and Snowflake (where the generated SQL
uses `GREATEST_IGNORE_NULLS`/`LEAST_IGNORE_NULLS`) *ignore* `NULL` arguments
(returning `NULL` only when every argument is `NULL`), while SQLite (scalar
`MAX`/`MIN`), MySQL, BigQuery and pre-24.12 ClickHouse (or 24.12+ with the
`least_greatest_legacy_null_behavior` setting) *propagate* `NULL`
(any `NULL` argument makes the result `NULL`). SLayer does not normalise this — wrap arguments in
`ifnull(...)` / `coalesce(...)` if you need one behaviour on every backend.
On SQL Server, `GREATEST` / `LEAST` require SQL Server 2022 or newer; earlier
versions reject the generated SQL.
* **`greatest` / `least` ignore `NULL` arguments on every backend** (NULL only when every argument is NULL); on SQL Server they need 2022 or later.

* **`log10` / `log2` keep their single-argument form** on the backends that
provide one, rather than becoming the generic two-argument `LOG(base, x)`.
Expand All @@ -222,6 +212,31 @@ Five consequences worth knowing:
written against either SQL convention binds, and both emit the target's own
form.

## The portable function library

Every library function, operator and pattern language means the same on SQLite, Postgres, DuckDB, MySQL, ClickHouse, SQL Server, Trino, Snowflake and BigQuery — Postgres's value where databases disagree — and is verified on each of them.

* **Out-of-domain arguments yield `NULL`** (`asin(2)` over a column, `div(a, 0)`, a surrogate `chr`), and a literal one is a plan-time error.
* **Strings count characters, not bytes**; `left` / `right` with a negative count drop characters from the other end, `lpad` / `rpad` truncate, `split_part`'s index is a non-zero integer literal (negative from the end, `''` past it), `concat_ws` skips `NULL`s, `chr` / `ascii` speak Unicode code points; `div` truncates toward zero.
* **A version-gated function fails at plan time** when the server reports an older version: regex needs SQL Server 2025, `greatest` / `least` SQL Server 2022; JSON reads and `TRY_CAST` return `NULL` on malformed input only on Postgres 16+.
* **`date_trunc('month', x)` is the granularity call `month(x)`** in every position.

### Regular expressions

`regexp_like(x, p[, 'i'])` (also `x ~ p`, `~*`, `!~`, `!~*`), `regexp_extract(x, p[, group])` and `regexp_replace(x, p, replacement[, 'i'])` take a pattern literal or `{variable}` in one portable subset — literals, `.`, `[…]` / `[^…]`, `* + ? {m,n}`, `^ $`, `|`, `( )`, `\d \w \s` and their negations, `\n \t \r`, a backslash before a metacharacter — where `.` matches newlines, `^` / `$` are the whole string's ends, `\d \w \s` are ASCII, matching is case-sensitive without `'i'`, and a replacement's `\1`–`\9` insert a group and `\\` a backslash (`$` and `&` are literal); anything else (lookaround, backreferences, lazy quantifiers, `(?…)`, `\b`, `\p{…}`, POSIX classes) is a plan-time error naming the construct, and a repetition such as `'[0-9]{3}'` needs no brace escaping.

### JSON

`json_value(x, '$.a[0]."b c"' [RETURNING INT | DOUBLE | BOOLEAN | DATE | TIMESTAMP | TEXT])` reads one scalar from JSON text or a native JSON column (`NULL` when missing, JSON null, an object or an array; `NULL` when the value cannot convert), the arrows `x -> 'a' ->> 'b'` and `x ->> '$.a.b'` spell the same read as TEXT, and a path is `$` followed by `.key`, `."quoted key"` and `[n]` steps only.

### Text beside a number

A TEXT operand meeting a number in a comparison, `IN`, `BETWEEN`, `IS [NOT] DISTINCT FROM`, arithmetic or `greatest` / `least` reads as `TRY_CAST(text AS DOUBLE)` (a non-number is `NULL`; a TEXT literal converts at plan time or is an error), so `amount_txt > 100` compares numerically everywhere — integers beyond 2^53 lose precision, so compare those with `TRY_CAST(x AS BIGINT)`, and a number function or aggregation over TEXT (`sum(amount_txt)`) still needs an explicit `TRY_CAST`.

### String matching

`=`, `<`, `IN` and `LIKE` follow the column's collation (except SQLite's `LIKE`, which is case-sensitive), `ILIKE` and the regex `'i'` flag are the case-insensitive forms, and every `LIKE` / `ILIKE` pattern reads `%` / `_` as wildcards, `\` as escaping `%`, `_` and itself (literal before anything else) and every other character, `[` included, as itself.

## Date and time functions

Five portable functions cover extraction, differences, arithmetic and the clock. Each renders
Expand Down
1 change: 1 addition & 0 deletions docs/database-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@ plus `$GCP_PROJECT_ID` for billing). The `bigquery://` driver requires the
`PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY x)` to BigQuery's analytic
form). Use a custom `Aggregation` with `APPROX_QUANTILES(x, 100)[OFFSET(N)]`
when you need it.
- **SLayer timestamps are `DATETIME`.** `now()`, timestamp literals, `TRY_CAST(… AS TIMESTAMP)`, `json_value(… RETURNING TIMESTAMP)` and the date functions produce naive `DATETIME` values (UTC wall clock), and a zoned `TIMESTAMP` column, which ingestion marks with `db_type: TIMESTAMP` (re-ingest older models), compares against them unconverted while the other side is cast to `TIMESTAMP`.
- **No native EXPLAIN.** BigQuery has no SQL-level `EXPLAIN`. The
`BigqueryDialect.explain_prefix` is `None`, so `engine.execute(...,
explain=True)` returns the dry-run SQL unchanged rather than an execution
Expand Down
Loading
Loading