diff --git a/.changeset/cli-skills-proxy.md b/.changeset/cli-skills-proxy.md new file mode 100644 index 00000000..04406154 --- /dev/null +++ b/.changeset/cli-skills-proxy.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Add a `chkit skills` command that proxies to the external `skills` CLI (e.g. `chkit skills add obsessiondb/chkit` runs `npx skills add obsessiondb/chkit`). The agent skill is installed by the separate `skills` tool, not a chkit subcommand, so users who reached for `chkit skills add …` previously hit "Unknown command: skills". The command forwards its arguments and passes through the underlying exit code, and is handled before config loading so it works without a project. diff --git a/.changeset/create-chkit-next-steps-dedup.md b/.changeset/create-chkit-next-steps-dedup.md new file mode 100644 index 00000000..882920b0 --- /dev/null +++ b/.changeset/create-chkit-next-steps-dedup.md @@ -0,0 +1,6 @@ +--- +"create-chkit": patch +"@chkit/plugin-obsessiondb": patch +--- + +Print the "Next steps" block once and with the correct runner for the selected package manager. `create-chkit` previously printed it twice — once package-manager-aware and once from onboarding with a hardcoded `bunx` — so `--package-manager npm` users were told to run `bunx chkit …`. Onboarding now derives the runner (`npx` / `pnpm dlx` / `yarn dlx` / `bunx`) from the package manager, and `create-chkit` only prints its own next-steps when onboarding is skipped, removing the duplicate. diff --git a/.changeset/init-connect-runbook-consistency.md b/.changeset/init-connect-runbook-consistency.md new file mode 100644 index 00000000..c7f42040 --- /dev/null +++ b/.changeset/init-connect-runbook-consistency.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Make `chkit init` consistent with `create-chkit` for connecting a database, and stop hiding plugin import failures. In a non-TTY shell `init` now prints the same connect runbook `create-chkit` does (when the obsessiondb plugin is installed) instead of silently skipping it; `--yes` still keeps init a silent file-writer for CI. The dynamic plugin import now only degrades silently when the plugin is genuinely not installed — any other load failure propagates instead of a false silent pass. The static next-steps also use `npx` rather than a hardcoded `bunx`. diff --git a/.changeset/migrate-async-poll-resilience.md b/.changeset/migrate-async-poll-resilience.md new file mode 100644 index 00000000..2b15e659 --- /dev/null +++ b/.changeset/migrate-async-poll-resilience.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Keep polling an async data-load migration through transient gateway errors instead of aborting. A single HTTP 524 (or other transient failure) on a status-poll request no longer cancels the migration: the server-side query keeps running, so chkit tolerates a bounded number of poll errors and only gives up after the budget, with an explicit message that the load may still be running and that re-running re-attaches via the deterministic `query_id`. Only a real query exception, or a submit-time failure, is fatal. This affects only operations marked `mode=async` (data loads); ordinary schema DDL is synchronous and unaffected. diff --git a/.changeset/obsessiondb-json-no-bare-string.md b/.changeset/obsessiondb-json-no-bare-string.md new file mode 100644 index 00000000..c3021392 --- /dev/null +++ b/.changeset/obsessiondb-json-no-bare-string.md @@ -0,0 +1,6 @@ +--- +"@chkit/plugin-obsessiondb": patch +"chkit": patch +--- + +Make `--json` always emit a JSON object, never a bare JSON-encoded string. `printOutput` now wraps any plain string printed under `--json` in `{ schemaVersion, message }`, closing the whole class of bug at the serializer so no command can leak a bare string. `chkit obsessiondb whoami` gains a structured envelope (`status: logged_in | not_logged_in | session_expired`), and `chkit obsessiondb service list` emits a single object with a `services[]` array instead of one JSON line per service (which was not valid single-JSON). Previously these commands `JSON.stringify`'d a prose string (e.g. `"Not logged in…"`), breaking any pipe to `jq`. Text-mode output is unchanged. Note: this changes the `--json` output shape of `whoami` and `service list` from a string to an object. diff --git a/.changeset/obsessiondb-logout-no-session.md b/.changeset/obsessiondb-logout-no-session.md new file mode 100644 index 00000000..67c75529 --- /dev/null +++ b/.changeset/obsessiondb-logout-no-session.md @@ -0,0 +1,5 @@ +--- +"@chkit/plugin-obsessiondb": patch +--- + +`chkit obsessiondb logout` now reports "No active session." when there are no stored credentials, instead of always printing "Logged out." (which implied it had ended a session that never existed). Logout stays idempotent and exits 0 either way; only the message changes. diff --git a/.changeset/pull-route-through-executor.md b/.changeset/pull-route-through-executor.md new file mode 100644 index 00000000..eef19290 --- /dev/null +++ b/.changeset/pull-route-through-executor.md @@ -0,0 +1,5 @@ +--- +"@chkit/plugin-pull": patch +--- + +Route `chkit pull` introspection through the host-provided executor instead of always opening its own ClickHouse connection. When an ObsessionDB service is selected, pull now introspects through the ObsessionDB API (the same executor `generate`/`migrate`/`status` use) rather than silently falling back to `http://localhost:8123` and failing with "connection refused" while printing `using service `. A direct ClickHouse target is unchanged, custom introspectors still open their own raw connection, and a run with no reachable target now errors with an actionable message instead of a misleading localhost fallback. diff --git a/.github/workflows/publish-chkit-py.yml b/.github/workflows/publish-chkit-py.yml new file mode 100644 index 00000000..b713a29e --- /dev/null +++ b/.github/workflows/publish-chkit-py.yml @@ -0,0 +1,66 @@ +name: Publish chkit-py to PyPI + +# Publishes on tags like `chkit-py-v0.2.0` via PyPI trusted publishing (OIDC): +# no API tokens stored in the repo. One-time setup on pypi.org: project +# `chkit-py` → Publishing → add a trusted publisher for +# obsessiondb/chkit with workflow `publish-chkit-py.yml`. +on: + push: + tags: + - "chkit-py-v*" + +jobs: + build: + runs-on: ubuntu-latest + defaults: + run: + working-directory: chkit_python + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Verify tag matches package version + run: | + version=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])") + tag="${GITHUB_REF_NAME#chkit-py-v}" + if [ "$version" != "$tag" ]; then + echo "Tag $GITHUB_REF_NAME does not match pyproject version $version" >&2 + exit 1 + fi + + - name: Build sdist and wheel + run: | + python -m pip install build twine + python -m build + python -m twine check dist/* + + - name: Smoke-test the wheel + run: | + python -m venv /tmp/wheeltest + /tmp/wheeltest/bin/pip install dist/*.whl + /tmp/wheeltest/bin/python -c "import chkit, chkit_plugin_backfill, chkit_plugin_codegen, chkit_plugin_obsessiondb; print(chkit.__version__)" + /tmp/wheeltest/bin/chkit --version + + - uses: actions/upload-artifact@v4 + with: + name: chkit-py-dist + path: chkit_python/dist/ + + publish: + needs: build + runs-on: ubuntu-latest + environment: pypi + permissions: + id-token: write + steps: + - uses: actions/download-artifact@v4 + with: + name: chkit-py-dist + path: dist/ + + - uses: pypa/gh-action-pypi-publish@release/v1 + with: + packages-dir: dist/ diff --git a/README.md b/README.md index 4054d8f4..839c06aa 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,13 @@ # chkit -**ClickHouse schema and migration CLI for TypeScript projects.** +**ClickHouse schema and migration toolkit for TypeScript and Python.** [![npm version](https://img.shields.io/npm/v/chkit?label=npm)](https://www.npmjs.com/package/chkit) [![CI](https://github.com/obsessiondb/chkit/actions/workflows/ci.yml/badge.svg)](https://github.com/obsessiondb/chkit/actions/workflows/ci.yml) [![Docs](https://img.shields.io/badge/docs-chkit.obsessiondb.com-blue)](https://chkit.obsessiondb.com) -Define your ClickHouse tables, views, and materialized views in TypeScript. chkit diffs your schema, generates migration SQL, applies it safely, and keeps your dev and production databases in sync -- all from the command line. +Define your ClickHouse tables, views, materialized views, and dictionaries in TypeScript or Python. chkit diffs your schema, generates migration SQL, applies it safely, and keeps your dev and production databases in sync -- all from the command line. > **Status: beta.** chkit powers production workloads and the CLI surface and schema DSL are stable. We may still make small breaking changes to UX and internal APIs before 1.0. @@ -116,6 +116,10 @@ See the [configuration docs](https://chkit.obsessiondb.com/configuration/overvie | [`@chkit/plugin-backfill`](packages/plugin-backfill) | Backfill plugin for data migrations | | [`@chkit/plugin-obsessiondb`](packages/plugin-obsessiondb) | ObsessionDB integration: auto-rewrite `Shared` engines for ClickHouse targets | +## Python + +chkit is also available for Python as [`chkit-py`](https://pypi.org/project/chkit-py/) (`pip install chkit-py`) — same CLI, same schema semantics, with config and schema files written as `.py`. The port lives in [`chkit_python/`](chkit_python). + ## Documentation Full documentation is available at **[chkit.obsessiondb.com](https://chkit.obsessiondb.com)**. diff --git a/apps/docs/astro.config.mjs b/apps/docs/astro.config.mjs index 091e0afb..988faec0 100644 --- a/apps/docs/astro.config.mjs +++ b/apps/docs/astro.config.mjs @@ -82,6 +82,10 @@ export default defineConfig({ label: 'Plugins', autogenerate: { directory: 'plugins' }, }, + { + label: 'Python', + autogenerate: { directory: 'python' }, + }, { label: 'CLI Reference', autogenerate: { directory: 'cli' }, diff --git a/apps/docs/src/components/Footer.astro b/apps/docs/src/components/Footer.astro index afe17a9e..39810154 100644 --- a/apps/docs/src/components/Footer.astro +++ b/apps/docs/src/components/Footer.astro @@ -18,7 +18,7 @@ const repo = 'https://github.com/obsessiondb/chkit';
ch-kit -

Headless TypeScript O(A)RM for ClickHouse!

+

Headless O(A)RM for ClickHouse — TypeScript & Python!

Ports
diff --git a/apps/docs/src/components/Hero.astro b/apps/docs/src/components/Hero.astro index 210dadf1..23ba4765 100644 --- a/apps/docs/src/components/Hero.astro +++ b/apps/docs/src/components/Hero.astro @@ -75,7 +75,7 @@ if (image) { TypeScript Python - Coming soon - +
{ diff --git a/apps/docs/src/content/docs/ai-agents.md b/apps/docs/src/content/docs/ai-agents.md index 2dd2ed45..3e3b7ecc 100644 --- a/apps/docs/src/content/docs/ai-agents.md +++ b/apps/docs/src/content/docs/ai-agents.md @@ -23,7 +23,7 @@ Every docs page is available as raw Markdown by appending `.md` to its URL — t ## What chkit is -chkit is a ClickHouse schema and migration toolkit for TypeScript. Schemas are defined in TypeScript, diffed into migration SQL, applied to ClickHouse, and verified against the live database. +chkit is a ClickHouse schema and migration toolkit for TypeScript and Python. Schemas are defined in TypeScript or Python, diffed into migration SQL, applied to ClickHouse, and verified against the live database. You drive chkit with shell commands, plus an installable agent skill that loads its full command surface, schema DSL, and workflows into your context. @@ -34,6 +34,7 @@ chkit's interactive CLI asks these questions when a human runs it. You run it no 1. **New project or existing project?** - *New / empty directory* → scaffold from a curated example with `create-chkit` (Step 3a). - *Existing TypeScript project* → install chkit and run `chkit init` in place (Step 3b). + - *Existing Python project* → `pip install chkit-py`, then `chkit init` in place (config and schema are written as `.py` files; plugins ship inside chkit-py). 2. **Is there an existing ClickHouse database with tables to manage?** - *Yes* → add [`@chkit/plugin-pull`](/plugins/pull/) and introspect the live tables into schema files, so the user starts from real tables instead of the blank example (Step 5). @@ -130,12 +131,12 @@ chkit check # CI gate: pending, checksums, drift, plugins ## Which plugins to recommend -Plugins are npm packages registered in the `plugins` array of `clickhouse.config.ts`. Recommend only what the project needs: +In TypeScript, plugins are npm packages registered in the `plugins` array of `clickhouse.config.ts`; in Python they ship inside chkit-py and are registered in `clickhouse.config.py`. Recommend only what the project needs: | If the project needs to... | Recommend | Notes | |----------------------------|-----------|-------| | Adopt chkit on an **existing** ClickHouse database | [`@chkit/plugin-pull`](/plugins/pull/) | Introspects the live database into local schema files so the user starts from real tables, not a blank example. | -| Generate **TypeScript types** (and optional Zod schemas) from the schema | [`@chkit/plugin-codegen`](/plugins/codegen/) | Keeps application row types in sync with the schema definitions. | +| Generate **typed row models** — TypeScript types (and optional Zod schemas), or Pydantic models in Python — from the schema | [`@chkit/plugin-codegen`](/plugins/codegen/) | Keeps application row types in sync with the schema definitions. | | **Backfill** historical data into materialized views | [`@chkit/plugin-backfill`](/plugins/backfill/) | Time-windowed loads with checkpoints, for large or resumable backfills. | | Deploy to **ObsessionDB** | [`@chkit/plugin-obsessiondb`](/obsessiondb/overview/) | First-class ObsessionDB integration; rewrites `Shared` engines when targeting non-ObsessionDB ClickHouse. | diff --git a/apps/docs/src/content/docs/cli/codegen.md b/apps/docs/src/content/docs/cli/codegen.md index 6a330d49..b0490048 100644 --- a/apps/docs/src/content/docs/cli/codegen.md +++ b/apps/docs/src/content/docs/cli/codegen.md @@ -1,11 +1,11 @@ --- title: "chkit codegen" -description: "Generate TypeScript types, ingestion functions, and runtime migration modules from schema definitions." +description: "Generate typed row models (TypeScript types or Pydantic models), plus TypeScript-only ingestion functions and runtime migration modules from schema definitions." sidebar: order: 9 --- -Shortcut for `chkit plugin codegen codegen`. Generates TypeScript row types, optional Zod schemas, ingestion functions, and runtime migration modules from your schema definitions. +Shortcut for `chkit plugin codegen codegen`. Generates TypeScript row types, optional Zod schemas (in Python: Pydantic models), ingestion functions, and runtime migration modules from your schema definitions. ## Synopsis diff --git a/apps/docs/src/content/docs/cli/generate.md b/apps/docs/src/content/docs/cli/generate.md index a40fc016..ba0ce188 100644 --- a/apps/docs/src/content/docs/cli/generate.md +++ b/apps/docs/src/content/docs/cli/generate.md @@ -5,7 +5,7 @@ sidebar: order: 3 --- -Compares your current TypeScript schema definitions against the previous snapshot, computes a migration plan, and writes migration SQL and an updated snapshot. +Compares your current schema definitions against the previous snapshot, computes a migration plan, and writes migration SQL and an updated snapshot. ## Synopsis @@ -217,4 +217,4 @@ chkit generate --rename-dictionary old_db.old_dict=new_db.new_dict - [The migration workflow](/guides/migration-workflow/) — why generate is offline, and what to commit alongside the SQL - [`chkit init`](/cli/init/) — scaffold a project before your first generate - [`chkit migrate`](/cli/migrate/) — apply generated migrations to ClickHouse -- [`chkit codegen`](/cli/codegen/) — manually trigger TypeScript type generation +- [`chkit codegen`](/cli/codegen/) — manually trigger type generation diff --git a/apps/docs/src/content/docs/cli/init.md b/apps/docs/src/content/docs/cli/init.md index b841d132..360f55ab 100644 --- a/apps/docs/src/content/docs/cli/init.md +++ b/apps/docs/src/content/docs/cli/init.md @@ -36,6 +36,8 @@ Writes two files relative to the current working directory, leaving any that alr 1. **`clickhouse.config.ts`** — project config with sensible defaults: `schema: './src/db/schema/**/*.ts'`, `outDir: './chkit'`, `migrationsDir: './chkit/migrations'`, `metaDir: './chkit/meta'`, an empty `plugins` array, and a `clickhouse` block reading from `CLICKHOUSE_URL`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, and `CLICKHOUSE_DB`. 2. **`src/db/schema/example.ts`** — a starter `MergeTree` table named `events` with columns `id` (`UInt64`), `source` (`String`), and `ingested_at` (`DateTime64(3)`). +Under [chkit-py](/python/overview/), the same command writes `clickhouse.config.py` and `src/db/schema/example.py` instead. + ### 2. Install dependencies If `@chkit/core` does not already resolve from the project, `init` makes the project runnable: it writes a minimal `package.json` when none exists, then installs `chkit`, `@chkit/core`, and `@chkit/plugin-obsessiondb` as dev dependencies using the detected package manager (`npm`, `pnpm`, `yarn`, or `bun`; defaults to `bun`). This is why `init` works in a brand-new empty folder, not just an existing project. A failed install never aborts `init` — it prints the manual install command and continues. diff --git a/apps/docs/src/content/docs/cli/overview.mdx b/apps/docs/src/content/docs/cli/overview.mdx index 0d53f247..f49374a5 100644 --- a/apps/docs/src/content/docs/cli/overview.mdx +++ b/apps/docs/src/content/docs/cli/overview.mdx @@ -8,7 +8,7 @@ sidebar: import { Image } from 'astro:assets'; import commandConnections from '../../../assets/command-connections.png'; -The `chkit` CLI manages ClickHouse schemas, migrations, drift detection, and CI checks from the command line. It follows a workflow-oriented design: define your schema in TypeScript, generate migrations, apply them, and verify everything stays in sync. +The `chkit` CLI manages ClickHouse schemas, migrations, drift detection, and CI checks from the command line. It follows a workflow-oriented design: define your schema in TypeScript or Python, generate migrations, apply them, and verify everything stays in sync. ## Commands @@ -21,8 +21,8 @@ The `chkit` CLI manages ClickHouse schemas, migrations, drift detection, and CI | [`chkit drift`](/cli/drift/) | Compare snapshot against live ClickHouse and report differences | | [`chkit check`](/cli/check/) | Run policy checks for CI gates (pending, checksums, drift, plugins) | | [`chkit query`](/cli/query/) | Run an ad-hoc SQL query against the configured target | -| [`chkit pull`](/cli/pull/) | Introspect live ClickHouse and generate a TypeScript schema file | -| [`chkit codegen`](/cli/codegen/) | Generate TypeScript types from schema definitions | +| [`chkit pull`](/cli/pull/) | Introspect live ClickHouse and generate a schema file | +| [`chkit codegen`](/cli/codegen/) | Generate typed row models from schema definitions | | [`chkit plugin`](/cli/plugin/) | List or run plugin commands | ## Connection requirements @@ -51,7 +51,7 @@ These flags are available on every command that loads a config file: | Flag | Type | Default | Description | |------|------|---------|-------------| -| `--config ` | string | `clickhouse.config.ts` | Path to the chkit config file | +| `--config ` | string | `clickhouse.config.ts` / `clickhouse.config.py` | Path to the chkit config file | | `--json` | boolean | `false` | Emit machine-readable JSON output | | `--table ` | string | — | Narrow some commands to matching tables (exact name or trailing wildcard prefix, e.g. `events_*`). Effect varies per command — see note below | | `--help` | boolean | — | Show help text | diff --git a/apps/docs/src/content/docs/cli/plugin.md b/apps/docs/src/content/docs/cli/plugin.mdx similarity index 81% rename from apps/docs/src/content/docs/cli/plugin.md rename to apps/docs/src/content/docs/cli/plugin.mdx index 652324ff..435b8882 100644 --- a/apps/docs/src/content/docs/cli/plugin.md +++ b/apps/docs/src/content/docs/cli/plugin.mdx @@ -5,6 +5,8 @@ sidebar: order: 11 --- +import { Tabs, TabItem } from '@astrojs/starlight/components'; + Lists registered plugins, lists a plugin's commands, or runs a specific plugin command. ## Synopsis @@ -38,17 +40,28 @@ With both a plugin name and command name, executes the command. Any additional a Some plugins have top-level CLI shortcuts: - `chkit codegen` is equivalent to `chkit plugin codegen codegen` -- `chkit pull` is equivalent to `chkit plugin pull schema` +- `chkit pull` is equivalent to `chkit plugin pull schema` (TypeScript; in Python `pull` is a built-in command with no plugin behind it) ### Plugin registration -Plugins are registered inline in the `plugins` array of `clickhouse.config.ts`: +Plugins are registered inline in the `plugins` array of your config: -```ts -import { codegen } from '@chkit/plugin-codegen' + + + ```ts + import { codegen } from '@chkit/plugin-codegen' -plugins: [codegen({ outFile: './types.ts' })] -``` + plugins: [codegen({ outFile: './types.ts' })] + ``` + + + ```python + from chkit_plugin_codegen import codegen + + "plugins": [codegen({"outFile": "./models.py"})] + ``` + + ### Plugin lifecycle hooks diff --git a/apps/docs/src/content/docs/cli/pull.md b/apps/docs/src/content/docs/cli/pull.md index 00aa084a..757b0f12 100644 --- a/apps/docs/src/content/docs/cli/pull.md +++ b/apps/docs/src/content/docs/cli/pull.md @@ -1,11 +1,11 @@ --- title: "chkit pull" -description: "Introspect live ClickHouse and generate a TypeScript schema file." +description: "Introspect live ClickHouse and generate a schema file." sidebar: order: 8 --- -Shortcut for `chkit plugin pull schema`. Introspects your live ClickHouse instance and generates a deterministic TypeScript schema file. +Shortcut for `chkit plugin pull schema`. Introspects your live ClickHouse instance and generates a deterministic schema file (TypeScript, or Python under chkit-py). ## Synopsis diff --git a/apps/docs/src/content/docs/configuration/overview.md b/apps/docs/src/content/docs/configuration/overview.mdx similarity index 50% rename from apps/docs/src/content/docs/configuration/overview.md rename to apps/docs/src/content/docs/configuration/overview.mdx index 5762535f..b5a58509 100644 --- a/apps/docs/src/content/docs/configuration/overview.md +++ b/apps/docs/src/content/docs/configuration/overview.mdx @@ -1,9 +1,11 @@ --- title: "Configuration Overview" -description: "clickhouse.config.ts structure and defaults." +description: "clickhouse.config.ts / clickhouse.config.py structure and defaults." --- -`chkit` is configured through `clickhouse.config.ts`. +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +`chkit` is configured through `clickhouse.config.ts` (TypeScript) or `clickhouse.config.py` (Python). The option keys and defaults are identical. ## Core Fields @@ -20,35 +22,75 @@ Migration state (the journal of applied migrations) is not stored in `metaDir`. ## Example -```ts -import { defineConfig } from '@chkit/core' - -export default defineConfig({ - schema: './src/db/schema/**/*.ts', - outDir: './chkit', - migrationsDir: './chkit/migrations', - metaDir: './chkit/meta', - clickhouse: { - url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', - username: process.env.CLICKHOUSE_USER ?? 'default', - password: process.env.CLICKHOUSE_PASSWORD ?? '', - database: process.env.CLICKHOUSE_DB ?? 'default', - }, -}) -``` + + + ```ts + import { defineConfig } from '@chkit/core' + + export default defineConfig({ + schema: './src/db/schema/**/*.ts', + outDir: './chkit', + migrationsDir: './chkit/migrations', + metaDir: './chkit/meta', + clickhouse: { + url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', + username: process.env.CLICKHOUSE_USER ?? 'default', + password: process.env.CLICKHOUSE_PASSWORD ?? '', + database: process.env.CLICKHOUSE_DB ?? 'default', + }, + }) + ``` + + + ```python + import os + + from chkit import define_config + + config = define_config( + { + "schema": "./src/db/schema/**/*.py", + "outDir": "./chkit", + "migrationsDir": "./chkit/migrations", + "metaDir": "./chkit/meta", + "clickhouse": { + "url": os.environ.get("CLICKHOUSE_URL", "http://localhost:8123"), + "username": os.environ.get("CLICKHOUSE_USER", "default"), + "password": os.environ.get("CLICKHOUSE_PASSWORD", ""), + "database": os.environ.get("CLICKHOUSE_DB", "default"), + }, + } + ) + ``` + + ## Cluster mode (`ON CLUSTER`) For self-managed multi-node ClickHouse clusters, set `clickhouse.cluster` to the cluster name from your server's `remote_servers` config: -```ts -clickhouse: { - url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', - password: process.env.CLICKHOUSE_PASSWORD ?? '', - database: 'default', - cluster: 'my_cluster', -}, -``` + + + ```ts + clickhouse: { + url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', + password: process.env.CLICKHOUSE_PASSWORD ?? '', + database: 'default', + cluster: 'my_cluster', + }, + ``` + + + ```python + "clickhouse": { + "url": os.environ.get("CLICKHOUSE_URL", "http://localhost:8123"), + "password": os.environ.get("CLICKHOUSE_PASSWORD", ""), + "database": "default", + "cluster": "my_cluster", + }, + ``` + + When `cluster` is set, chkit: @@ -65,4 +107,4 @@ Leave `cluster` unset for single-node servers, ClickHouse Cloud, or ObsessionDB, Project-scoped commands (`generate`, `migrate`, `status`, `drift`, `check`, `codegen`, `pull`) always require a project config in the working directory. -[`chkit query`](/cli/query/) is the exception: when no project config is found, chkit falls back to a user-profile config at `~/.config/chkit/config.ts` (honoring `XDG_CONFIG_HOME`). This lets ad-hoc queries run from any directory. If ObsessionDB credentials are present (`~/.config/chkit/credentials.json`), chkit synthesizes a minimal query-only config automatically, so `chkit query` works after `chkit obsessiondb login` without a local config file at all. +[`chkit query`](/cli/query/) is the exception: when no project config is found, chkit falls back to a user-profile config at `~/.config/chkit/config.ts` (`config.py` in Python; honoring `XDG_CONFIG_HOME`). This lets ad-hoc queries run from any directory. If ObsessionDB credentials are present (`~/.config/chkit/credentials.json`), chkit synthesizes a minimal query-only config automatically, so `chkit query` works after `chkit obsessiondb login` without a local config file at all. diff --git a/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx b/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx index 530c271d..fa2f2d7c 100644 --- a/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx +++ b/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx @@ -10,6 +10,10 @@ import Command from '../../../components/Command.astro'; Install chkit alongside your existing application code, drop a minimal config and starter schema into the current directory, and produce your first migration. Use this path when you already have a TypeScript project that talks to ClickHouse. +:::note[Working in Python?] +`pip install chkit-py`, then run `chkit init` in your project — it writes `clickhouse.config.py` and a starter schema. The rest of this page's flow (generate → migrate → check) is identical; only the install step differs. See the [Python overview](/python/overview/). +::: + ## Prerequisites - Node.js 20+ or Bun 1.3.5+ @@ -47,7 +51,7 @@ On an interactive run, `chkit init` also offers to connect a database — includ ## 5. Verify -`chkit status` shows which migrations have been applied. `chkit check` confirms the live schema matches your TypeScript definitions. +`chkit status` shows which migrations have been applied. `chkit check` confirms the live schema matches your schema definitions. diff --git a/apps/docs/src/content/docs/getting-started/index.mdx b/apps/docs/src/content/docs/getting-started/index.mdx index 032e2e4a..0a253621 100644 --- a/apps/docs/src/content/docs/getting-started/index.mdx +++ b/apps/docs/src/content/docs/getting-started/index.mdx @@ -9,7 +9,11 @@ sidebar: import { LinkCard, CardGrid } from '@astrojs/starlight/components'; import CopyPromptButton from '../../../components/CopyPromptButton.astro'; -chkit is a ClickHouse schema and migration toolkit for TypeScript projects. Pick the path that matches what you're working on. +chkit is a ClickHouse schema and migration toolkit for TypeScript and Python projects. Pick the path that matches what you're working on. + +:::note[Working in Python?] +chkit also ships as [chkit-py](/python/overview/) — same CLI, same schema semantics, `pip install chkit-py`. The reference pages on this site carry synced TypeScript/Python tabs. +::: ## Let an agent set it up @@ -28,7 +32,7 @@ Prefer to do it yourself? Pick a path below. @@ -46,5 +50,5 @@ Once either path is working: - [The migration workflow](/guides/migration-workflow/) — how generate, snapshot, and migrate fit together, and what to commit - [CLI reference](/cli/overview/) — every command, flag, and expected output - [Configuration](/configuration/overview/) — wire up `clickhouse.config.ts` -- [Schema DSL](/schema/dsl-reference/) — define tables, views, and materialized views +- [Schema DSL](/schema/dsl-reference/) — define tables, views, materialized views, and dictionaries - [Troubleshooting](/guides/troubleshooting/) — fixes for common errors if a command fails diff --git a/apps/docs/src/content/docs/getting-started/with-an-example.mdx b/apps/docs/src/content/docs/getting-started/with-an-example.mdx index 50248ccf..4aa9a2b0 100644 --- a/apps/docs/src/content/docs/getting-started/with-an-example.mdx +++ b/apps/docs/src/content/docs/getting-started/with-an-example.mdx @@ -10,6 +10,10 @@ import Command from '../../../components/Command.astro'; `create-chkit` scaffolds a working chkit project by downloading a curated example from the chkit repository and wiring it up against your chosen package manager. Use this path when you want a known-good project as a starting point. +:::note[Working in Python?] +The `create-chkit` examples are TypeScript projects. For Python, start with `pip install chkit-py` and run `chkit init` in your project instead — see the [Python overview](/python/overview/). +::: + ## Prerequisites - Node.js 20+ or Bun 1.3.5+ diff --git a/apps/docs/src/content/docs/guides/ci-cd.md b/apps/docs/src/content/docs/guides/ci-cd.mdx similarity index 59% rename from apps/docs/src/content/docs/guides/ci-cd.md rename to apps/docs/src/content/docs/guides/ci-cd.mdx index 271c09af..183450a8 100644 --- a/apps/docs/src/content/docs/guides/ci-cd.md +++ b/apps/docs/src/content/docs/guides/ci-cd.mdx @@ -3,6 +3,8 @@ title: CI/CD Integration description: Run chkit schema validation, migration deployment, and type checking in continuous integration pipelines. --- +import { Tabs, TabItem } from '@astrojs/starlight/components'; + chkit is designed to run unattended in CI pipelines. Every command supports a `--json` flag for machine-readable output, and the CLI automatically detects non-interactive environments so it never blocks on prompts. Three primary CI use cases: @@ -46,19 +48,42 @@ When non-interactive, `chkit migrate` without `--apply` prints the migration pla Your `clickhouse.config.ts` can export a function that receives a `ChxConfigEnv` object with `command` and `mode` fields. Use this to vary config per environment: -```ts -import { defineConfig } from '@chkit/core' - -export default defineConfig((env) => ({ - schema: './schema/**/*.ts', - clickhouse: { - url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', - username: process.env.CLICKHOUSE_USER ?? 'default', - password: process.env.CLICKHOUSE_PASSWORD ?? '', - database: process.env.CLICKHOUSE_DB ?? 'default', - }, -})) -``` + + + ```ts + import { defineConfig } from '@chkit/core' + + export default defineConfig((env) => ({ + schema: './schema/**/*.ts', + clickhouse: { + url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', + username: process.env.CLICKHOUSE_USER ?? 'default', + password: process.env.CLICKHOUSE_PASSWORD ?? '', + database: process.env.CLICKHOUSE_DB ?? 'default', + }, + })) + ``` + + + ```python + import os + + from chkit import define_config + + config = define_config( + lambda env: { + "schema": "./schema/**/*.py", + "clickhouse": { + "url": os.environ.get("CLICKHOUSE_URL", "http://localhost:8123"), + "username": os.environ.get("CLICKHOUSE_USER", "default"), + "password": os.environ.get("CLICKHOUSE_PASSWORD", ""), + "database": os.environ.get("CLICKHOUSE_DB", "default"), + }, + } + ) + ``` + + ## Check policies @@ -71,18 +96,36 @@ The `chkit check` command evaluates three policies: | `failOnDrift` | `true` | Fail if live schema drifts from snapshot | | `failOnExtraObjects` | `false` | Fail if ClickHouse has objects not in your schema (off by default so chkit coexists with unmanaged tables on a shared database) | -The first three default to `true`, so checks are strict out of the box. `failOnExtraObjects` defaults to `false` — enable it only when chkit owns the entire database. Override them in `clickhouse.config.ts`: - -```ts -export default defineConfig({ - schema: './schema/**/*.ts', - check: { - failOnPending: true, - failOnChecksumMismatch: true, - failOnDrift: false, // disable drift checking - }, -}) -``` +The first three default to `true`, so checks are strict out of the box. `failOnExtraObjects` defaults to `false` — enable it only when chkit owns the entire database. Override them in your config: + + + + ```ts + export default defineConfig({ + schema: './schema/**/*.ts', + check: { + failOnPending: true, + failOnChecksumMismatch: true, + failOnDrift: false, // disable drift checking + }, + }) + ``` + + + ```python + config = define_config( + { + "schema": "./schema/**/*.py", + "check": { + "failOnPending": True, + "failOnChecksumMismatch": True, + "failOnDrift": False, # disable drift checking + }, + } + ) + ``` + + The `--strict` flag forces all three policies to `true`, overriding any config. **Use `--strict` in CI** to ensure no permissive config setting leaks through. @@ -90,60 +133,121 @@ Plugin checks (like `codegen`) are also evaluated automatically — a plugin is ## GitHub Actions: schema validation on PRs -```yaml -name: Schema Validation -on: pull_request - -jobs: - validate: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - uses: oven-sh/setup-bun@v2 - with: - bun-version: '1.3.5' - - - run: bun install --frozen-lockfile - - - name: Check schema consistency - run: bunx chkit check --strict --json - - - name: Verify generated types - run: bunx chkit codegen --check --json -``` + + + ```yaml + name: Schema Validation + on: pull_request + + jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: '1.3.5' + + - run: bun install --frozen-lockfile + + - name: Check schema consistency + run: bunx chkit check --strict --json + + - name: Verify generated types + run: bunx chkit codegen --check --json + ``` + + + ```yaml + name: Schema Validation + on: pull_request + + jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - run: pip install chkit-py + + - name: Check schema consistency + run: chkit check --strict --json + + - name: Verify generated models + run: chkit codegen --check --json + ``` + + This workflow runs on every pull request. Both commands exit with code 1 on failure, which fails the GitHub Actions step. ## GitHub Actions: migration deployment on merge -```yaml -name: Deploy Migrations -on: - push: - branches: [main] - -jobs: - deploy: - runs-on: ubuntu-latest - env: - CLICKHOUSE_URL: ${{ secrets.CLICKHOUSE_URL }} - CLICKHOUSE_USER: ${{ secrets.CLICKHOUSE_USER }} - CLICKHOUSE_PASSWORD: ${{ secrets.CLICKHOUSE_PASSWORD }} - CLICKHOUSE_DB: ${{ secrets.CLICKHOUSE_DB }} - - steps: - - uses: actions/checkout@v4 - - - uses: oven-sh/setup-bun@v2 - with: - bun-version: '1.3.5' - - - run: bun install --frozen-lockfile - - - name: Apply migrations - run: bunx chkit migrate --apply --json -``` + + + ```yaml + name: Deploy Migrations + on: + push: + branches: [main] + + jobs: + deploy: + runs-on: ubuntu-latest + env: + CLICKHOUSE_URL: ${{ secrets.CLICKHOUSE_URL }} + CLICKHOUSE_USER: ${{ secrets.CLICKHOUSE_USER }} + CLICKHOUSE_PASSWORD: ${{ secrets.CLICKHOUSE_PASSWORD }} + CLICKHOUSE_DB: ${{ secrets.CLICKHOUSE_DB }} + + steps: + - uses: actions/checkout@v4 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: '1.3.5' + + - run: bun install --frozen-lockfile + + - name: Apply migrations + run: bunx chkit migrate --apply --json + ``` + + + ```yaml + name: Deploy Migrations + on: + push: + branches: [main] + + jobs: + deploy: + runs-on: ubuntu-latest + env: + CLICKHOUSE_URL: ${{ secrets.CLICKHOUSE_URL }} + CLICKHOUSE_USER: ${{ secrets.CLICKHOUSE_USER }} + CLICKHOUSE_PASSWORD: ${{ secrets.CLICKHOUSE_PASSWORD }} + CLICKHOUSE_DB: ${{ secrets.CLICKHOUSE_DB }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - run: pip install chkit-py + + - name: Apply migrations + run: chkit migrate --apply --json + ``` + + :::caution This workflow does **not** pass `--allow-destructive`. If a migration contains destructive operations (dropping tables or columns), the command exits with code 3 and the deployment stops. See [Handling destructive migrations](#handling-destructive-migrations-in-ci) for how to handle this safely. @@ -151,58 +255,121 @@ This workflow does **not** pass `--allow-destructive`. If a migration contains d ## GitLab CI -```yaml -stages: - - validate - - deploy - -.bun-setup: &bun-setup - image: oven/bun:1.3.5 - before_script: - - bun install --frozen-lockfile - -schema-check: - <<: *bun-setup - stage: validate - script: - - bunx chkit check --strict --json - - bunx chkit codegen --check --json - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - -deploy-migrations: - <<: *bun-setup - stage: deploy - script: - - bunx chkit migrate --apply --json - variables: - CLICKHOUSE_URL: $CLICKHOUSE_URL - CLICKHOUSE_USER: $CLICKHOUSE_USER - CLICKHOUSE_PASSWORD: $CLICKHOUSE_PASSWORD - CLICKHOUSE_DB: $CLICKHOUSE_DB - rules: - - if: $CI_COMMIT_BRANCH == "main" -``` + + + ```yaml + stages: + - validate + - deploy + + .bun-setup: &bun-setup + image: oven/bun:1.3.5 + before_script: + - bun install --frozen-lockfile + + schema-check: + <<: *bun-setup + stage: validate + script: + - bunx chkit check --strict --json + - bunx chkit codegen --check --json + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + + deploy-migrations: + <<: *bun-setup + stage: deploy + script: + - bunx chkit migrate --apply --json + variables: + CLICKHOUSE_URL: $CLICKHOUSE_URL + CLICKHOUSE_USER: $CLICKHOUSE_USER + CLICKHOUSE_PASSWORD: $CLICKHOUSE_PASSWORD + CLICKHOUSE_DB: $CLICKHOUSE_DB + rules: + - if: $CI_COMMIT_BRANCH == "main" + ``` + + + ```yaml + stages: + - validate + - deploy + + .py-setup: &py-setup + image: python:3.12-slim + before_script: + - pip install chkit-py + + schema-check: + <<: *py-setup + stage: validate + script: + - chkit check --strict --json + - chkit codegen --check --json + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + + deploy-migrations: + <<: *py-setup + stage: deploy + script: + - chkit migrate --apply --json + variables: + CLICKHOUSE_URL: $CLICKHOUSE_URL + CLICKHOUSE_USER: $CLICKHOUSE_USER + CLICKHOUSE_PASSWORD: $CLICKHOUSE_PASSWORD + CLICKHOUSE_DB: $CLICKHOUSE_DB + rules: + - if: $CI_COMMIT_BRANCH == "main" + ``` + + ## Generic CI setup For any CI system, use this shell script: -```bash -#!/usr/bin/env bash -set -euo pipefail - -bun install --frozen-lockfile - -# Validate schema -bunx chkit check --strict --json -bunx chkit codegen --check --json - -# Deploy on main branch only -if [ "${BRANCH:-}" = "main" ]; then - bunx chkit migrate --apply --json -fi -``` + + + ```sh + #!/usr/bin/env bash + set -euo pipefail + + bun install --frozen-lockfile + + # Validate schema + bunx chkit check --strict --json + bunx chkit codegen --check --json + + # Deploy on main branch only + if [ "${BRANCH:-}" = "main" ]; then + bunx chkit migrate --apply --json + fi + ``` + + + ```sh + #!/usr/bin/env bash + set -euo pipefail + + pip install chkit-py + + # Validate schema + chkit check --strict --json + chkit codegen --check --json + + # Deploy on main branch only + if [ "${BRANCH:-}" = "main" ]; then + chkit migrate --apply --json + fi + ``` + + + +:::note +The remaining examples on this page are shown in the TypeScript form. For Python, the swap is always the same: replace the Bun setup steps with `pip install chkit-py` and drop the `bunx` prefix — the `chkit` commands, flags, JSON output, and exit codes are identical. +::: ## Handling destructive migrations in CI diff --git a/apps/docs/src/content/docs/guides/migration-workflow.mdx b/apps/docs/src/content/docs/guides/migration-workflow.mdx index 70eb91b4..7838c537 100644 --- a/apps/docs/src/content/docs/guides/migration-workflow.mdx +++ b/apps/docs/src/content/docs/guides/migration-workflow.mdx @@ -21,7 +21,7 @@ chkit uses the second approach. `chkit generate` computes the diff and writes it ## The diff needs a baseline: that's the snapshot -To compute where the schema is against where it should be, the diff needs both sides. The target side is your TypeScript schema. The baseline, the last-known state, has to come from somewhere. +To compute where the schema is against where it should be, the diff needs both sides. The target side is your schema code. The baseline, the last-known state, has to come from somewhere. There are two ways to supply that baseline. Introspect a live database, which ties every diff to a reachable instance and to which instance you point at. Or read a committed file. chkit keeps the baseline in a file: `chkit/meta/snapshot.json`. @@ -52,14 +52,14 @@ The record of which migrations have run is deliberately not a file. It lives in This is the workflow for a team where only some developers can reach production. Assume one person has a one-time connection to prod and nobody else does. -1. **Seed the project once.** The person with access runs [`chkit pull`](/cli/pull/) against prod to introspect the existing schema into TypeScript, then `chkit generate` to produce the initial snapshot. Commit the schema files, the migration(s), and `snapshot.json`. +1. **Seed the project once.** The person with access runs [`chkit pull`](/cli/pull/) against prod to introspect the existing schema into schema files, then `chkit generate` to produce the initial snapshot. Commit the schema files, the migration(s), and `snapshot.json`. ```sh chkit pull --out-file ./src/db/schema/pulled.ts # connects to prod, one time chkit generate --name baseline # offline; writes snapshot.json ``` -2. **Everyone else works offline.** Any developer edits the TypeScript schema and runs `generate`. No production access and no local ClickHouse are needed, because the snapshot is the baseline being diffed against. +2. **Everyone else works offline.** Any developer edits the schema files and runs `generate`. No production access and no local ClickHouse are needed, because the snapshot is the baseline being diffed against. ```sh # edit src/db/schema/*.ts diff --git a/apps/docs/src/content/docs/guides/troubleshooting.md b/apps/docs/src/content/docs/guides/troubleshooting.mdx similarity index 86% rename from apps/docs/src/content/docs/guides/troubleshooting.md rename to apps/docs/src/content/docs/guides/troubleshooting.mdx index 9546b3e2..e5dd2fe3 100644 --- a/apps/docs/src/content/docs/guides/troubleshooting.md +++ b/apps/docs/src/content/docs/guides/troubleshooting.mdx @@ -3,6 +3,8 @@ title: Troubleshooting description: Common chkit errors mapped to their causes and fixes — missing dependencies, connection and auth failures, rejected migrations, blocked destructive operations, and drift. --- +import { Tabs, TabItem } from '@astrojs/starlight/components'; + A reference for the errors you are most likely to hit when running chkit, what causes each, and how to fix it. Errors are grouped by the stage where they surface: loading the project, connecting to ClickHouse, and running migrations. ## Quick reference @@ -27,19 +29,38 @@ The config (`clickhouse.config.ts`) and your schema files import `@chkit/core`, Install the dependencies in the project directory: -```sh -bun add -d chkit @chkit/core -``` + + + ```sh + bun add -d chkit @chkit/core + ``` + + + ```sh + pip install chkit-py + ``` + The Python equivalent of this error is a `ModuleNotFoundError: No module named 'chkit'` from your schema files — same cause, same fix. + + ### `Unknown file extension ".ts"` Older chkit versions could not load a TypeScript config under plain Node. Recent versions bundle a loader, so the fix is to upgrade: -```sh -bun add -d chkit@latest -``` - -Under Bun this never occurred; under Node it now works the same way. + + + ```sh + bun add -d chkit@latest + ``` + Under Bun this never occurred; under Node it now works the same way. + + + ```sh + pip install --upgrade chkit-py + ``` + This error is TypeScript-specific; Python configs (`clickhouse.config.py`) are plain modules and never hit it. + + ## Connecting to ClickHouse diff --git a/apps/docs/src/content/docs/index.mdx b/apps/docs/src/content/docs/index.mdx index f62801f2..5cd9d7d7 100644 --- a/apps/docs/src/content/docs/index.mdx +++ b/apps/docs/src/content/docs/index.mdx @@ -1,12 +1,12 @@ --- title: chkit Documentation -description: Public docs for chkit, a ClickHouse schema and migration CLI for TypeScript projects. +description: Public docs for chkit, a ClickHouse schema and migration CLI for TypeScript and Python. template: splash prev: false next: false hero: title: Run schema changes with confidence. - tagline: ClickHouse schema management and migrations, in TypeScript. + tagline: ClickHouse schema management and migrations, in TypeScript or Python. image: alt: A worker assembling scaffolding file: ../../assets/chkit-scaffold.png diff --git a/apps/docs/src/content/docs/obsessiondb/getting-started.md b/apps/docs/src/content/docs/obsessiondb/getting-started.mdx similarity index 86% rename from apps/docs/src/content/docs/obsessiondb/getting-started.md rename to apps/docs/src/content/docs/obsessiondb/getting-started.mdx index 1b8d3a4c..8f79477d 100644 --- a/apps/docs/src/content/docs/obsessiondb/getting-started.md +++ b/apps/docs/src/content/docs/obsessiondb/getting-started.mdx @@ -5,11 +5,13 @@ sidebar: order: 2 --- +import { Tabs, TabItem } from '@astrojs/starlight/components'; + Connect chkit to ObsessionDB without copying URLs or tokens by hand: scaffold a project and pick a connection in one prompt, or sign up and claim a free dev instance straight from the CLI. ## Connect in one step -When you scaffold a project with `bun create chkit@latest` or run `chkit init` in an existing one, chkit asks how you want to connect: +When you scaffold a project with `bun create chkit@latest` (TypeScript) or run `chkit init` in an existing one (both languages), chkit asks how you want to connect: ``` Claim a free ObsessionDB dev instance email code, ready in seconds @@ -28,22 +30,42 @@ The rest of this page covers each path as standalone CLI commands, which is also ## Install the plugin -```sh -bun add -d @chkit/plugin-obsessiondb -``` - -Register it in your `clickhouse.config.ts`: - -```ts -import { defineConfig } from '@chkit/core' -import { obsessiondb } from '@chkit/plugin-obsessiondb' - -export default defineConfig({ - schema: './src/db/schema/**/*.ts', - outDir: './chkit', - plugins: [obsessiondb()], -}) -``` + + + ```sh + bun add -d @chkit/plugin-obsessiondb + ``` + + Register it in your `clickhouse.config.ts`: + + ```ts + import { defineConfig } from '@chkit/core' + import { obsessiondb } from '@chkit/plugin-obsessiondb' + + export default defineConfig({ + schema: './src/db/schema/**/*.ts', + outDir: './chkit', + plugins: [obsessiondb()], + }) + ``` + + + The plugin ships inside `chkit-py`. Register it in your `clickhouse.config.py`: + + ```python + from chkit import define_config + from chkit_plugin_obsessiondb import obsessiondb + + config = define_config( + { + "schema": "./src/db/schema/**/*.py", + "outDir": "./chkit", + "plugins": [obsessiondb()], + } + ) + ``` + + You don't need a `clickhouse` block — once a service is selected, the plugin routes SQL through the ObsessionDB API. diff --git a/apps/docs/src/content/docs/obsessiondb/overview.md b/apps/docs/src/content/docs/obsessiondb/overview.mdx similarity index 59% rename from apps/docs/src/content/docs/obsessiondb/overview.md rename to apps/docs/src/content/docs/obsessiondb/overview.mdx index 7dca509b..46d68e19 100644 --- a/apps/docs/src/content/docs/obsessiondb/overview.md +++ b/apps/docs/src/content/docs/obsessiondb/overview.mdx @@ -5,6 +5,8 @@ sidebar: order: 1 --- +import { Tabs, TabItem } from '@astrojs/starlight/components'; + chkit ships a dedicated integration with [ObsessionDB](https://obsessiondb.com), the managed ClickHouse-compatible database that provides `Shared` engine variants and a hosted API for queries and backfills. ## What it gives you @@ -17,25 +19,50 @@ chkit ships a dedicated integration with [ObsessionDB](https://obsessiondb.com), ## Install -```sh -bun add -d @chkit/plugin-obsessiondb -``` - -Register it in your `clickhouse.config.ts`: - -```ts -import { defineConfig } from '@chkit/core' -import { obsessiondb } from '@chkit/plugin-obsessiondb' - -export default defineConfig({ - schema: './src/db/schema/**/*.ts', - outDir: './chkit', - plugins: [obsessiondb()], - clickhouse: { - url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', - }, -}) -``` + + + ```sh + bun add -d @chkit/plugin-obsessiondb + ``` + + Register it in your `clickhouse.config.ts`: + + ```ts + import { defineConfig } from '@chkit/core' + import { obsessiondb } from '@chkit/plugin-obsessiondb' + + export default defineConfig({ + schema: './src/db/schema/**/*.ts', + outDir: './chkit', + plugins: [obsessiondb()], + clickhouse: { + url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123', + }, + }) + ``` + + + The plugin ships inside `chkit-py` — nothing extra to install. Register it in your `clickhouse.config.py`: + + ```python + import os + + from chkit import define_config + from chkit_plugin_obsessiondb import obsessiondb + + config = define_config( + { + "schema": "./src/db/schema/**/*.py", + "outDir": "./chkit", + "plugins": [obsessiondb()], + "clickhouse": { + "url": os.environ.get("CLICKHOUSE_URL", "http://localhost:8123"), + }, + } + ) + ``` + + The plugin hooks into `generate`, `migrate`, `status`, `drift`, `check`, and `query`. diff --git a/apps/docs/src/content/docs/plugins/backfill.md b/apps/docs/src/content/docs/plugins/backfill.mdx similarity index 86% rename from apps/docs/src/content/docs/plugins/backfill.md rename to apps/docs/src/content/docs/plugins/backfill.mdx index 91024329..e0a9645e 100644 --- a/apps/docs/src/content/docs/plugins/backfill.md +++ b/apps/docs/src/content/docs/plugins/backfill.mdx @@ -8,6 +8,8 @@ sidebar: variant: caution --- +import { Tabs, TabItem } from '@astrojs/starlight/components'; + This document covers practical usage of the optional `backfill` plugin. :::caution[Alpha] @@ -50,11 +52,13 @@ When ObsessionDB is the active target (logged in with a service selected), `plan ## Plugin setup -In `clickhouse.config.ts`, register `backfill(...)` from `@chkit/plugin-backfill`. +Register `backfill(...)` in your config's `plugins` array. -```ts -import { defineConfig } from '@chkit/core' -import { backfill } from '@chkit/plugin-backfill' + + + ```ts + import { defineConfig } from '@chkit/core' + import { backfill } from '@chkit/plugin-backfill' export default defineConfig({ schema: './src/db/schema/**/*.ts', @@ -75,7 +79,31 @@ export default defineConfig({ }), ], }) -``` + ``` + + + ```python + from chkit import define_config + from chkit_plugin_backfill import backfill + + config = define_config( + { + "schema": "./src/db/schema/**/*.py", + "plugins": [ + backfill( + { + "stateDir": "./chkit/backfill", + "chunkHours": 6, + "timeColumn": "created_at", + } + ), + ], + } + ) + ``` + The Python plugin is at full parity with the TypeScript plugin: `plan`, `run`, `resume`, `status`, `cancel`, `doctor`, and managed `submit` all work end-to-end. + + The `run` and `resume` commands execute SQL against ClickHouse when a connection is configured. Configure `clickhouse` at the top level of `clickhouse.config.ts`: @@ -113,26 +141,52 @@ The backfill plugin needs a time column to build WHERE clauses for each chunk. I Schema-level configuration is the recommended approach when different tables use different time columns. Define it directly in the `table()` call: -```ts -import { table } from '@chkit/core' - -export const events = table({ - database: 'app', - name: 'events', - columns: [ - { name: 'event_time', type: 'DateTime' }, - { name: 'id', type: 'UInt64' }, - ], - engine: 'MergeTree', - orderBy: ['event_time', 'id'], - primaryKey: ['event_time', 'id'], - plugins: { - backfill: { timeColumn: 'event_time' }, - }, -}) -``` - -This requires importing `@chkit/plugin-backfill` somewhere in the project (typically in `clickhouse.config.ts`) to activate the type augmentation. The `plugins.backfill` object is fully typed — autocomplete and type errors work as expected. + + + ```ts + import { table } from '@chkit/core' + + export const events = table({ + database: 'app', + name: 'events', + columns: [ + { name: 'event_time', type: 'DateTime' }, + { name: 'id', type: 'UInt64' }, + ], + engine: 'MergeTree', + orderBy: ['event_time', 'id'], + primaryKey: ['event_time', 'id'], + plugins: { + backfill: { timeColumn: 'event_time' }, + }, + }) + ``` + + This requires importing `@chkit/plugin-backfill` somewhere in the project (typically in `clickhouse.config.ts`) to activate the type augmentation. The `plugins.backfill` object is fully typed — autocomplete and type errors work as expected. + + + ```python + from chkit import table + + events = table( + database="app", + name="events", + columns=[ + {"name": "event_time", "type": "DateTime"}, + {"name": "id", "type": "UInt64"}, + ], + engine="MergeTree", + order_by=["event_time", "id"], + primary_key=["event_time", "id"], + plugins={ + "backfill": {"timeColumn": "event_time"}, + }, + ) + ``` + + In Python `plugins` is a plain dict; the field is metadata only and never affects diffing. + + ## Options diff --git a/apps/docs/src/content/docs/plugins/codegen.md b/apps/docs/src/content/docs/plugins/codegen.mdx similarity index 67% rename from apps/docs/src/content/docs/plugins/codegen.md rename to apps/docs/src/content/docs/plugins/codegen.mdx index fd704b2e..c4259446 100644 --- a/apps/docs/src/content/docs/plugins/codegen.md +++ b/apps/docs/src/content/docs/plugins/codegen.mdx @@ -5,11 +5,13 @@ sidebar: order: 2 --- -This document covers practical usage of the optional `codegen` plugin. +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +This document covers practical usage of the optional `codegen` plugin. In TypeScript it emits row types (plus optional Zod schemas, ingest helpers, and a runtime migration module); in Python it emits one Pydantic model per table and dictionary — Pydantic covers both static typing and runtime validation, so there is no separate Zod-style output. ## What it does -- Generates deterministic TypeScript row types from chkit schema definitions. +- Generates deterministic row types from chkit schema definitions — TypeScript interfaces, or Pydantic models in Python. - Generates a typed interface (and optional Zod schema) for each `dictionary()` from its `attributes` — dictionaries are always included, regardless of `includeViews`. - Optionally generates Zod schemas from the same definitions. - Optionally generates typed ingestion functions for inserting rows into ClickHouse tables. Generated ingest helpers gzip-compress request bodies by default and can opt out per call. @@ -32,41 +34,73 @@ The plugin is designed so your existing chkit workflow can stay the same. ## Plugin setup -In `clickhouse.config.ts`, register `codegen(...)` from `@chkit/plugin-codegen`. - -:::note -`zod` is a peer dependency (`^4.0.0`). Install it alongside the plugin — generated Zod schemas (`emitZod: true`) import `zod` from your project, so they resolve against your own copy rather than a bundled one. - -```sh -bun add -d @chkit/plugin-codegen zod -``` -::: - -Recommended typed setup: - -```ts -import { defineConfig } from '@chkit/core' -import { codegen } from '@chkit/plugin-codegen' - -export default defineConfig({ - schema: './src/db/schema/**/*.ts', - plugins: [ - codegen({ - outFile: './src/generated/chkit-types.ts', - emitZod: false, - emitIngest: false, - ingestOutFile: './src/generated/chkit-ingest.ts', - emitMigrations: false, - migrationsOutFile: './src/generated/chkit-migrations.ts', - tableNameStyle: 'pascal', - bigintMode: 'string', - includeViews: false, - runOnGenerate: true, - failOnUnsupportedType: true, - }), - ], -}) -``` +Register `codegen(...)` in your config's `plugins` array. + + + + :::note + `zod` is a peer dependency (`^4.0.0`). Install it alongside the plugin — generated Zod schemas (`emitZod: true`) import `zod` from your project, so they resolve against your own copy rather than a bundled one. + + ```sh + bun add -d @chkit/plugin-codegen zod + ``` + ::: + + Recommended typed setup: + + ```ts + import { defineConfig } from '@chkit/core' + import { codegen } from '@chkit/plugin-codegen' + + export default defineConfig({ + schema: './src/db/schema/**/*.ts', + plugins: [ + codegen({ + outFile: './src/generated/chkit-types.ts', + emitZod: false, + emitIngest: false, + ingestOutFile: './src/generated/chkit-ingest.ts', + emitMigrations: false, + migrationsOutFile: './src/generated/chkit-migrations.ts', + tableNameStyle: 'pascal', + bigintMode: 'string', + includeViews: false, + runOnGenerate: true, + failOnUnsupportedType: true, + }), + ], + }) + ``` + + + The plugin ships inside `chkit-py` — nothing extra to install. + + ```python + from chkit import define_config + from chkit_plugin_codegen import codegen + + config = define_config( + { + "schema": "./src/db/schema/**/*.py", + "plugins": [ + codegen( + { + "outFile": "./src/generated/chkit_models.py", + "tableNameStyle": "pascal", + "bigintMode": "int", + "includeViews": False, + "runOnGenerate": True, + "failOnUnsupportedType": True, + } + ), + ], + } + ) + ``` + + The output is a single module with one Pydantic model per table and dictionary. The `emitZod` / `emitIngest` / `emitMigrations` emitters are TypeScript-only by design — Pydantic already provides runtime validation, and the ingest/migration-module emitters target JS runtimes. + + ## Options @@ -77,11 +111,13 @@ export default defineConfig({ - `emitMigrations` (default: `false`) - `migrationsOutFile` (default: `./src/generated/chkit-migrations.ts`) - `tableNameStyle` (default: `pascal`) values: `pascal | camel | raw` -- `bigintMode` (default: `string`) values: `string | bigint` +- `bigintMode` (default: `string`) values: `string | bigint` — in Python the values are `int | str` (default `int`; the TS spellings are accepted as aliases) - `includeViews` (default: `false`) - `runOnGenerate` (default: `true`) - `failOnUnsupportedType` (default: `true`) +Python supports `outFile` (default `./src/generated/chkit_models.py`), `tableNameStyle`, `bigintMode`, `includeViews`, `runOnGenerate`, and `failOnUnsupportedType`; the `emit*` options are TypeScript-only (see Plugin setup above). + Invalid option values fail fast at startup via plugin config validation. ## Commands diff --git a/apps/docs/src/content/docs/plugins/overview.md b/apps/docs/src/content/docs/plugins/overview.mdx similarity index 50% rename from apps/docs/src/content/docs/plugins/overview.md rename to apps/docs/src/content/docs/plugins/overview.mdx index f58a2058..dcabb115 100644 --- a/apps/docs/src/content/docs/plugins/overview.md +++ b/apps/docs/src/content/docs/plugins/overview.mdx @@ -5,23 +5,47 @@ sidebar: order: 1 --- -Plugins extend chkit with capabilities that don't belong in the core CLI — code generation, schema introspection, data backfill, ObsessionDB integration, and anything else you want to bolt on. They're regular npm packages that you register in `clickhouse.config.ts`: - -```ts -import { defineConfig } from '@chkit/core' -import { codegen } from '@chkit/plugin-codegen' -import { pull } from '@chkit/plugin-pull' - -export default defineConfig({ - schema: './src/db/schema/**/*.ts', - outDir: './chkit', - plugins: [ - codegen({ outFile: './src/generated/chkit-types.ts' }), - pull({ outFile: './src/db/schema/pulled.ts' }), - ], - // ... -}) -``` +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +Plugins extend chkit with capabilities that don't belong in the core CLI — code generation, schema introspection, data backfill, ObsessionDB integration, and anything else you want to bolt on. In TypeScript they're regular npm packages; in Python they ship inside `chkit-py`. Either way, you register them in your config: + + + + ```ts + import { defineConfig } from '@chkit/core' + import { codegen } from '@chkit/plugin-codegen' + import { pull } from '@chkit/plugin-pull' + + export default defineConfig({ + schema: './src/db/schema/**/*.ts', + outDir: './chkit', + plugins: [ + codegen({ outFile: './src/generated/chkit-types.ts' }), + pull({ outFile: './src/db/schema/pulled.ts' }), + ], + // ... + }) + ``` + + + ```python + from chkit import define_config + from chkit_plugin_codegen import codegen + + config = define_config( + { + "schema": "./src/db/schema/**/*.py", + "outDir": "./chkit", + "plugins": [ + codegen({"outFile": "./src/generated/chkit_models.py"}), + ], + # ... + } + ) + ``` + In Python, `pull` is a built-in CLI command ([`chkit pull`](/cli/pull/)) rather than a registered plugin. + + ## How plugins hook in @@ -33,8 +57,8 @@ You can author your own plugins; the existing official plugins are the reference ## Official plugins -If you deploy to [ObsessionDB](https://obsessiondb.com), start at the dedicated [ObsessionDB section](/obsessiondb/overview/) — `@chkit/plugin-obsessiondb` is documented there as a first-class integration rather than as a plain plugin. +If you deploy to [ObsessionDB](https://obsessiondb.com), start at the dedicated [ObsessionDB section](/obsessiondb/overview/) — `@chkit/plugin-obsessiondb` (Python: `chkit_plugin_obsessiondb`) is documented there as a first-class integration rather than as a plain plugin. -- [`@chkit/plugin-codegen`](/plugins/codegen/) — TypeScript row types and optional Zod schemas, generated from your schema files. -- [`@chkit/plugin-pull`](/plugins/pull/) — introspect a live ClickHouse database into local schema files. Useful for adopting chkit on an existing database. +- [`@chkit/plugin-codegen`](/plugins/codegen/) — TypeScript row types and optional Zod schemas (Python: Pydantic models), generated from your schema files. +- [`@chkit/plugin-pull`](/plugins/pull/) — introspect a live ClickHouse database into local schema files. Useful for adopting chkit on an existing database. Built into the Python CLI as `chkit pull`. - [`@chkit/plugin-backfill`](/plugins/backfill/) — time-windowed data backfill with checkpoints, for materialized views and historical data loads. diff --git a/apps/docs/src/content/docs/plugins/pull.md b/apps/docs/src/content/docs/plugins/pull.mdx similarity index 60% rename from apps/docs/src/content/docs/plugins/pull.md rename to apps/docs/src/content/docs/plugins/pull.mdx index ec2b93a4..03d6a28f 100644 --- a/apps/docs/src/content/docs/plugins/pull.md +++ b/apps/docs/src/content/docs/plugins/pull.mdx @@ -5,14 +5,16 @@ sidebar: order: 3 --- -This document covers practical usage of the optional `pull` plugin. +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +This document covers practical usage of `pull`. In TypeScript it is an optional plugin (`@chkit/plugin-pull`); in Python, [`chkit pull`](/cli/pull/) is a built-in CLI command — same introspection, no registration needed. ## What it does - Connects to a live ClickHouse instance and introspects table metadata (columns, engines, indexes, projections, partitioning, TTL, settings). - Introspects views and materialized views (including `TO` clause parsing). - Introspects dictionaries (attributes — including `HIERARCHICAL`/`BIDIRECTIONAL`/`INJECTIVE`/`IS_OBJECT_ID` modifiers — primary key, `SOURCE`/`LAYOUT`/`LIFETIME`/`RANGE`/`SETTINGS`), preserving ClickHouse's `[HIDDEN]` password redaction — see [Credential handling](#credential-handling-hidden-passwords). -- Generates a deterministic TypeScript schema file using `@chkit/core` builders. +- Generates a deterministic schema file using the chkit builders — TypeScript (`@chkit/core`) or Python (`chkit`). - Supports filtering by database and dry-run previews. ## How it fits your workflow @@ -20,29 +22,40 @@ This document covers practical usage of the optional `pull` plugin. The plugin is designed for bootstrapping a chkit project from an existing ClickHouse deployment. - [`chkit pull`](/cli/pull/) (alias for `chkit plugin pull schema`): - - Connects to ClickHouse, introspects all schema objects, and writes a TypeScript schema file. + - Connects to ClickHouse, introspects all schema objects, and writes a schema file in your project's language. - Generated file works directly with [`chkit generate`](/cli/generate/) and [`chkit check`](/cli/check/). - Dry-run mode previews the output without writing to disk. ## Plugin setup -In `clickhouse.config.ts`, register `pull(...)` from `@chkit/plugin-pull`. - -```ts -import { defineConfig } from '@chkit/core' -import { pull } from '@chkit/plugin-pull' - -export default defineConfig({ - schema: './src/db/schema/**/*.ts', - plugins: [ - pull({ - outFile: './src/db/schema/pulled.ts', - databases: ['analytics'], - overwrite: false, - }), - ], -}) -``` + + + In `clickhouse.config.ts`, register `pull(...)` from `@chkit/plugin-pull`. + + ```ts + import { defineConfig } from '@chkit/core' + import { pull } from '@chkit/plugin-pull' + + export default defineConfig({ + schema: './src/db/schema/**/*.ts', + plugins: [ + pull({ + outFile: './src/db/schema/pulled.ts', + databases: ['analytics'], + overwrite: false, + }), + ], + }) + ``` + + + No setup — `chkit pull` is built into the CLI. Configure behavior with flags: + + ```sh + chkit pull --out-file src/db/schema/pulled.py --database analytics + ``` + + ## Options @@ -61,7 +74,7 @@ Invalid option values fail fast at startup via plugin config validation. Useful flags: - `--out-file ` — Override output file path. -- `--database ` — Filter to databases (comma-separated or repeated). +- `--database ` — Filter to databases (repeat the flag for several). - `--dryrun` — Preview output without writing. - `--force` / `--overwrite` — Overwrite existing output file. @@ -69,10 +82,12 @@ Exit codes: 0 (success), 1 (runtime error), 2 (config error). ## Generated output format -The plugin produces a TypeScript module that imports builders from `@chkit/core` and exports a default schema. +The output is a schema module that imports the chkit builders and exports the pulled definitions. -```ts -import { schema, table, view, materializedView } from '@chkit/core' + + + ```ts + import { schema, table, view, materializedView } from '@chkit/core' // Pulled from live ClickHouse metadata via chkit plugin pull schema @@ -103,7 +118,43 @@ const app_events_mv = materializedView({ }) export default schema(app_events, app_events_view, app_events_mv) -``` + ``` + + + ```python + # Schema pulled from live ClickHouse metadata via `chkit pull`. + from chkit import ColumnDefinition, TableRef, materialized_view, schema, table, view + + app_events = table( + database="app", + name="events", + engine="MergeTree()", + columns=[ + ColumnDefinition(name="id", type="UInt64"), + ColumnDefinition(name="received_at", type="DateTime64(3)", default="fn:now64(3)"), + ], + primary_key=["id"], + order_by=["id"], + partition_by="toYYYYMM(received_at)", + ) + + app_events_view = view( + database="app", + name="events_view", + as_="SELECT id FROM app.events", + ) + + app_events_mv = materialized_view( + database="app", + name="events_mv", + to=TableRef(database="app", name="events_rollup"), + as_="SELECT id, count() AS c FROM app.events GROUP BY id", + ) + + definitions = schema(app_events, app_events_view, app_events_mv) + ``` + + Tables may also include `uniqueKey`, `ttl`, `settings`, `indexes`, and `projections` when present in the source metadata. @@ -111,9 +162,11 @@ Tables may also include `uniqueKey`, `ttl`, `settings`, `indexes`, and `projecti By default, ClickHouse redacts inline `SOURCE(...)` passwords to `[HIDDEN]` on introspection (`system.tables.create_table_query`, `SHOW CREATE DICTIONARY`), and chkit does not attempt to work around that. When a pulled dictionary's `source` contains `[HIDDEN]`, `chkit pull` prints a console warning (and includes it in a `warnings` array in `--json` output), and the generated file emits the source verbatim with a leading comment: -```ts -// NOTE: password redacted by ClickHouse — replace '[HIDDEN]' with your credential (e.g. process.env.X). -const default_users_dict = dictionary({ + + + ```ts + // NOTE: password redacted by ClickHouse — replace '[HIDDEN]' with your credential (e.g. process.env.X). + const default_users_dict = dictionary({ database: "default", name: "users_dict", attributes: [ @@ -125,13 +178,41 @@ const default_users_dict = dictionary({ layout: "HASHED()", lifetime: "300", }) -``` + ``` + + + ```python + # NOTE: password redacted by ClickHouse — replace '[HIDDEN]' with your credential. + default_users_dict = dictionary( + database="default", + name="users_dict", + attributes=[ + DictionaryAttribute(name="id", type="UInt64"), + DictionaryAttribute(name="name", type="String"), + ], + primary_key=["id"], + source="MYSQL(host 'db' port 3306 user 'reader' password '[HIDDEN]' db 'app' table 'users')", + layout="HASHED()", + lifetime="300", + ) + ``` + + Replace `[HIDDEN]` with a real credential — typically an environment-variable interpolation, matching how you'd author the dictionary by hand (see [Credentials in `source`](/schema/dsl-reference/#credentials-in-source)): -```ts -source: `MYSQL(host 'db' port 3306 user 'reader' password '${process.env.MYSQL_PASSWORD}' db 'app' table 'users')`, -``` + + + ```ts + source: `MYSQL(host 'db' port 3306 user 'reader' password '${process.env.MYSQL_PASSWORD}' db 'app' table 'users')`, + ``` + + + ```python + source=f"MYSQL(host 'db' port 3306 user 'reader' password '{os.environ['MYSQL_PASSWORD']}' db 'app' table 'users')", + ``` + + For round-trip fidelity without a manual edit, use [named collections](https://clickhouse.com/docs/operations/named-collections) on the ClickHouse side instead of an inline password — chkit does not require this, but it's the ClickHouse-native way to avoid the redaction entirely. diff --git a/apps/docs/src/content/docs/python/core-api.md b/apps/docs/src/content/docs/python/core-api.md new file mode 100644 index 00000000..c3359994 --- /dev/null +++ b/apps/docs/src/content/docs/python/core-api.md @@ -0,0 +1,150 @@ +--- +title: Python Core API +description: The chkit.core pipeline as library functions — loading, validation, diffing, planning, snapshots, and SQL rendering. +sidebar: + order: 3 +--- + +`chkit.core` exposes the pipeline underneath the CLI as plain functions over Pydantic models — load definitions, validate them, diff against a snapshot, and render SQL, all without the CLI. + +```python +from chkit import ( + load_schema_definitions, + validate_definitions, + plan_diff, + to_create_sql, + apply_on_cluster_to_plan, + define_config, + resolve_config, +) +from chkit.core import create_snapshot, canonicalize_definitions, assert_valid_definitions +``` + +## The pipeline + +`chkit generate` is, in essence, this sequence: + +```python +from chkit import load_schema_definitions, plan_diff + +old = snapshot.definitions # last applied state (empty list on first run) +new = load_schema_definitions("./src/db/schema/**/*.py") +plan = plan_diff(old, new) + +for op in plan.operations: + print(op.risk, op.type, op.sql) +``` + +Each stage is available on its own. + +## Loading definitions + +### `load_schema_definitions(schema_globs, *, cwd=None)` + +Resolves one glob (or a list of globs), imports each matching Python module, collects every module-level schema definition, and returns them canonicalized. Raises `SchemaLoaderError` when no files match (`NO_MATCH_MESSAGE`) and `ModuleLoadError` when a schema module fails to import. + +```python +definitions = load_schema_definitions("./src/db/schema/**/*.py") +``` + +### `collect_definitions_from_module(mod)` + +The lower-level collector: takes a module's `__dict__`-style mapping, walks its values (including nested lists and tuples), and returns the schema definitions found, deduplicated and canonicalized. + +## Canonicalization + +### `canonicalize_definitions(definitions)` + +Normalizes definitions into the canonical form the diff engine compares — engine normalization, key-clause splitting, codec canonicalization — deduplicates by identity, and sorts deterministically (tables, then views, then materialized views; then by database and name). Both sides of every diff are canonicalized first, so cosmetic differences (`'MergeTree'` vs `'MergeTree()'`, `['id, org_id']` vs `['id', 'org_id']`) never produce operations. + +### `definition_key(definition)` + +Returns the stable identity string `":."` used for deduplication and operation keys. + +## Validation + +### `validate_definitions(definitions)` + +Returns a list of `ValidationIssue` objects — empty when the definitions are valid. Each issue has a `code` (e.g. `duplicate_column_name`, `order_by_missing_column`, `codec_chain_must_end_with_general`), the offending object's `kind`, `database`, and `name`, and a human-readable `message`. + +### `assert_valid_definitions(definitions)` + +Same checks, but raises `ChxValidationError` (carrying the issue list as `.issues`) instead of returning them. The planner and SQL renderer call this internally, so invalid definitions cannot reach SQL generation. + +## Planning + +### `plan_diff(old_definitions, new_definitions)` + +Canonicalizes both sides, validates the new side, and returns a `MigrationPlan`: + +- `operations` — ordered `MigrationOperation` list; each has a `type` (e.g. `create_table`, `alter_table_add_column`, `drop_table`), a `key` identifying the object, a `risk` level, and the rendered `sql`. +- `risk_summary` — counts of `safe` / `caution` / `danger` operations. +- `rename_suggestions` — detected drop+add column pairs that look like renames, each with the `confirmation_sql` to apply the rename instead. + +Risk levels drive CLI behavior: `danger` operations (drops, table recreates) are blocked by `chkit migrate` unless `--allow-destructive` is passed. + +```python +plan = plan_diff(old, new) +if plan.risk_summary.danger > 0: + raise SystemExit("plan contains destructive operations") +``` + +## Snapshots + +### `create_snapshot(definitions)` + +Canonicalizes the definitions and returns a `SnapshotV1` — `version: 1`, a UTC `generated_at` timestamp, and the canonical definition list. Serialized with camelCase aliases it has the same JSON shape as the TypeScript `chkit/meta/snapshot.json`: + +```python +snapshot = create_snapshot(definitions) +payload = snapshot.model_dump(mode="json", by_alias=True) +``` + +Loading goes through the same model: `SnapshotV1.model_validate_json(text)` accepts snapshots written by either implementation. + +## SQL rendering + +### `to_create_sql(definition)` + +Renders a single definition to its `CREATE TABLE` / `CREATE VIEW` / `CREATE MATERIALIZED VIEW` DDL string. Validates first, so it raises `ChxValidationError` on an invalid definition. + +### `apply_on_cluster_to_plan(plan, cluster)` + +Post-pass that stamps `ON CLUSTER ''` into every DDL statement of a plan (and into each rename suggestion's confirmation SQL). No-op when `cluster` is `None`. This is how `clickhouse.cluster` from the config takes effect — the planner itself stays cluster-agnostic. + +## Configuration + +### `define_config(config)` + +Anchors the config object in `clickhouse.config.py`. Accepts a `ChxUserConfig` instance, a plain dict (validated through Pydantic on entry), or a callable `(env: ChxConfigEnv) -> config` for dynamic per-command configs (see the [CI/CD guide](/guides/ci-cd/)): + +```python +import os + +from chkit import define_config + +config = define_config( + { + "schema": "./src/db/schema/**/*.py", + "outDir": "./chkit", + "clickhouse": { + "url": os.environ.get("CLICKHOUSE_URL", "http://localhost:8123"), + "username": os.environ.get("CLICKHOUSE_USER", "default"), + "password": os.environ.get("CLICKHOUSE_PASSWORD", ""), + "database": os.environ.get("CLICKHOUSE_DB", "default"), + }, + } +) +``` + +### `resolve_config(config)` + +Fills in defaults and returns a `ChxResolvedConfig`: `out_dir` defaults to `./chkit`, `migrations_dir` to `/migrations`, `meta_dir` to `/meta`; the `check` flags default to `true` and `safety.allowDestructive` to `false`; ClickHouse credentials default to `default` / empty password / database `default`. A `clickhouse.cluster` value is validated here (identifier or `{macro}` form) — invalid names fail fast with a `ValueError`. + +The option keys and defaults match the TypeScript config — see the [Configuration Overview](/configuration/overview/). + +## Related + +- [Schema DSL Reference](/schema/dsl-reference/) — building the definitions this pipeline consumes, with synced TypeScript/Python examples. +- [CLI: `chkit generate`](/cli/generate/) — the CLI wrapper around `plan_diff`. +- [CLI: `chkit migrate`](/cli/migrate/) — how plans are applied and journaled. diff --git a/apps/docs/src/content/docs/python/overview.md b/apps/docs/src/content/docs/python/overview.md new file mode 100644 index 00000000..1ddee478 --- /dev/null +++ b/apps/docs/src/content/docs/python/overview.md @@ -0,0 +1,62 @@ +--- +title: Python Overview +description: chkit-py is the Python port of chkit — same schema DSL, diff engine, and migration pipeline, importable as chkit. +sidebar: + order: 1 +--- + +`chkit-py` is the Python port of chkit — the same schema DSL, canonicalization, diff engine, migration planner, and SQL rendering, written in strict, fully typed Python. + +## Install + +```sh +pip install chkit-py +chkit --help +``` + +The package is named `chkit-py` on PyPI; the import name is `chkit`. + +## Quickstart + +```sh +pip install chkit-py +chkit init # scaffold clickhouse.config.py + example schema +chkit generate --name init # diff schema vs snapshot, write migrations/*.sql +chkit migrate --apply # apply pending migrations +chkit status # show applied / pending counts +chkit check --strict # CI gate (pending, drift, checksum) +``` + +The CLI commands, flags, exit codes, and `--json` output match the TypeScript CLI — the [CLI Reference](/cli/overview/) applies to both. Config lives in `clickhouse.config.py` instead of `clickhouse.config.ts`, and schema files are Python modules instead of TypeScript modules. + +## Design + +- **Type safety first.** Every public surface is annotated. Ships clean under `mypy --strict` and `pyright` strict mode. +- **Pydantic v2 models.** All schema objects are frozen, validated at construction, and reject unknown fields — typos surface as validation errors instead of being silently ignored. +- **Imperative core.** Pure functions over data; minimal classes outside of Pydantic models and the CLI shell. +- **No magic.** No dynamic imports and no runtime introspection of user code beyond what Pydantic provides. + +## Interoperability with TypeScript chkit + +Both implementations produce the same artifacts, so a project (or a team) can mix them: + +- **Snapshots** — models serialize with the same camelCase JSON field names as `@chkit/core`, so `chkit/meta/snapshot.json` is readable by either implementation. +- **Journal** — migrations are recorded in the same ClickHouse `_chkit_migrations` table with the same schema and checksums. +- **SQL** — the planner and renderer emit the same DDL for the same schema, including `ON CLUSTER` stamping when `clickhouse.cluster` is set. + +## Differences from the TypeScript version + +The CLI, plugin set, and backfill engine are at full parity. The only remaining differences are by design (Python convention or ecosystem difference): + +- `chkit skills` proxy and the `create-chkit` scaffolder — use `chkit init` instead. +- `deps.ts`-style dependency auto-install — install packages explicitly with `pip`. + +## These pages + +This section covers what is Python-specific: install, interoperability, and the [Core API](/python/core-api/) — loading, validation, diffing, planning, snapshots, and SQL rendering as library functions. The schema DSL itself is documented once for both languages, with synced language tabs, in the [Schema DSL Reference](/schema/dsl-reference/). + +## Related + +- [Schema DSL Reference](/schema/dsl-reference/) — `table()`, `view()`, `materialized_view()`, `dictionary()` with TypeScript/Python tabs. +- [CLI Reference](/cli/overview/) — commands and flags, shared by both implementations. +- [Configuration Overview](/configuration/overview/) — config keys with tabbed examples, identical modulo file extension. diff --git a/apps/docs/src/content/docs/schema/dsl-reference.md b/apps/docs/src/content/docs/schema/dsl-reference.mdx similarity index 58% rename from apps/docs/src/content/docs/schema/dsl-reference.md rename to apps/docs/src/content/docs/schema/dsl-reference.mdx index a83c9979..8236164c 100644 --- a/apps/docs/src/content/docs/schema/dsl-reference.md +++ b/apps/docs/src/content/docs/schema/dsl-reference.mdx @@ -1,87 +1,159 @@ --- title: Schema DSL Reference -description: Complete reference for chkit schema definition functions, column types, and table options. +description: Complete reference for chkit schema definition functions, column types, and table options — in TypeScript and Python. sidebar: order: 3 --- -Schema files are TypeScript files that export definitions using functions from `@chkit/core`. All exported definitions are collected when chkit loads schema files matched by the `schema` glob in your [configuration](/configuration/overview/). - -```ts -import { schema, table, view, materializedView, dictionary } from '@chkit/core' -``` +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +Schema files export definitions using functions from `@chkit/core` (TypeScript) or `chkit` (Python, via [chkit-py](/python/overview/)). All exported definitions are collected when chkit loads schema files matched by the `schema` glob in your [configuration](/configuration/overview/). The two implementations share every field's semantics — pick your language once and the whole page follows. + + + + ```ts + import { schema, table, view, materializedView, dictionary } from '@chkit/core' + ``` + + + ```python + from chkit import schema, table, view, materialized_view, dictionary + ``` + + + +:::note[Python calling conventions] +The Python DSL accepts both the TypeScript camelCase names and snake_case equivalents everywhere — `primary_key` or `primaryKey`, `renamed_from` or `renamedFrom`, `maxRows` or `max_rows` — so examples port with their keys unchanged. Because `as` is a Python keyword, `view()` and `materialized_view()` take the SELECT body as `as_`. Columns, indexes, projections, and attributes accept plain dicts (validated on entry) or model instances (`ColumnDefinition`, `SkipIndexSet`, ...). Field tables on this page use the camelCase names. +::: ## `schema()` Groups definitions into a single array for export. -```ts -schema(...definitions: SchemaDefinition[]): SchemaDefinition[] -``` - -```ts -export default schema(users, events) -``` - -You can also export definitions individually -- any exported value with a valid `kind` is discovered automatically. + + + ```ts + export default schema(users, events) + ``` + Any exported value with a valid `kind` is also discovered automatically. + + + ```python + definitions = schema(users, events) + ``` + Any module-level definition is also discovered automatically, including definitions nested in lists or tuples. + + ## `table()` Creates a table definition. -```ts -table(input: Omit): TableDefinition -``` - **Minimal example:** -```ts -import { schema, table } from '@chkit/core' - -const users = table({ - database: 'app', - name: 'users', - columns: [ - { name: 'id', type: 'UInt64' }, - { name: 'email', type: 'String' }, - ], - engine: 'MergeTree', - primaryKey: ['id'], - orderBy: ['id'], -}) - -export default schema(users) -``` + + + ```ts + import { schema, table } from '@chkit/core' + + const users = table({ + database: 'app', + name: 'users', + columns: [ + { name: 'id', type: 'UInt64' }, + { name: 'email', type: 'String' }, + ], + engine: 'MergeTree', + primaryKey: ['id'], + orderBy: ['id'], + }) + + export default schema(users) + ``` + + + ```python + from chkit import schema, table + + users = table( + database="app", + name="users", + columns=[ + {"name": "id", "type": "UInt64"}, + {"name": "email", "type": "String"}, + ], + engine="MergeTree", + primary_key=["id"], + order_by=["id"], + ) + + definitions = schema(users) + ``` + + **Comprehensive example (all features):** -```ts -const events = table({ - database: 'analytics', - name: 'events', - columns: [ - { name: 'id', type: 'UInt64' }, - { name: 'org_id', type: 'String' }, - { name: 'source', type: 'LowCardinality(String)' }, - { name: 'payload', type: 'String', nullable: true }, - { name: 'received_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, - { name: 'status', type: 'String', default: 'pending', comment: 'Event processing status' }, - ], - engine: 'MergeTree', - primaryKey: ['id'], - orderBy: ['org_id', 'received_at', 'id'], - partitionBy: 'toYYYYMM(received_at)', - ttl: 'received_at + INTERVAL 90 DAY', - settings: { index_granularity: 8192 }, - indexes: [ - { name: 'idx_source', expression: 'source', type: 'set', maxRows: 0, granularity: 1 }, - ], - projections: [ - { name: 'p_recent', query: 'SELECT id ORDER BY received_at DESC LIMIT 10' }, - ], - comment: 'Raw ingested events', -}) -``` + + + ```ts + const events = table({ + database: 'analytics', + name: 'events', + columns: [ + { name: 'id', type: 'UInt64' }, + { name: 'org_id', type: 'String' }, + { name: 'source', type: 'LowCardinality(String)' }, + { name: 'payload', type: 'String', nullable: true }, + { name: 'received_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, + { name: 'status', type: 'String', default: 'pending', comment: 'Event processing status' }, + ], + engine: 'MergeTree', + primaryKey: ['id'], + orderBy: ['org_id', 'received_at', 'id'], + partitionBy: 'toYYYYMM(received_at)', + ttl: 'received_at + INTERVAL 90 DAY', + settings: { index_granularity: 8192 }, + indexes: [ + { name: 'idx_source', expression: 'source', type: 'set', maxRows: 0, granularity: 1 }, + ], + projections: [ + { name: 'p_recent', query: 'SELECT id ORDER BY received_at DESC LIMIT 10' }, + ], + comment: 'Raw ingested events', + }) + ``` + + + ```python + events = table( + database="analytics", + name="events", + columns=[ + {"name": "id", "type": "UInt64"}, + {"name": "org_id", "type": "String"}, + {"name": "source", "type": "LowCardinality(String)"}, + {"name": "payload", "type": "String", "nullable": True}, + {"name": "received_at", "type": "DateTime64(3)", "default": "fn:now64(3)"}, + {"name": "status", "type": "String", "default": "pending", "comment": "Event processing status"}, + ], + engine="MergeTree", + primary_key=["id"], + order_by=["org_id", "received_at", "id"], + partition_by="toYYYYMM(received_at)", + ttl="received_at + INTERVAL 90 DAY", + settings={"index_granularity": 8192}, + indexes=[ + {"name": "idx_source", "expression": "source", "type": "set", "maxRows": 0, "granularity": 1}, + ], + projections=[ + {"name": "p_recent", "query": "SELECT id ORDER BY received_at DESC LIMIT 10"}, + ], + comment="Raw ingested events", + ) + ``` + + ### Required fields @@ -167,10 +239,20 @@ See the ClickHouse [data types reference](https://clickhouse.com/docs/sql-refere When `true`, the column type is wrapped in `Nullable(...)` in the generated SQL. -```ts -{ name: 'payload', type: 'String', nullable: true } -// SQL: `payload` Nullable(String) -``` + + + ```ts + { name: 'payload', type: 'String', nullable: true } + // SQL: `payload` Nullable(String) + ``` + + + ```python + {"name": "payload", "type": "String", "nullable": True} + # SQL: `payload` Nullable(String) + ``` + + ### `default` (string | number | boolean, optional) @@ -180,10 +262,20 @@ Default value for the column. - **Number/boolean** values are rendered literally: `default: 0` produces `DEFAULT 0` - **`fn:` prefix** -- for function-call defaults, prefix the string with `fn:` to emit a raw SQL expression: -```ts -{ name: 'received_at', type: 'DateTime64(3)', default: 'fn:now64(3)' } -// SQL: `received_at` DateTime64(3) DEFAULT now64(3) -``` + + + ```ts + { name: 'received_at', type: 'DateTime64(3)', default: 'fn:now64(3)' } + // SQL: `received_at` DateTime64(3) DEFAULT now64(3) + ``` + + + ```python + {"name": "received_at", "type": "DateTime64(3)", "default": "fn:now64(3)"} + # SQL: `received_at` DateTime64(3) DEFAULT now64(3) + ``` + + ### `comment` (string, optional) @@ -197,14 +289,28 @@ Previous column name for rename tracking. See [Rename support](#rename-support). Sets the column compression codec, rendered as a `CODEC(...)` clause. A codec is an object with a `kind`, or an **array** forming a chain (zero or more preprocessors followed by exactly one general codec). -```ts -columns: [ - { name: 'ts', type: 'DateTime64(3)', codec: { kind: 'Delta', size: 4 } }, - { name: 'amount', type: 'Float64', codec: { kind: 'ZSTD', level: 3 } }, - // chain: preprocessor then general codec - { name: 'seq', type: 'UInt64', codec: [{ kind: 'DoubleDelta' }, { kind: 'LZ4HC', level: 9 }] }, -] -``` + + + ```ts + columns: [ + { name: 'ts', type: 'DateTime64(3)', codec: { kind: 'Delta', size: 4 } }, + { name: 'amount', type: 'Float64', codec: { kind: 'ZSTD', level: 3 } }, + // chain: preprocessor then general codec + { name: 'seq', type: 'UInt64', codec: [{ kind: 'DoubleDelta' }, { kind: 'LZ4HC', level: 9 }] }, + ] + ``` + + + ```python + columns=[ + {"name": "ts", "type": "DateTime64(3)", "codec": {"kind": "Delta", "size": 4}}, + {"name": "amount", "type": "Float64", "codec": {"kind": "ZSTD", "level": 3}}, + # chain: preprocessor then general codec + {"name": "seq", "type": "UInt64", "codec": [{"kind": "DoubleDelta"}, {"kind": "LZ4HC", "level": 9}]}, + ] + ``` + + **General codecs** (the compressor; at most one, and it must come last in a chain): @@ -223,13 +329,23 @@ columns: [ **Raw escape hatch** — for codecs not yet typed (new ClickHouse versions, unusual arg shapes), pass the inner expression through verbatim: -```ts -{ name: 'blob', type: 'String', codec: { kind: 'raw', expression: 'T64, LZ4' } } -// → CODEC(T64, LZ4) -``` + + + ```ts + { name: 'blob', type: 'String', codec: { kind: 'raw', expression: 'T64, LZ4' } } + // → CODEC(T64, LZ4) + ``` + + + ```python + {"name": "blob", "type": "String", "codec": {"kind": "raw", "expression": "T64, LZ4"}} + # → CODEC(T64, LZ4) + ``` + + :::caution -Use the typed `{ kind: 'ZSTD', level: 3 }` shape — an unrecognized shape such as `codec: { general: 'ZSTD' }` is **not** a valid `ColumnCodecSpec` and renders an empty `CODEC()`, silently shipping a column with no codec. +Use the typed `{ kind: 'ZSTD', level: 3 }` shape — an unrecognized shape such as `codec: { general: 'ZSTD' }` is **not** a valid `ColumnCodecSpec` and renders an empty `CODEC()`, silently shipping a column with no codec. (The Python port rejects unknown codec keys at validation time.) ::: Codec chains are validated (see [Validation rules](#validation-rules)): a chain must be non-empty, contain at most one general codec, and end with the general codec. @@ -255,21 +371,43 @@ Type-specific fields: | `tokenbf_v1` | `sizeBytes`, `hashFunctions`, `randomSeed` (all `number`) | — | Maps to `tokenbf_v1(size_bytes, n_hash, seed)` | | `ngrambf_v1` | `ngramSize`, `sizeBytes`, `hashFunctions`, `randomSeed` (all `number`) | — | Maps to `ngrambf_v1(n, size_bytes, n_hash, seed)` | -```ts -indexes: [ - { name: 'idx_source', expression: 'source', type: 'set', maxRows: 0, granularity: 1 }, - { name: 'idx_ts', expression: 'received_at', type: 'minmax', granularity: 3 }, - { - name: 'idx_body', - expression: 'body', - type: 'tokenbf_v1', - sizeBytes: 256, - hashFunctions: 2, - randomSeed: 0, - granularity: 1, - }, -] -``` + + + ```ts + indexes: [ + { name: 'idx_source', expression: 'source', type: 'set', maxRows: 0, granularity: 1 }, + { name: 'idx_ts', expression: 'received_at', type: 'minmax', granularity: 3 }, + { + name: 'idx_body', + expression: 'body', + type: 'tokenbf_v1', + sizeBytes: 256, + hashFunctions: 2, + randomSeed: 0, + granularity: 1, + }, + ] + ``` + + + ```python + indexes=[ + {"name": "idx_source", "expression": "source", "type": "set", "maxRows": 0, "granularity": 1}, + {"name": "idx_ts", "expression": "received_at", "type": "minmax", "granularity": 3}, + { + "name": "idx_body", + "expression": "body", + "type": "tokenbf_v1", + "sizeBytes": 256, + "hashFunctions": 2, + "randomSeed": 0, + "granularity": 1, + }, + ] + ``` + Model classes are importable when dicts feel too loose: `SkipIndexMinmax`, `SkipIndexSet`, `SkipIndexBloomFilter`, `SkipIndexTokenBF`, `SkipIndexNgramBF`. + + ## Projections @@ -290,12 +428,24 @@ An **index-only projection** stores no SELECT body. It reorders parts by a secon | `index` | `string` | Expression list to order by, e.g. `receiver, sender` | | `type` | `string` | Projection index type. ClickHouse currently accepts `basic` | -```ts -projections: [ - { name: 'p_recent', query: 'SELECT id ORDER BY received_at DESC LIMIT 10' }, - { name: 'by_receiver', index: 'receiver, sender', type: 'basic' }, -] -``` + + + ```ts + projections: [ + { name: 'p_recent', query: 'SELECT id ORDER BY received_at DESC LIMIT 10' }, + { name: 'by_receiver', index: 'receiver, sender', type: 'basic' }, + ] + ``` + + + ```python + projections=[ + {"name": "p_recent", "query": "SELECT id ORDER BY received_at DESC LIMIT 10"}, + {"name": "by_receiver", "index": "receiver, sender", "type": "basic"}, + ] + ``` + + The `index` expression is rendered the way ClickHouse normalizes it: a single expression is emitted bare (`INDEX receiver`), several are emitted as a tuple (`INDEX (receiver, sender)`), redundant parentheses are dropped, and a space follows every argument separator. Writing `'(receiver)'` and `'receiver'` therefore produce the same table, and neither reads as drift. @@ -305,10 +455,6 @@ A projection must be exactly one of the two kinds. Setting both `query` and `ind Creates a view definition. -```ts -view(input: Omit): ViewDefinition -``` - | Field | Type | Required | Description | |-------|------|----------|-------------| | `database` | `string` | yes | Database name | @@ -316,23 +462,34 @@ view(input: Omit): ViewDefinition | `as` | `string` | yes | SELECT query | | `comment` | `string` | no | View comment | -```ts -import { view } from '@chkit/core' - -const activeUsers = view({ - database: 'app', - name: 'active_users', - as: 'SELECT id, email FROM app.users WHERE active = 1', -}) -``` + + + ```ts + import { view } from '@chkit/core' + + const activeUsers = view({ + database: 'app', + name: 'active_users', + as: 'SELECT id, email FROM app.users WHERE active = 1', + }) + ``` + + + ```python + from chkit import view + + active_users = view( + database="app", + name="active_users", + as_="SELECT id, email FROM app.users WHERE active = 1", + ) + ``` + + ## `materializedView()` -Creates a materialized view definition. - -```ts -materializedView(input: Omit): MaterializedViewDefinition -``` +Creates a materialized view definition. In Python the factory is `materialized_view()`. | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -343,28 +500,59 @@ materializedView(input: Omit): MaterializedV | `as` | `string` | yes | SELECT query | | `comment` | `string` | no | View comment | -```ts -import { materializedView } from '@chkit/core' - -const eventCounts = materializedView({ - database: 'analytics', - name: 'event_counts_mv', - to: { database: 'analytics', name: 'event_counts' }, - as: 'SELECT org_id, count() AS total FROM analytics.events GROUP BY org_id', -}) -``` + + + ```ts + import { materializedView } from '@chkit/core' + + const eventCounts = materializedView({ + database: 'analytics', + name: 'event_counts_mv', + to: { database: 'analytics', name: 'event_counts' }, + as: 'SELECT org_id, count() AS total FROM analytics.events GROUP BY org_id', + }) + ``` + + + ```python + from chkit import materialized_view + + event_counts = materialized_view( + database="analytics", + name="event_counts_mv", + to={"database": "analytics", "name": "event_counts"}, + as_="SELECT org_id, count() AS total FROM analytics.events GROUP BY org_id", + ) + ``` + + For a refreshable (scheduled) materialized view, add the `refresh` field: -```ts -const dailyReport = materializedView({ - database: 'analytics', - name: 'daily_report_mv', - to: { database: 'analytics', name: 'daily_report' }, - refresh: { every: '1 DAY', offset: '2 HOUR' }, - as: 'SELECT toDate(ts) AS day, count() AS total FROM analytics.events GROUP BY day', -}) -``` + + + ```ts + const dailyReport = materializedView({ + database: 'analytics', + name: 'daily_report_mv', + to: { database: 'analytics', name: 'daily_report' }, + refresh: { every: '1 DAY', offset: '2 HOUR' }, + as: 'SELECT toDate(ts) AS day, count() AS total FROM analytics.events GROUP BY day', + }) + ``` + + + ```python + daily_report = materialized_view( + database="analytics", + name="daily_report_mv", + to={"database": "analytics", "name": "daily_report"}, + refresh={"every": "1 DAY", "offset": "2 HOUR"}, + as_="SELECT toDate(ts) AS day, count() AS total FROM analytics.events GROUP BY day", + ) + ``` + + See [Refreshable materialized views](/schema/refreshable-views/) for the full `refresh` field reference, including APPEND mode, `DEPENDS ON`, and the ClickHouse rules that chkit validates. @@ -372,28 +560,53 @@ See [Refreshable materialized views](/schema/refreshable-views/) for the full `r Creates a [ClickHouse dictionary](https://clickhouse.com/docs/sql-reference/dictionaries) definition — a key-value lookup structure backed by an external or in-database source, queried with `dictGet()`. -```ts -dictionary(input: Omit): DictionaryDefinition -``` - -```ts -import { dictionary } from '@chkit/core' - -const usersDict = dictionary({ - database: 'default', - name: 'users_dict', - attributes: [ - { name: 'id', type: 'UInt64' }, - { name: 'name', type: 'String' }, - { name: 'email', type: 'String', default: '' }, - ], - primaryKey: ['id'], - source: `MYSQL(host 'db' port 3306 user 'reader' password '${process.env.MYSQL_PASSWORD}' db 'app' table 'users')`, - layout: `HASHED()`, - lifetime: `300`, - comment: 'User lookup dictionary', -}) -``` + + + ```ts + import { dictionary } from '@chkit/core' + + const usersDict = dictionary({ + database: 'default', + name: 'users_dict', + attributes: [ + { name: 'id', type: 'UInt64' }, + { name: 'name', type: 'String' }, + { name: 'email', type: 'String', default: '' }, + ], + primaryKey: ['id'], + source: `MYSQL(host 'db' port 3306 user 'reader' password '${process.env.MYSQL_PASSWORD}' db 'app' table 'users')`, + layout: `HASHED()`, + lifetime: `300`, + comment: 'User lookup dictionary', + }) + ``` + + + ```python + import os + + from chkit import dictionary + + users_dict = dictionary( + database="default", + name="users_dict", + attributes=[ + {"name": "id", "type": "UInt64"}, + {"name": "name", "type": "String"}, + {"name": "email", "type": "String", "default": ""}, + ], + primary_key=["id"], + source=( + f"MYSQL(host 'db' port 3306 user 'reader' " + f"password '{os.environ['MYSQL_PASSWORD']}' db 'app' table 'users')" + ), + layout="HASHED()", + lifetime="300", + comment="User lookup dictionary", + ) + ``` + + ### Required fields @@ -437,11 +650,20 @@ Each entry in the `attributes` array is a `DictionaryAttribute`. ### Credentials in `source` -Inline credentials in `source` (e.g. a MySQL/PostgreSQL `password '...'`) should be interpolated from environment variables at schema-authoring time, the same way you'd handle any other secret in a TypeScript config file: - -```ts -source: `MYSQL(host 'db' password '${process.env.MYSQL_PASSWORD}' ...)`, -``` +Inline credentials in `source` (e.g. a MySQL/PostgreSQL `password '...'`) should be interpolated from environment variables at schema-authoring time, the same way you'd handle any other secret in a config file: + + + + ```ts + source: `MYSQL(host 'db' password '${process.env.MYSQL_PASSWORD}' ...)`, + ``` + + + ```python + source=f"MYSQL(host 'db' password '{os.environ['MYSQL_PASSWORD']}' ...)", + ``` + + ClickHouse redacts inline passwords back to `[HIDDEN]` on introspection (`SHOW CREATE DICTIONARY`, `system.dictionaries`). A real password change diffs and migrates like any other field change. The one exception is a `source` that still carries the literal `[HIDDEN]` placeholder written by `chkit pull` — chkit never knows the real value in that case, so it excludes `source` from the diff entirely rather than risk rendering `[HIDDEN]` into DDL — see [Pull: credential handling](/plugins/pull/#credential-handling-hidden-passwords). @@ -451,7 +673,7 @@ ClickHouse has no `ALTER DICTIONARY` — every structural change to a dictionary ## Type system reference -The [codegen plugin](/plugins/codegen/) maps ClickHouse types to TypeScript types using these rules: +The [codegen plugin](/plugins/codegen/) maps ClickHouse types to TypeScript types using these rules (the Python codegen plugin emits Pydantic models with the analogous Python types — `string` → `str`, `number` → `int`/`float`, `T[]` → `list[T]`, and so on): | Category | ClickHouse Types | TypeScript Type | |----------|-----------------|-----------------| @@ -477,14 +699,28 @@ chkit tracks renames to avoid destructive drop-and-recreate operations. Set `renamedFrom` on a table definition to rename a table: -```ts -const users = table({ - database: 'app', - name: 'accounts', // new name - renamedFrom: { name: 'users' }, // old name - // ... -}) -``` + + + ```ts + const users = table({ + database: 'app', + name: 'accounts', // new name + renamedFrom: { name: 'users' }, // old name + // ... + }) + ``` + + + ```python + users = table( + database="app", + name="accounts", # new name + renamed_from={"name": "users"}, # old name + # ... + ) + ``` + + The `database` field in `renamedFrom` is optional and defaults to the table's current database. @@ -492,24 +728,49 @@ The `database` field in `renamedFrom` is optional and defaults to the table's cu Set `renamedFrom` on a column definition to rename a column: -```ts -columns: [ - { name: 'user_email', type: 'String', renamedFrom: 'email' }, -] -``` + + + ```ts + columns: [ + { name: 'user_email', type: 'String', renamedFrom: 'email' }, + ] + ``` + + + ```python + columns=[ + {"name": "user_email", "type": "String", "renamedFrom": "email"}, + ] + ``` + + ### Dictionary rename Set `renamedFrom` on a dictionary definition to rename a dictionary. This emits a single `RENAME DICTIONARY IF EXISTS ... TO ...` statement instead of a `drop_dictionary` + `create_dictionary` pair: -```ts -const lookupDict = dictionary({ - database: 'app', - name: 'lookup_dict', // new name - renamedFrom: { name: 'users_dict' }, // old name - // ... -}) -``` + + + ```ts + const lookupDict = dictionary({ + database: 'app', + name: 'lookup_dict', // new name + renamedFrom: { name: 'users_dict' }, // old name + // ... + }) + ``` + + + ```python + lookup_dict = dictionary( + database="app", + name="lookup_dict", # new name + renamed_from={"name": "users_dict"}, # old name + # ... + ) + ``` + + The `database` field in `renamedFrom` is optional and defaults to the dictionary's current database. @@ -517,26 +778,50 @@ Table, column, and dictionary renames can all be overridden by CLI flags: `--ren ## Plugin configuration -The `plugins` field on a table definition provides per-table configuration for plugins. Each plugin that supports table-level config augments the `TablePlugins` interface via TypeScript declaration merging, so the available keys and their types depend on which plugin packages are imported. - -```ts -import { table } from '@chkit/core' - -const events = table({ - database: 'app', - name: 'events', - columns: [ - { name: 'event_time', type: 'DateTime' }, - { name: 'id', type: 'UInt64' }, - ], - engine: 'MergeTree', - orderBy: ['event_time', 'id'], - primaryKey: ['event_time', 'id'], - plugins: { - backfill: { timeColumn: 'event_time' }, - }, -}) -``` +The `plugins` field on a table definition provides per-table configuration for plugins. In TypeScript, each plugin that supports table-level config augments the `TablePlugins` interface via declaration merging; in Python it is a plain dict. + + + + ```ts + import { table } from '@chkit/core' + + const events = table({ + database: 'app', + name: 'events', + columns: [ + { name: 'event_time', type: 'DateTime' }, + { name: 'id', type: 'UInt64' }, + ], + engine: 'MergeTree', + orderBy: ['event_time', 'id'], + primaryKey: ['event_time', 'id'], + plugins: { + backfill: { timeColumn: 'event_time' }, + }, + }) + ``` + + + ```python + from chkit import table + + events = table( + database="app", + name="events", + columns=[ + {"name": "event_time", "type": "DateTime"}, + {"name": "id", "type": "UInt64"}, + ], + engine="MergeTree", + order_by=["event_time", "id"], + primary_key=["event_time", "id"], + plugins={ + "backfill": {"timeColumn": "event_time"}, + }, + ) + ``` + + Currently supported plugin keys: diff --git a/apps/docs/src/content/docs/schema/overview.md b/apps/docs/src/content/docs/schema/overview.md deleted file mode 100644 index 40c65dd4..00000000 --- a/apps/docs/src/content/docs/schema/overview.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Schema Overview -description: How chkit thinks about ClickHouse schema and where to learn each piece of the DSL. -sidebar: - order: 1 ---- - -In chkit, your ClickHouse schema lives in TypeScript files. You declare tables, views, materialized views, and dictionaries as plain values using functions from `@chkit/core`, group them with `schema()`, and let chkit handle the rest — diffing them against the database, generating migration SQL, and applying it safely. - -A typical schema file looks like this: - -```ts -import { schema, table } from '@chkit/core' - -const events = table({ - database: 'default', - name: 'events', - engine: 'MergeTree', - columns: [ - { name: 'id', type: 'UInt64' }, - { name: 'source', type: 'String' }, - { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, - ], - primaryKey: ['id'], - orderBy: ['id'], - partitionBy: 'toYYYYMM(ingested_at)', -}) - -export default schema(events) -``` - -chkit discovers schema files using the `schema` glob in your [configuration](/configuration/overview/), so you can split definitions across as many files as you like. - -## Concepts - -- **Definitions** — tables, views, materialized views, and dictionaries are values created with `table()`, `view()`, `materializedView()`, and `dictionary()`. They describe the *desired* state of your database. -- **Schema groups** — `schema(...)` collects definitions into a single export, but any exported definition is also discovered automatically. -- **Diff + plan** — when you run `chkit generate`, chkit compares your schema to the live database (or the last applied state) and emits migration SQL. -- **Engines** — `MergeTree`, `ReplacingMergeTree`, `AggregatingMergeTree`, and their `Shared` variants for [ObsessionDB](https://obsessiondb.com) are all first-class. - -## Reference - -- [DSL Reference](/schema/dsl-reference/) — every function, option, and column type. -- [Refreshable Views](/schema/refreshable-views/) — using ClickHouse refreshable materialized views from chkit. - -## Related - -- [Configuration Overview](/configuration/overview/) — where the `schema` glob is set. -- [CLI: `chkit generate`](/cli/generate/) — how schema changes become migration SQL. -- [CLI: `chkit pull`](/cli/pull/) — bootstrap schema files from an existing ClickHouse database. -- [ObsessionDB: Engine Rewriting](/obsessiondb/engine-rewriting/) — how `Shared*` engines are stripped when the target isn't ObsessionDB. diff --git a/apps/docs/src/content/docs/schema/overview.mdx b/apps/docs/src/content/docs/schema/overview.mdx new file mode 100644 index 00000000..3e38562a --- /dev/null +++ b/apps/docs/src/content/docs/schema/overview.mdx @@ -0,0 +1,79 @@ +--- +title: Schema Overview +description: How chkit thinks about ClickHouse schema and where to learn each piece of the DSL. +sidebar: + order: 1 +--- + +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +In chkit, your ClickHouse schema lives in TypeScript or Python files. You declare tables, views, materialized views, and dictionaries as plain values using functions from `@chkit/core` (or `chkit` in Python), group them with `schema()`, and let chkit handle the rest — diffing them against the database, generating migration SQL, and applying it safely. + +A typical schema file looks like this: + + + + ```ts + import { schema, table } from '@chkit/core' + + const events = table({ + database: 'default', + name: 'events', + engine: 'MergeTree', + columns: [ + { name: 'id', type: 'UInt64' }, + { name: 'source', type: 'String' }, + { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, + ], + primaryKey: ['id'], + orderBy: ['id'], + partitionBy: 'toYYYYMM(ingested_at)', + }) + + export default schema(events) + ``` + + + ```python + from chkit import schema, table + + events = table( + database="default", + name="events", + engine="MergeTree", + columns=[ + {"name": "id", "type": "UInt64"}, + {"name": "source", "type": "String"}, + {"name": "ingested_at", "type": "DateTime64(3)", "default": "fn:now64(3)"}, + ], + primary_key=["id"], + order_by=["id"], + partition_by="toYYYYMM(ingested_at)", + ) + + definitions = schema(events) + ``` + + + +chkit discovers schema files using the `schema` glob in your [configuration](/configuration/overview/), so you can split definitions across as many files as you like. + +## Concepts + +- **Definitions** — tables, views, materialized views, and dictionaries are values created with `table()`, `view()`, `materializedView()`, and `dictionary()`. They describe the *desired* state of your database. +- **Schema groups** — `schema(...)` collects definitions into a single export, but any exported definition is also discovered automatically. +- **Diff + plan** — when you run `chkit generate`, chkit compares your schema to the live database (or the last applied state) and emits migration SQL. +- **Engines** — `MergeTree`, `ReplacingMergeTree`, `AggregatingMergeTree`, and their `Shared` variants for [ObsessionDB](https://obsessiondb.com) are all first-class. + +## Reference + +- [DSL Reference](/schema/dsl-reference/) — every function, option, and column type, in both languages. +- [Refreshable Views](/schema/refreshable-views/) — using ClickHouse refreshable materialized views from chkit. + +## Related + +- [Configuration Overview](/configuration/overview/) — where the `schema` glob is set. +- [CLI: `chkit generate`](/cli/generate/) — how schema changes become migration SQL. +- [CLI: `chkit pull`](/cli/pull/) — bootstrap schema files from an existing ClickHouse database. +- [Python Overview](/python/overview/) — installing and using the Python implementation, `chkit-py`. +- [ObsessionDB: Engine Rewriting](/obsessiondb/engine-rewriting/) — how `Shared*` engines are stripped when the target isn't ObsessionDB. diff --git a/apps/docs/src/content/docs/schema/refreshable-views.md b/apps/docs/src/content/docs/schema/refreshable-views.mdx similarity index 65% rename from apps/docs/src/content/docs/schema/refreshable-views.md rename to apps/docs/src/content/docs/schema/refreshable-views.mdx index e6134520..aeaa448e 100644 --- a/apps/docs/src/content/docs/schema/refreshable-views.md +++ b/apps/docs/src/content/docs/schema/refreshable-views.mdx @@ -5,24 +5,46 @@ sidebar: order: 2 --- +import { Tabs, TabItem } from '@astrojs/starlight/components'; + ClickHouse [refreshable materialized views](https://clickhouse.com/docs/materialized-view/refreshable-materialized-view) (RMVs) periodically re-execute a SELECT on a schedule, rather than firing per-INSERT like regular incremental MVs. They've been production-ready since ClickHouse 24.10. chkit models an RMV as a regular `materializedView()` with an extra `refresh` field: -```ts -import { materializedView } from '@chkit/core' - -const dailyReport = materializedView({ - database: 'analytics', - name: 'daily_report_mv', - to: { database: 'analytics', name: 'daily_report' }, - refresh: { - every: '1 DAY', - offset: '2 HOUR', - }, - as: 'SELECT toDate(ts) AS day, count() AS total FROM analytics.events GROUP BY day', -}) -``` + + + ```ts + import { materializedView } from '@chkit/core' + + const dailyReport = materializedView({ + database: 'analytics', + name: 'daily_report_mv', + to: { database: 'analytics', name: 'daily_report' }, + refresh: { + every: '1 DAY', + offset: '2 HOUR', + }, + as: 'SELECT toDate(ts) AS day, count() AS total FROM analytics.events GROUP BY day', + }) + ``` + + + ```python + from chkit import materialized_view + + daily_report = materialized_view( + database="analytics", + name="daily_report_mv", + to={"database": "analytics", "name": "daily_report"}, + refresh={ + "every": "1 DAY", + "offset": "2 HOUR", + }, + as_="SELECT toDate(ts) AS day, count() AS total FROM analytics.events GROUP BY day", + ) + ``` + + Without a `refresh` field, the definition is a plain incremental materialized view. @@ -50,43 +72,87 @@ Intervals use the ClickHouse form: ` ` where `` is `SECOND` APPEND is useful for periodic snapshots where you want history, e.g.: -```ts -const hourlySnapshot = materializedView({ - database: 'analytics', - name: 'hourly_snapshot_mv', - to: { database: 'analytics', name: 'hourly_snapshots' }, - refresh: { - every: '1 HOUR', - append: true, - }, - as: 'SELECT now() AS snapshot_ts, org_id, count() AS cnt FROM analytics.events GROUP BY org_id', -}) -``` + + + ```ts + const hourlySnapshot = materializedView({ + database: 'analytics', + name: 'hourly_snapshot_mv', + to: { database: 'analytics', name: 'hourly_snapshots' }, + refresh: { + every: '1 HOUR', + append: true, + }, + as: 'SELECT now() AS snapshot_ts, org_id, count() AS cnt FROM analytics.events GROUP BY org_id', + }) + ``` + + + ```python + hourly_snapshot = materialized_view( + database="analytics", + name="hourly_snapshot_mv", + to={"database": "analytics", "name": "hourly_snapshots"}, + refresh={ + "every": "1 HOUR", + "append": True, + }, + as_="SELECT now() AS snapshot_ts, org_id, count() AS cnt FROM analytics.events GROUP BY org_id", + ) + ``` + + ## Chaining refreshes with `DEPENDS ON` Use `dependsOn` to ensure an RMV runs only after its upstream MVs have refreshed: -```ts -const hourlyBase = materializedView({ - database: 'analytics', - name: 'hourly_base_mv', - to: { database: 'analytics', name: 'hourly_base' }, - refresh: { every: '1 HOUR' }, - as: 'SELECT ...', -}) - -const hourlyAggregate = materializedView({ - database: 'analytics', - name: 'hourly_aggregate_mv', - to: { database: 'analytics', name: 'hourly_aggregate' }, - refresh: { - every: '1 HOUR', - dependsOn: [{ database: 'analytics', name: 'hourly_base_mv' }], - }, - as: 'SELECT ... FROM analytics.hourly_base GROUP BY ...', -}) -``` + + + ```ts + const hourlyBase = materializedView({ + database: 'analytics', + name: 'hourly_base_mv', + to: { database: 'analytics', name: 'hourly_base' }, + refresh: { every: '1 HOUR' }, + as: 'SELECT ...', + }) + + const hourlyAggregate = materializedView({ + database: 'analytics', + name: 'hourly_aggregate_mv', + to: { database: 'analytics', name: 'hourly_aggregate' }, + refresh: { + every: '1 HOUR', + dependsOn: [{ database: 'analytics', name: 'hourly_base_mv' }], + }, + as: 'SELECT ... FROM analytics.hourly_base GROUP BY ...', + }) + ``` + + + ```python + hourly_base = materialized_view( + database="analytics", + name="hourly_base_mv", + to={"database": "analytics", "name": "hourly_base"}, + refresh={"every": "1 HOUR"}, + as_="SELECT ...", + ) + + hourly_aggregate = materialized_view( + database="analytics", + name="hourly_aggregate_mv", + to={"database": "analytics", "name": "hourly_aggregate"}, + refresh={ + "every": "1 HOUR", + "dependsOn": [{"database": "analytics", "name": "hourly_base_mv"}], + }, + as_="SELECT ... FROM analytics.hourly_base GROUP BY ...", + ) + ``` + + `DEPENDS ON` is only supported with `REFRESH EVERY` — ClickHouse rejects it when paired with `REFRESH AFTER`. chkit enforces this at `generate` / `check` time (`refresh_depends_on_requires_every`). diff --git a/apps/docs/src/content/docs/tutorials/first-schema.md b/apps/docs/src/content/docs/tutorials/first-schema.md deleted file mode 100644 index faed3339..00000000 --- a/apps/docs/src/content/docs/tutorials/first-schema.md +++ /dev/null @@ -1,188 +0,0 @@ ---- -title: "Tutorial: your first schema" -description: Build a chkit project from an empty folder — migrate the starter table to a live database, insert and query rows, then evolve the schema. -sidebar: - order: 1 ---- - -A hands-on walkthrough from an empty folder to a live, version-controlled table. You'll scaffold a chkit project, deploy the starter `events` table it generates, put data in it, query it back, then add a column and ship the change — the full chkit loop, start to finish. - -Every step is a real command. Run them in order and you'll end with a working project you can keep building on. - -## What you'll need - -- Node.js 20+ or Bun 1.3.5+ -- An email inbox you can reach - -No ClickHouse to install: this tutorial claims a free ObsessionDB dev instance straight from the CLI. The commands use `bun`; `npm`, `pnpm`, and `yarn` work the same way. - -## 1. Create the project - -Start in a new, empty folder and run `chkit init`: - -```sh -mkdir chkit-tutorial -cd chkit-tutorial -bunx chkit@latest init -``` - -In an empty directory, `init` does the full bootstrap: it writes `clickhouse.config.ts` and a starter schema at `src/db/schema/example.ts`, creates a `package.json`, and installs `chkit`, `@chkit/core`, and `@chkit/plugin-obsessiondb` so the project is runnable. - -It then shows the connect prompt: - -``` -Claim a free ObsessionDB dev instance email code, ready in seconds -I already have an ObsessionDB account log in and pick a service -I already have a ClickHouse instance connect with env vars -Configure later -``` - -Choose **Claim a free ObsessionDB dev instance**, enter your email, and paste the 6-digit code from your inbox. chkit creates a personal organization, provisions a free instance, selects it (written to `.chkit/obsessiondb.json`), and registers the ObsessionDB plugin in your config. - -:::note -Already run your own ClickHouse? Pick **I already have a ClickHouse instance** instead and set `CLICKHOUSE_URL`. Every command below works the same against a direct ClickHouse — see [Getting Started with ObsessionDB](/obsessiondb/getting-started/) for the alternatives. -::: - -Confirm the connection works: - -```sh -bunx chkit query "SELECT 1" -``` - -A single row back means you're connected. - -## 2. Look at the starter schema - -`init` scaffolded a table for you at `src/db/schema/example.ts` — an `events` table that's a good shape for ingesting application or analytics events: - -```ts -// src/db/schema/example.ts -import { schema, table } from '@chkit/core' - -const events = table({ - database: 'default', - name: 'events', - engine: 'MergeTree', - columns: [ - { name: 'id', type: 'UInt64' }, - { name: 'source', type: 'String' }, - { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, - ], - primaryKey: ['id'], - orderBy: ['id'], - partitionBy: 'toYYYYMM(ingested_at)', -}) - -export default schema(events) -``` - -A `table()` definition maps directly to a ClickHouse `CREATE TABLE`: a `MergeTree` engine, three columns, ordered by `id`, and partitioned by month. `ingested_at` carries `default: 'fn:now64(3)'`, so the database fills it in automatically. The types here (`UInt64`, `String`, `DateTime64(3)`) are ClickHouse-native; see the [Schema DSL reference](/schema/dsl-reference/) for the full type system and table options. - -Use it as-is for now — you'll change it later. - -## 3. Generate the migration - -`chkit generate` diffs your schema against the previous snapshot and writes migration SQL. There's no snapshot yet, so this produces a `CREATE TABLE`: - -```sh -bunx chkit generate --name create_events -``` - -Open the file it wrote under `chkit/migrations/` — chkit shows you the exact SQL before anything is applied: - -```sql --- operation: create_table key=table:default.events risk=safe -CREATE TABLE default.events -( - id UInt64, - source String, - ingested_at DateTime64(3) DEFAULT now64(3) -) -ENGINE = MergeTree -PARTITION BY toYYYYMM(ingested_at) -ORDER BY id; -``` - -## 4. Apply it - -```sh -bunx chkit migrate --apply -``` - -This runs the pending migration against the instance you claimed and records it in the migration journal. - -## 5. Verify the table exists - -Check migration state, confirm the live schema matches your code, and look at the table directly: - -```sh -bunx chkit status -bunx chkit check -bunx chkit query "DESCRIBE events" -``` - -`status` lists the applied migration, `check` confirms the database matches your TypeScript definitions, and `DESCRIBE` shows the live columns. - -:::caution -ClickHouse and ObsessionDB DDL is eventually consistent — it isn't instant. If `status` or `check` runs immediately after `migrate --apply`, give it a moment and re-run so it doesn't race the cluster. -::: - -## 6. Insert and query rows - -The table is empty. Put a couple of rows in with `chkit query` — `ingested_at` is left out, so the database fills it from its default: - -```sh -bunx chkit query "INSERT INTO events (id, source) VALUES (1, 'web'), (2, 'mobile')" -``` - -Then read them back: - -```sh -bunx chkit query "SELECT count() FROM events" -bunx chkit query "SELECT id, source, ingested_at FROM events ORDER BY id" -``` - -You now have a schema in code and matching data in a live database. - -## 7. Evolve the schema - -Schemas change. Add a `level` column to the `events` table in `src/db/schema/example.ts`: - -```ts - columns: [ - { name: 'id', type: 'UInt64' }, - { name: 'source', type: 'String' }, - { name: 'level', type: 'String' }, - { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, - ], -``` - -Generate a migration for the change and review it — this time it's an `ALTER TABLE`, not a recreate: - -```sh -bunx chkit generate --name add_level_column -``` - -```sql --- operation: add_column key=table:default.events risk=safe -ALTER TABLE default.events ADD COLUMN level String AFTER source; -``` - -Apply it and confirm the column landed: - -```sh -bunx chkit migrate --apply -bunx chkit check -bunx chkit query "DESCRIBE events" -``` - -`check` passes again, and `DESCRIBE` now lists `level`. That's the whole chkit loop: **edit the schema → `generate` → review the SQL → `migrate` → verify** — repeat it for every change from here on. - -## Where to next - -- [The migration workflow](/guides/migration-workflow/) — how the pieces fit, what to commit, and how a team without production access ships changes -- [The CLI reference](/cli/overview/) — every command and flag used above -- [Schema DSL reference](/schema/dsl-reference/) — columns, engines, views, and materialized views -- [Configuration](/configuration/overview/) — what `clickhouse.config.ts` controls -- [Getting Started with ObsessionDB](/obsessiondb/getting-started/) — other ways to connect, and non-interactive setup -- [CI/CD integration](/guides/ci-cd/) — run `generate`, `migrate`, and `check` in a pipeline diff --git a/apps/docs/src/content/docs/tutorials/first-schema.mdx b/apps/docs/src/content/docs/tutorials/first-schema.mdx new file mode 100644 index 00000000..39d11e87 --- /dev/null +++ b/apps/docs/src/content/docs/tutorials/first-schema.mdx @@ -0,0 +1,321 @@ +--- +title: "Tutorial: your first schema" +description: Build a chkit project from an empty folder — migrate the starter table to a live database, insert and query rows, then evolve the schema. +sidebar: + order: 1 +--- + +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +A hands-on walkthrough from an empty folder to a live, version-controlled table. You'll scaffold a chkit project, deploy the starter `events` table it generates, put data in it, query it back, then add a column and ship the change — the full chkit loop, start to finish. + +Every step is a real command. Run them in order and you'll end with a working project you can keep building on. Pick your language once — the tabs stay in sync. + +## What you'll need + +- Node.js 20+ or Bun 1.3.5+ (TypeScript) — or Python 3.11+ (Python) +- An email inbox you can reach + +No ClickHouse to install: this tutorial claims a free ObsessionDB dev instance straight from the CLI. The TypeScript commands use `bun`; `npm`, `pnpm`, and `yarn` work the same way. + +## 1. Create the project + +Start in a new, empty folder and run `chkit init`: + + + + ```sh + mkdir chkit-tutorial + cd chkit-tutorial + bunx chkit@latest init + ``` + + In an empty directory, `init` does the full bootstrap: it writes `clickhouse.config.ts` and a starter schema at `src/db/schema/example.ts`, creates a `package.json`, and installs `chkit`, `@chkit/core`, and `@chkit/plugin-obsessiondb` so the project is runnable. + + + ```sh + mkdir chkit-tutorial + cd chkit-tutorial + pip install chkit-py + chkit init + ``` + + `init` writes `clickhouse.config.py` and a starter schema at `src/db/schema/example.py`. The `chkit` CLI and the ObsessionDB plugin ship inside the `chkit-py` package, so there is nothing else to install. + + + +It then shows the connect prompt: + +``` +Claim a free ObsessionDB dev instance email code, ready in seconds +I already have an ObsessionDB account log in and pick a service +I already have a ClickHouse instance connect with env vars +Configure later +``` + +Choose **Claim a free ObsessionDB dev instance**, enter your email, and paste the 6-digit code from your inbox. chkit creates a personal organization, provisions a free instance, selects it (written to `.chkit/obsessiondb.json`), and registers the ObsessionDB plugin in your config. + +:::note +Already run your own ClickHouse? Pick **I already have a ClickHouse instance** instead and set `CLICKHOUSE_URL`. Every command below works the same against a direct ClickHouse — see [Getting Started with ObsessionDB](/obsessiondb/getting-started/) for the alternatives. +::: + +Confirm the connection works: + + + + ```sh + bunx chkit query "SELECT 1" + ``` + + + ```sh + chkit query "SELECT 1" + ``` + + + +A single row back means you're connected. + +## 2. Look at the starter schema + +`init` scaffolded a table for you — an `events` table that's a good shape for ingesting application or analytics events: + + + + ```ts + // src/db/schema/example.ts + import { schema, table } from '@chkit/core' + + const events = table({ + database: 'default', + name: 'events', + engine: 'MergeTree', + columns: [ + { name: 'id', type: 'UInt64' }, + { name: 'source', type: 'String' }, + { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, + ], + primaryKey: ['id'], + orderBy: ['id'], + partitionBy: 'toYYYYMM(ingested_at)', + }) + + export default schema(events) + ``` + + + ```python + # src/db/schema/example.py + from chkit import schema, table + + events = table( + database="default", + name="events", + engine="MergeTree", + columns=[ + {"name": "id", "type": "UInt64"}, + {"name": "source", "type": "String"}, + {"name": "ingested_at", "type": "DateTime64(3)", "default": "fn:now64(3)"}, + ], + primaryKey=["id"], + orderBy=["id"], + partitionBy="toYYYYMM(ingested_at)", + ) + + definitions = schema(events) + ``` + + + +A `table()` definition maps directly to a ClickHouse `CREATE TABLE`: a `MergeTree` engine, three columns, ordered by `id`, and partitioned by month. `ingested_at` carries `default: 'fn:now64(3)'`, so the database fills it in automatically. The types here (`UInt64`, `String`, `DateTime64(3)`) are ClickHouse-native; see the [Schema DSL reference](/schema/dsl-reference/) for the full type system and table options. + +Use it as-is for now — you'll change it later. + +## 3. Generate the migration + +`chkit generate` diffs your schema against the previous snapshot and writes migration SQL. There's no snapshot yet, so this produces a `CREATE TABLE`: + + + + ```sh + bunx chkit generate --name create_events + ``` + + + ```sh + chkit generate --name create_events + ``` + + + +Open the file it wrote under `chkit/migrations/` — chkit shows you the exact SQL before anything is applied: + +```sql +-- operation: create_table key=table:default.events risk=safe +CREATE TABLE default.events +( + id UInt64, + source String, + ingested_at DateTime64(3) DEFAULT now64(3) +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(ingested_at) +ORDER BY id; +``` + +## 4. Apply it + + + + ```sh + bunx chkit migrate --apply + ``` + + + ```sh + chkit migrate --apply + ``` + + + +This runs the pending migration against the instance you claimed and records it in the migration journal. + +## 5. Verify the table exists + +Check migration state, confirm the live schema matches your code, and look at the table directly: + + + + ```sh + bunx chkit status + bunx chkit check + bunx chkit query "DESCRIBE events" + ``` + + + ```sh + chkit status + chkit check + chkit query "DESCRIBE events" + ``` + + + +`status` lists the applied migration, `check` confirms the database matches your schema definitions, and `DESCRIBE` shows the live columns. + +:::caution +ClickHouse and ObsessionDB DDL is eventually consistent — it isn't instant. If `status` or `check` runs immediately after `migrate --apply`, give it a moment and re-run so it doesn't race the cluster. +::: + +## 6. Insert and query rows + +The table is empty. Put a couple of rows in with `chkit query` — `ingested_at` is left out, so the database fills it from its default: + + + + ```sh + bunx chkit query "INSERT INTO events (id, source) VALUES (1, 'web'), (2, 'mobile')" + ``` + + + ```sh + chkit query "INSERT INTO events (id, source) VALUES (1, 'web'), (2, 'mobile')" + ``` + + + +Then read them back: + + + + ```sh + bunx chkit query "SELECT count() FROM events" + bunx chkit query "SELECT id, source, ingested_at FROM events ORDER BY id" + ``` + + + ```sh + chkit query "SELECT count() FROM events" + chkit query "SELECT id, source, ingested_at FROM events ORDER BY id" + ``` + + + +You now have a schema in code and matching data in a live database. + +## 7. Evolve the schema + +Schemas change. Add a `level` column to the `events` table: + + + + ```ts + columns: [ + { name: 'id', type: 'UInt64' }, + { name: 'source', type: 'String' }, + { name: 'level', type: 'String' }, + { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, + ], + ``` + + + ```python + columns=[ + {"name": "id", "type": "UInt64"}, + {"name": "source", "type": "String"}, + {"name": "level", "type": "String"}, + {"name": "ingested_at", "type": "DateTime64(3)", "default": "fn:now64(3)"}, + ], + ``` + + + +Generate a migration for the change and review it — this time it's an `ALTER TABLE`, not a recreate: + + + + ```sh + bunx chkit generate --name add_level_column + ``` + + + ```sh + chkit generate --name add_level_column + ``` + + + +```sql +-- operation: add_column key=table:default.events risk=safe +ALTER TABLE default.events ADD COLUMN level String AFTER source; +``` + +Apply it and confirm the column landed: + + + + ```sh + bunx chkit migrate --apply + bunx chkit check + bunx chkit query "DESCRIBE events" + ``` + + + ```sh + chkit migrate --apply + chkit check + chkit query "DESCRIBE events" + ``` + + + +`check` passes again, and `DESCRIBE` now lists `level`. That's the whole chkit loop: **edit the schema → `generate` → review the SQL → `migrate` → verify** — repeat it for every change from here on. + +## Where to next + +- [The migration workflow](/guides/migration-workflow/) — how the pieces fit, what to commit, and how a team without production access ships changes +- [The CLI reference](/cli/overview/) — every command and flag used above +- [Schema DSL reference](/schema/dsl-reference/) — columns, engines, views, materialized views, and dictionaries +- [Configuration](/configuration/overview/) — what `clickhouse.config.ts` / `clickhouse.config.py` controls +- [Getting Started with ObsessionDB](/obsessiondb/getting-started/) — other ways to connect, and non-interactive setup +- [CI/CD integration](/guides/ci-cd/) — run `generate`, `migrate`, and `check` in a pipeline diff --git a/apps/docs/src/integrations/raw-markdown.ts b/apps/docs/src/integrations/raw-markdown.ts index 86e60527..0394893c 100644 --- a/apps/docs/src/integrations/raw-markdown.ts +++ b/apps/docs/src/integrations/raw-markdown.ts @@ -1,9 +1,10 @@ import { readFileSync, readdirSync, statSync, writeFileSync, mkdirSync } from 'node:fs'; import { join, dirname, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; import type { AstroIntegration } from 'astro'; const BASE_URL = 'https://chkit.obsessiondb.com'; -const SITE_TAGLINE = 'ClickHouse schema management and migration toolkit for TypeScript.'; +const SITE_TAGLINE = 'ClickHouse schema management and migration toolkit for TypeScript and Python.'; interface DocEntry { slug: string; @@ -27,8 +28,12 @@ function extractFrontmatter(content: string): { title: string; description: stri } // Strip extension and collapse "index" / "/index" into the directory slug. +// Windows `relative()` emits backslashes — normalize so slugs are URL-shaped. function toSlug(rel: string): string { - return rel.replace(/\.mdx?$/, '').replace(/(^|\/)index$/, ''); + return rel + .replaceAll('\\', '/') + .replace(/\.mdx?$/, '') + .replace(/(^|\/)index$/, ''); } function collectMarkdownFiles(srcDir: string, destDir: string): DocEntry[] { @@ -97,7 +102,7 @@ function generateLlmsTxt(entries: DocEntry[]): string { '', `> ${SITE_TAGLINE}`, '', - 'chkit defines ClickHouse schemas in TypeScript, diffs them into migration SQL, applies migrations, and verifies the live database stays in sync. Each link below points to the raw Markdown of that page.', + 'chkit defines ClickHouse schemas in TypeScript or Python, diffs them into migration SQL, applies migrations, and verifies the live database stays in sync. Each link below points to the raw Markdown of that page.', '', '## Docs', '', @@ -116,8 +121,8 @@ export default function rawMarkdown(): AstroIntegration { name: 'raw-markdown', hooks: { 'astro:build:done': ({ dir, logger }) => { - const srcDir = new URL('../src/content/docs/', dir).pathname; - const distDir = new URL(dir).pathname; + const srcDir = fileURLToPath(new URL('../src/content/docs/', dir)); + const distDir = fileURLToPath(dir); const rawDir = join(distDir, '_raw'); const entries = collectMarkdownFiles(srcDir, rawDir); diff --git a/apps/docs/src/styles/custom.css b/apps/docs/src/styles/custom.css index 12aa66d0..d6ea7d50 100644 --- a/apps/docs/src/styles/custom.css +++ b/apps/docs/src/styles/custom.css @@ -953,37 +953,12 @@ starlight-toc a[aria-current='true'] { background: var(--surface-3); } -.chk-badge--soon:hover { - color: var(--txt-soft); -} - .chk-badge-logo { width: 1.15rem; height: 1.15rem; flex: 0 0 auto; } -/* "Coming soon" variant — dimmed badge with a small tag. */ -.chk-badge--soon { - color: var(--txt-muted); - cursor: default; -} - -.chk-badge--soon .chk-badge-logo { - opacity: 0.55; -} - -.chk-badge-soon { - font-family: var(--sl-font-mono); - font-size: 0.5625rem; - letter-spacing: 0.08em; - text-transform: uppercase; - color: var(--accent); - border: 1px solid var(--accent-dim); - padding: 0.1em 0.4em; - margin-inline-start: 0.1rem; -} - /* Wrapper holding the section label + the card grid. */ .chk-actions { display: flex; @@ -1389,11 +1364,6 @@ kbd { color: var(--accent); } -.chk-footer-soon { - font-size: 0.875rem; - color: var(--txt-muted); -} - /* ════════════════════════════════════════════════════ Blog listing — card grid (ObsessionDB-style) Restyles starlight-blog's .posts / .preview markup from a vertical diff --git a/chkit_python/.gitignore b/chkit_python/.gitignore new file mode 100644 index 00000000..b2a72b6f --- /dev/null +++ b/chkit_python/.gitignore @@ -0,0 +1,17 @@ +__pycache__/ +*.py[cod] +*.egg-info/ +.eggs/ +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +.pyright/ +dist/ +build/ +.venv/ +venv/ +.env +.coverage +htmlcov/ +.venv-dev/ +**/__pycache__/** diff --git a/chkit_python/CHANGELOG.md b/chkit_python/CHANGELOG.md new file mode 100644 index 00000000..81cbe519 --- /dev/null +++ b/chkit_python/CHANGELOG.md @@ -0,0 +1,194 @@ +# Changelog + +## 0.2.0 — 2026-08-10 + +**Full parity with the TypeScript chkit.** Every remaining gap is closed; +the parity decision log lives in `DRIFT.md`. + +### Added +- **Dictionary primitive** — `dictionary()` DSL, validation (8 codes), + `CREATE / CREATE OR REPLACE / RENAME / DROP DICTIONARY` planning with + `[HIDDEN]`-password handling, `--rename-dictionary`, create-dictionary + parser, pull introspection + rendering, codegen Pydantic models, drift + and safety-marker coverage. +- **Phase-2 backfill engine** — chunking planner (partition slices, byte + budgets, all split strategies), chunk-execution SQL builder with MV + replay (every feeding MV via `UNION ALL`, chunks sized from the MV + source), async submit/poll execution loop with atomic checkpointing, + real `plan` / `run` / `resume` / `doctor` commands, managed + `backfill submit` to ObsessionDB jobs with console deep-links, and the + `on_check` findings (`backfill_required_pending`, ...). +- **Index-only projections** (`{"index": ..., "type": ...}`) and + **function expressions in `primaryKey`/`orderBy`**. +- **CLI**: top-level `chkit codegen` and `chkit obsessiondb ` + shortcuts; `chkit plugin ` now forwards the command's own + `--flags`. +- **Config**: function-style configs — `define_config(lambda env: ...)` + with `ChxConfigEnv(command, mode)`; `check.failOnExtraObjects`; + per-table `plugins` field on `table()`. + +### Fixed +- Wheel now packages `chkit_plugin_codegen` and `chkit_plugin_backfill` + (previously missing from `pip install chkit-py`). +- `ClickHouseClient.submit()` crashed on every live call (unsupported + `query_id=` kwarg) — affected `migrate --apply` async statements. +- Snapshots serialize with `exclude_none`, matching TS `JSON.stringify` + key omission so TS tooling reads Python-written snapshots correctly. +- JS-fidelity fixes across ports: `Number()`/`String()` semantics for + chunk boundaries, `Date.parse` sub-millisecond truncation, WHATWG + `URL.origin` environment fingerprints (TS-written plans now run under + Python), JS `\s` whitespace class in key-clause comparison, `??` vs + truthiness in drift primary-key fallback. +- Plugin command `--json` output prints real JSON (was Python dict repr). +- Table-clause parsing no longer swallows clauses when a projection's + SELECT contains `ORDER BY`, and a primary key derived from `ORDER BY` + no longer reads as drift. + +## 0.1.4 — 2026-06-05 + +Documentation refresh — no code changes. + +### Added +- `PARITY.md` — TypeScript ↔ Python parity matrix listing every TS module / + command / flag, what's 1:1 today, what's intentionally deferred, and the + rationale behind each deferral. Lives at the repo root so contributors can + pick a deferred item and port it without spelunking through the TS source. + +### Changed +- `README.md` rewritten: + - Drops the "port-in-progress" note (the first base is done). + - Adds an explicit TypeScript-parity section summarising what's covered. + - Adds a Quickstart that walks through `init` → `generate` → `migrate --apply` + → `status` / `check` / `drift`. + - Points contributors at `PARITY.md` for the full divergence matrix. + +## 0.1.3 — 2026-06-05 + +Major **TS parity** release. The CLI now matches the TypeScript reference 1:1 +across `generate`, `migrate`, `status`, `check`, and `drift`. + +### Added +- **Migration SQL artifact format** matches `packages/codegen/src/index.ts`: + - Header comments: `chkit-migration-format: v1`, `generated-at`, + `cli-version`, `definition-count`, `operation-count`, + `rename-suggestion-count`, `risk-summary`. + - Per-operation comments: `-- operation: key= risk=`. + - Rename hint comments: `-- rename-suggestion: ...`. + - Filename: `_.sql` with `_001`, `_002`, ... suffix on + collision (matches TS `safeName` + `collisionIndex` behaviour). +- **`chkit generate --name `** replaces the old `--label`. Adds + `--migration-id ` (override timestamp prefix) and `--dryrun` (print plan + without writing artifacts), matching the TS flag set verbatim. +- **`chkit migrate`** defaults to **plan/preview** like TS. Use `--apply` (or + alias `--execute`) to actually run statements. Added `--allow-destructive` + for migrations whose plan contains `risk=danger` operations (exit code 3 + when blocked, mirroring TS). +- **ClickHouse-backed journal**: applied migrations are recorded in a + `_chkit_migrations` table in the target database, schema identical to + `packages/cli/src/runtime/journal-store.ts` + (`ReplacingMergeTree(applied_at) ORDER BY (name)`). Both TS and Python now + share the same journal table. Override the name via + `CHKIT_JOURNAL_TABLE`. +- **Checksum mismatch detection** in `status`, `check`, and `migrate`: + re-hashes each `.sql` file and compares against the checksum recorded in + the journal. Mismatches block applies. +- **`chkit check --strict`** flag matches TS: enables every policy + (`failOnPending`, `failOnChecksumMismatch`, `failOnDrift`) regardless of + config. Exit code 1 when any policy fails. +- **`safe_name`, `safe_migration_id`, `checksum_sql`** public helpers in + `chkit.cli.migration_store`. + +### Changed +- **Snapshot file** ends with a trailing newline (`json + "\n"`), matching + the TS write. +- **`status` output text** matches TS verbatim: + ``` + Migrations directory: + Total migrations: + Applied: + Pending: + ``` + Database-missing warning ("Database X does not exist on the target server") + reproduced verbatim. +- **`pending_migrations`** and the journal now key by filename (with `.sql`), + matching the TS `MigrationJournalEntry.name`. Pre-0.1.3 keys (stem-only) + in `meta/applied.json` are no longer compatible; if you had an offline + `applied.json`, either delete it or re-apply via `chkit migrate --apply` + to populate the new ClickHouse journal. +- **`chkit init`** template prompts users to run `chkit generate --name init` + followed by `chkit migrate --apply` (was `--label init` + `migrate`). + +### Migration note for 0.1.0 / 0.1.1 / 0.1.2 users +The journal moved from `meta/applied.json` to the ClickHouse `_chkit_migrations` +table. To migrate an existing project: + +1. Run `chkit migrate --apply`. Already-applied migrations will fail with + "table already exists"; the journal will not record them. +2. Recommended: clear the old `applied.json` once the ClickHouse journal is + the source of truth. + +If you need to manually seed the journal from a `applied.json`, insert rows +into `_chkit_migrations` matching the schema in +`packages/cli/src/runtime/journal-store.ts`. + +## 0.1.2 — 2026-06-05 + +### Fixed +- `chkit init` is now a 1:1 port of the TypeScript `init` command: + - Writes `clickhouse.config.py` (matching the TS `clickhouse.config.ts` + convention) instead of `chkit.config.py`. + - Scaffolds the schema at `src/db/schema/example.py` (was `schema/events.py`). + - The generated config exports `migrationsDir`, `metaDir`, and a `plugins` + list explicitly, mirroring the TS template. + - ClickHouse credentials default through `os.environ.get(...)` instead of + being hardcoded. + - Example schema columns match TS (`id` / `source` / `ingested_at` with + `partitionBy: toYYYYMM(ingested_at)`). + - Prints the same "Next steps" message + docs link as TS. + - Removed the `--out` flag; init scaffolds into the current working + directory like the TS version. +- `ChxUserConfig` now accepts a `plugins` list (was silently rejected as an + extra field, which broke `chkit generate` for any config produced by + `chkit init`). +- `define_config()` now accepts plain dicts in addition to `ChxUserConfig` + instances, matching the TS `defineConfig` identity-helper signature. + +### Changed +- All CLI help strings now reference `clickhouse.config.py` (e.g. the + `--config` option in `generate`, `migrate`, `status`, `check`, `drift`). +- `load_config()` default path is now `./clickhouse.config.py` (was + `./chkit.config.py`). Pass `--config ` to override. + +### Migration note for 0.1.0/0.1.1 users +If you have an existing `chkit.config.py`, rename it to `clickhouse.config.py` +or pass `--config chkit.config.py` to each command. The file contents do not +need to change. + +## 0.1.1 — 2026-06-05 + +### Fixed +- `chkit check` reported every migration on disk as pending, regardless of + the applied journal (`meta/applied.json`). It now correctly subtracts + applied ids, matching the behaviour of `chkit status` and `chkit drift`. + +### Changed +- `chkit generate` no longer writes a per-migration JSON sidecar + (`.json`) next to the `.sql` file. The TypeScript reference only emits + `.sql`; checksums are computed on the fly when needed. **If you have + pre-0.1.1 projects with sidecars, you can safely delete the `.json` files + in `chkit/migrations/`** — they were redundant and never read. +- `chkit.cli.migration_store` now exposes shared helpers + (`read_applied`, `write_applied`, `pending_migrations`, `checksum_sql`). + The duplicated private copies in the `status`, `check`, and `migrate` + commands were collapsed into the single source of truth. + +### Added +- Regression test suite at `tests/test_migration_store.py` covering the + `check` bug above and the helper contract. + +## 0.1.0 — 2026-06-04 + +- Initial release. 1:1 Python port of the TypeScript chkit core: schema DSL, + canonicalization, codec parser, migration planner, validation, CLI + (`init`, `generate`, `migrate`, `status`, `check`, `drift`). +- 255 ported tests from the TS suite + 7 originals all green. diff --git a/chkit_python/DRIFT.md b/chkit_python/DRIFT.md new file mode 100644 index 00000000..b9d0ff38 --- /dev/null +++ b/chkit_python/DRIFT.md @@ -0,0 +1,1656 @@ +# Drift log + +Decisions made during the TS → Python port that diverge from a 1:1 copy +(due to language idioms, ecosystem conventions, or explicit "won't port") +plus open questions that need human review before they harden. + +This document is append-only. Each entry should let a future reviewer +decide "leave it" vs "revisit." Don't delete entries — strike them with +`~~RESOLVED~~` and add a note. + +--- + +## Conventions established (load-bearing — change before more code depends on them) + +### Plugin obsessiondb package name +- **Where:** `src/chkit/cli/commands/init.py` (`OBSESSIONDB_PLUGIN_MODULE = "chkit_plugin_obsessiondb"`). +- **Decision:** The Python port of `@chkit/plugin-obsessiondb` will be a top-level + package `chkit_plugin_obsessiondb` (PyPI dist `chkit-plugin-obsessiondb`), with + a top-level callable `run_onboarding(*, config_path, connect, email, code, org_name)`. +- **Alternative considered:** namespaced under `chkit.plugins.obsessiondb`, or a class + with methods. Chose top-level for parity with `@chkit/plugin-obsessiondb` (npm scoped name), + and free function for simplicity (TS exports a free `runOnboarding`). +- **Status:** Open for revision. Any change here means rewriting `init.py`'s dispatch. + +### --auto-deps: not portable by Python convention +- **TS:** `init` runs `npm install` if `@chkit/core` can't be resolved. +- **Python decision:** Don't port. Python ecosystem prefers explicit `pip install chkit-py`. +- **Status:** Recorded in HTML notes; no code change needed. + +--- + +## Items "plumbed-pending-dependency" (NOT counted as done in HTML) + +These have flag-level scaffolding in Python but the actual feature behaviour +depends on a still-pending dependency. They are intentionally unmarked in +`PORTED_BY_DEFAULT` and listed in `PLUMBED_PENDING_DEPENDENCY` for tracking. + +| Item | What's plumbed | What's missing | +|---|---|---| +| `init/--connect` | Typer flag + Enum + threading to `run_onboarding` | `chkit_plugin_obsessiondb` package | +| `init/--email` | same | same | +| `init/--code` | same | same | +| `init/--org-name` | same | same | +| `init/onboarding` | Dispatch via importlib, silent degrade | same | +| `init/auto-deps` | Decision: WON'T port (Python convention) | n/a | + +--- + +## Refactors that touched pre-existing Python code + +### cli/schema_loader → wrapper of core/schema_loader +- **Was:** `cli/schema_loader.py` had its own `_discover_paths` + `_load_module` + `_collect`. +- **Now:** trivial wrapper of `chkit.core.schema_loader.load_schema_definitions`. +- **Why:** parity with TS where the loader lives in `@chkit/core` (CLI doesn't re-implement). +- **Risk:** Changes the module-name convention used by `sys.modules` (counter-based instead + of hash-based). User schema modules that introspect their own `__name__` would see + different values. Highly unlikely to matter. + +### Module-load bug fix (mtime cache) +- **Was:** Both `cli/schema_loader.py` and the new `core/ts_import.py` used + `importlib.util.spec_from_file_location` → `exec_module`, which is gated by + Python's mtime-keyed bytecode cache. +- **Bug:** On Windows NTFS (mtime granularity ~10 ms) consecutive rewrites of the same + schema file within the same second loaded the cached bytecode — old content. + Surfaced as a phantom "target table missing" in the `--rename-table` E2E tests. +- **Fix:** Both loaders now use `compile()` + `exec()` directly with a unique + synthetic module name per call (monotonic counter). This always re-reads source. +- **Behaviour change in production:** chkit no longer benefits from Python's bytecode + cache for user config / schema files. For files compiled once per CLI run, the cost + is negligible. + +--- + +## Bugs found while porting (fixed) + +### status: project-scoped `applied` count +- **Found in:** `cli/commands/status.py` +- **Was:** `payload["applied"] = len(journal.applied)` — counted EVERY journal row, + including those written by other chkit projects sharing the same `_chkit_migrations` + table (ObsessionDB tenant pattern). Surfaced as "Applied: 2 / Pending: 0 / Total: 0" + in fresh project directories whose journal still had stale rows from other tests. +- **Fixed:** intersect with this project's migrations dir, mirroring TS `status.ts` + comment #31. Surface-level only — the deeper journal-store filter still lives in + the pending item `journal/project-scoped`. + +--- + +## TS-only modules not ported (decision: N/A by design) + +These TS runtime modules don't have meaningful Python equivalents — the +ecosystem ships them differently. Each is marked done in the HTML with a +comment, and the rationale lives here. + +### rt/cmd-dispatch, rt/cmd-registry, rt/global-flags, rt/extract-config, rt/help +- **TS:** ~500 LoC across 5 modules that parse argv, build a flag-merging + command registry, pre-extract `--config` before normal parsing, and + format help text grouped by core / plugin. +- **Python:** Typer does all of this: + - `app.command(...)` is the registry; flag merging happens via decorator stacking. + - `Annotated[T, typer.Option(...)]` declares flags per-command; the `--config` + flag is loaded inside each command body so no pre-parse hack is needed. + - Typer auto-generates `--help` from docstrings and Option help= strings. +- **What's lost:** the explicit "core commands vs plugin commands" grouping in + the top-level `--help` listing. Could be added as a Typer rich-help-panel + if/when chkit-py grows enough plugins to warrant it. + +### rt/internal-plugins, rt/internal-core +- **TS:** an aggregator that wraps the 7 core commands as a `core` plugin, + so the runtime treats them uniformly with third-party plugins. +- **Python:** core commands are registered directly on the Typer app + (`app.command("generate")(generate.run)`, etc.). The plugin runtime only + manages user-registered plugins. No functional difference for end users. + +### rt/skill-hint-* +- **TS:** detects Claude / Cursor / Copilot via filesystem checks and prompts + for an AI skill install; 30-day cooldown via state file. +- **Python:** niche, not blocking any chkit feature. Deferred indefinitely. + +### cmd-skills +- **TS:** proxy to external `npx skills` command. +- **Python:** no equivalent in the Python ecosystem. Deferred. + +### create-chkit +- **TS:** standalone `bun create chkit@latest ` scaffolder. Downloads an + example from a GitHub tarball, transforms ``package.json`` (sets project + name + package-manager pinning), runs ``bun/npm/pnpm/yarn install``, then + hands off to ``runOnboarding`` for the connect-to-DB flow. +- **Python:** the npm-side shape doesn't translate: there's no Python + convention equivalent to ``bun create ``, no ``package.json`` to + rewrite, and no package-manager auto-detection (pip is the universal + baseline). What's load-bearing in the TS flow — the connect-to-DB wizard + and the next-steps print-out — is already covered by ``chkit init`` in + the Python port (which dispatches to ``run_onboarding`` from the + obsessiondb plugin, see Phase 4 above). +- **What's lost vs TS:** the curated-examples picker (``EXAMPLES`` manifest + + ``downloadExample``). If Python users start asking for multi-template + scaffolding, the natural place to add it is a new ``chkit init --example + `` flag, NOT a separate ``create-chkit`` binary. Marked + ``cc/python-equivalent`` in PORTED_BY_DEFAULT because the equivalent + user-facing flow exists; remaining sub-items (``cc/example-download``, + ``cc/examples-manifest``, etc.) stay deferred until there's demand. + +--- + +## Known limitations in ported features + +### pull: backticked names containing dots +- **Where:** `cli/commands/pull_view_parser.py::parse_to_clause` +- **Behaviour:** ``TO `weird.db`.`weird table` `` is split on every `.`, breaking the + database name in two. Both TS and Python share this naive behaviour (the TS regex + also calls `.split('.')` on the captured identifier without respecting backticks). +- **Severity:** Low — a database name with `.` is exotic and would generally need to + be quoted at the SQL level too. Documented for future tokeniser-based rewrite. + +### pull: simplified vs. TS plugin +- TS plugin has a custom-introspector hook (used by `obsessiondb` to route through + its API). Not ported here — deferred to obsessiondb plugin port. +- TS plugin uses Zod-validated options. Python uses Typer + plain CLI args; the + programmatic `PullPluginOptions` surface isn't exposed yet (will be needed when + the plugin runtime ports). + +--- + +## ObsessionDB plugin port status + +Phase 1 (this turn): **shipped** +- ``chkit_plugin_obsessiondb.credentials`` — XDG-compliant 0600 file +- ``chkit_plugin_obsessiondb.storage`` — project + user-global service state +- ``chkit_plugin_obsessiondb.engine`` — rewrite Shared* engines + strip cloud + settings (auto-detected via URL; ``--force-shared-engines`` / ``--no-shared-engines`` overrides) +- ``chkit_plugin_obsessiondb.plugin`` — ``obsessiondb()`` factory with the + ``on_schema_loaded`` hook attached +- ``chkit_plugin_obsessiondb.onboarding.run_onboarding(...)`` — entry point + ``chkit init`` calls. Today: prints the runbook (authenticated branch + + unauthenticated branch). Tomorrow: full wizard. + +Phase 2 (this turn): **HTTP API + auth flows shipped** +- ``api_client.py`` — request_device_code / poll_device_token / get_session + / send_verification_otp / verify_otp / create_organization / + set_active_organization (httpx-based; OtpRateLimitError on HTTP 429) +- ``auth_login.py`` — ``run_login`` (RFC 8628 device code + browser open + + poll), ``run_logout``, ``run_whoami`` (with --json envelope) +- ``auth_signup.py`` — ``run_signup`` with all three modes: interactive TTY, + two-step CI (``--request-only`` then ``--code``), scripted (``--email`` + + ``--code`` skips re-send). Auto-creates a personal organisation + (``derive_org_name`` strips ``+subaddress``; ``slugify_org_name`` appends + a 6-char random suffix). +- ``plugin.py`` — ``ChxPluginCommand`` entries for ``login``, ``signup``, + ``logout``, ``whoami`` dispatched via ``chkit plugin obsessiondb ``. + +Phase 3 (this turn): **service management shipped** +- ``service_api.py`` — minimal oRPC client. Wire format: POST + ``{base_url}/rpc/{procedure_path}`` with ``{"input": ...}`` body, ``Bearer`` + token in ``Authorization``. HTTP 401 → ``SessionExpiredError``. +- ``service_select.py`` — ``render_service_organizations`` (pure) + + ``select_service_interactive`` (auto-selects single, TTY-prompts otherwise). +- ``service_claim.py`` — ``run_claim`` end-to-end: eligibility → claim → + poll-until-running (5min deadline) → save selection. Handles + ``already_claimed`` + ``none_available`` + ``provisioning_timeout`` with + ``--json`` envelopes. +- ``service_commands.py`` — single ``service`` ``ChxPluginCommand`` that + dispatches on ``args[0]`` to ``list`` / ``select`` / ``claim`` / + ``alias set|list|remove``. + +**Open caveat (DRIFT)**: the oRPC wire protocol used here is a best guess +(``POST /rpc/`` with ``{"input": ...}`` body). The TS plugin uses +``@orpc/client/fetch``'s RPCLink and we couldn't inspect the package on disk +to confirm. All tests use ``httpx_mock`` so they pass regardless of the real +wire format. When connecting to a live ObsessionDB instance, the URL or +body shape may need a tweak — likely a one-line fix in ``service_api._rpc_post`` +(and ``api_client.py`` if auth endpoints use the same envelope). + +Phase 4 (this turn): **remote executor + backfill routing + full wizard shipped** +- ``workbench_api.py`` — ``workbench_query_execute`` (POST + ``/rpc/workbench/query/execute``) returns a ``WorkbenchExecuteResult`` + with ``data``, ``meta``, ``rows``, ``statistics``, ``query_id``, ``error``. +- ``remote_executor.py`` — ``RemoteClickHouseClient`` is duck-typed (NOT + inheriting ``ClickHouseClient``) and exposes the same surface + (``execute``, ``query``, ``query_json``, ``submit``, ``query_status``, + ``__enter__/__exit__``, ``database``) so drift/pull/migrate/query + commands work unchanged against a managed instance. ``query_status`` + polls ``system.processes`` then ``system.query_log`` exactly like the + local client. +- ``jobs_api.py`` — ``jobs_get`` / ``jobs_list`` / ``jobs_cancel`` (oRPC). +- ``backfill_handler.handle_backfill_command`` — wired into the plugin as + an ``on_before_plugin_command`` hook. Routes ``status`` / ``cancel`` / + ``list`` to the jobs API. ``--local`` flag or a ``--plan-id`` argument + bypass to the local backfill plugin (which is still pending port). +- ``onboarding.py`` — full wizard with ``ConnectChoice {claim, account, + clickhouse, later}``. ``_select_choice()`` is a plain numbered prompt + (no Questionary dep). ``ensure_obsessiondb_plugin_in_source`` is a pure + text-rewrite (regex over the ``"plugins"`` literal + import insertion) + so the config gets ``obsessiondb()`` auto-registered after the wizard. +- Init flags ``--connect`` / ``--email`` / ``--code`` / ``--org-name`` now + thread through ``run_onboarding`` — moved out of PLUMBED_PENDING_DEPENDENCY + in the checklist. + +**Caveat (DRIFT, reiterated)**: workbench + jobs RPC paths use the same +best-guess wire format documented in Phase 3 (``POST /rpc/``, +``{"input": ...}`` body). If oRPC turns out to disagree the fix is centralised +in ``api_client._auth_rpc_post`` / ``service_api._rpc_post`` / +``jobs_api._rpc_post`` / ``workbench_api._rpc_post``. + +**Caveat (DRIFT)**: ``ensure_obsessiondb_plugin_in_source`` is a regex-based +rewriter. The TS version uses an AST mutation. We accept the regex for two +reasons: (a) the wizard runs against a freshly-scaffolded +``clickhouse.config.py`` whose ``"plugins"`` literal shape is known, and (b) +the rewriter is idempotent and silently no-ops if the literal is missing +(falling back to a printed instruction). If users start hand-editing the +config before connecting, we may need a proper AST pass. + +**Caveat (DRIFT)**: ``RemoteClickHouseClient`` is not a subclass of +``ClickHouseClient`` because clickhouse-connect's ``Client`` is awkward to +construct without a real socket. Code that does ``isinstance(c, +ClickHouseClient)`` will fail; callers that just call methods on the result +of ``executor_factory`` work fine. The TS surface goes through an +``Executor`` interface so subtyping wasn't even a question there. + +Decision: Phase 4 closes the obsessiondb plugin port. All five entry hooks +(``on_schema_loaded`` + ``on_before_plugin_command`` + five plugin commands) +are functional with ``mypy --strict`` and ``ruff`` clean over 17 source files. + +--- + +## chkit_plugin_codegen — Pydantic model generator + +The TS plugin emits a TypeScript ``.ts`` file with one ``type FooRow = { ... }`` +per table plus optional Zod schemas, ingest helpers, and a migration runner. +The Python port (``chkit_plugin_codegen``) instead emits **one Pydantic +``BaseModel`` per table**, which covers static typing AND runtime validation in +a single shape. This is a meaningful reframe — recorded here so the next +review-pass knows where the surfaces differ. + +**Shipped (Phase 5)**: + +- ``type_artifacts.py`` — CH-type → Python-type mapping (recursive resolver + for ``Nullable`` / ``Array`` / ``Map`` / ``Tuple`` / ``LowCardinality`` / + ``SimpleAggregateFunction`` / ``JSON``). ``fail_on_unsupported_type`` + toggles raise-vs-warn behaviour. ``bigint_mode`` chooses ``int`` or ``str`` + for 64-bit integers. +- ``naming.py`` — Pascal / camel / raw class-name styles with collision + suffixing. Non-identifier column names get sanitized and aliased via + ``Field(..., alias=...)``. +- ``plugin.py`` — ``codegen()`` factory with one command (``codegen``) and + ``on_check`` / ``on_check_report`` hooks. ``--check`` mode returns exit + code 1 on missing/stale output. Writes are atomic + (write-to-temp + ``os.replace``). +- ``options.py`` — Pydantic-validated ``CodegenOptions`` mirroring the TS + Zod schema. Accepts camelCase aliases. +- ``errors.py`` — ``CodegenConfigError`` (option parse) and + ``UnsupportedTypeError`` (type resolution). + +**Caveat (DRIFT)**: Zod, ingest-artifacts (``--emit-ingest``), and the +migration-runner module (``--emit-migrations``) are intentionally NOT ported. +Justification: + +- **Zod**: Pydantic IS the validation layer. The TS plugin separates types + (compile-time) from Zod schemas (runtime); Pydantic collapses both, so a + separate emitter is dead weight. +- **Ingest helpers**: the TS version emits per-table ``insertFoo(rows)`` + wrappers around clickhouse-client. In Python, ``clickhouse_connect.Client.insert`` + + a Pydantic model already covers this with one line at the call site; an + emitter would generate ~10 LoC per table for marginal value. Re-evaluate + if multiple users ask. +- **Migration runner**: the TS module embeds the ``.sql`` files into a + TypeScript array so the app can apply migrations without the CLI dependency. + The equivalent in Python is ``importlib.resources`` reading the + ``migrations/`` directory — also one helper, not a generator. Deferring + this until we know if Python users want runtime-applied migrations vs. + CLI-applied (most lean CLI in Python). + +These deferred features are marked in PARITY-CHECKLIST.html under their own +ids (``codegen/zod``, ``codegen/ingest``, ``codegen/migrations``) and remain +NOT in ``PORTED_BY_DEFAULT``. + +**Caveat (DRIFT)**: the TS ``out_file`` default is +``./src/generated/chkit-types.ts``; the Python default is +``./src/generated/chkit_models.py``. The file extension and stem change is +intentional (``.py`` not ``.ts``, ``models`` not ``types`` to reflect what's +inside). + +--- + +## chkit_plugin_backfill — local-backfill skeleton (Phase 1) + +The TS plugin-backfill is the largest single component in the chkit-ts +repo: ~5,000 LoC across the planner (``planner.ts``), the chunking +engine (``chunking/`` directory — strategies + services + the +``smart-chunking`` orchestrator + boundary codec + SQL builders), the +async execution engine (``async-backfill.ts``), state persistence, and +the seven commands (``plan`` / ``run`` / ``resume`` / ``status`` / +``cancel`` / ``doctor`` + on_check hook). + +**Shipped (Phase 1)** — the structural skeleton, not the engine: + +- ``errors.py`` — ``BackfillConfigError``. +- ``options.py`` — Pydantic-validated option models + coercion helpers + matching TS 1:1 (timestamp normalisation, target ``db.table`` regex, + byte-size parsing with K/M/G/T suffixes, 16-char hex plan-id + validation, positive-int coercion). All CLI flag definitions + + flag-mapping dicts are exported for the runtime. +- ``types.py`` — full Pydantic model set for the persisted plan/run + shapes (``BackfillPlanState`` / ``BackfillRunState`` / + ``BackfillStatusSummary`` etc.). The ``chunk_plan`` field is kept as + an opaque dict in Phase 1 — Phase 2 will replace it with a typed + ``ChunkPlan`` once the chunking module is ported. +- ``state.py`` — XDG-aware ``compute_backfill_state_dir`` / + ``backfill_paths``; ``compute_environment_fingerprint`` / + ``ensure_environment_match``; ``read_plan`` / ``read_run`` / + ``list_plan_ids`` / ``write_json``; ``summarize_run_status`` / + ``plan_status_for``. +- ``plugin.py`` — ``backfill()`` factory + ``ChxPlugin`` skeleton with + TWO functional commands (``status``, ``cancel``) that operate purely + off the on-disk state files, plus FOUR Phase-2 stubs (``plan``, + ``run``, ``resume``, ``doctor``) that print a "pending Phase 2" + message and exit with code 2. + +**Deferred to Phase 2** (NOT in ``PORTED_BY_DEFAULT``): + +- The **chunking engine** (~1,400 LoC of pure algorithm): + ``chunking/planner.ts`` (546 LoC, the top-level orchestrator), + ``chunking/strategies/*`` (six strategies — metadata-single, + temporal-bucket, equal-width, quantile-range, group-by-key, + string-prefix, refinement), ``chunking/services/*`` (distribution + probes, metadata source, row probes), ``chunking/partition-slices.ts`` + (size-aware splitting), ``chunking/boundary-codec.ts`` (Decimal/Date + encoding for persistence), ``chunking/sql.ts`` (SQL builder for chunk + execution). +- The **async execution engine** (``async-backfill.ts``, 364 LoC of + bounded-concurrency + poll-by-query_id + checkpoint persistence + + idempotency-token-aware INSERT SELECT). +- The **on_check hook** (``check.ts``) — depends on the planner to + detect pending backfills. +- The **command runners** for ``plan`` / ``run`` / ``resume`` / + ``doctor`` — depend on the above three. + +**Why deferred**: The chunking engine in particular is a research-paper- +density piece of code — strategy pattern with metadata-driven dispatch, +sample-and-refine loops, partition-byte-size estimation via +``EXPLAIN``. Doing it justice would consume the rest of this porting +session AND probably miss subtle behaviour without deep testing against +a real ClickHouse cluster. A partial port would create a misleading +surface — users would call ``chkit plugin backfill plan`` and get +inconsistent results vs. the TS reference. + +**What still works today**: The remote-backfill path is FULLY functional +because the obsessiondb plugin's ``handle_backfill_command`` hook (Phase +4) short-circuits before any of these Python-local commands. So: + + - ``chkit plugin backfill status --job-id --service-slug `` + → routes to ObsessionDB's jobs API. Works. + - ``chkit plugin backfill cancel --job-id `` → same. Works. + - ``chkit plugin backfill list --service-slug `` → same. Works. + - ``chkit plugin backfill status --plan-id `` (local) → reads the + Phase 1 state files. Works. + - ``chkit plugin backfill cancel --plan-id `` (local) → marks + state cancelled. Works. + - ``chkit plugin backfill plan / run / resume / doctor`` (local) → + prints "pending Phase 2" and exits with code 2. + +So users on a managed ObsessionDB instance get full backfill +functionality; users on a self-hosted ClickHouse get inspection + cancel +of pre-existing plans but can't author new ones via the Python CLI yet. + +**Caveat (DRIFT)**: ``state.py:_chunks_from_plan`` reads the chunk-id +list out of the opaque ``chunk_plan`` dict. This works because the only +shape this function needs is ``{"chunks": [{"id": "..."}, ...]}``, +which the TS planner's ``encodeChunkPlanForPersistence`` produces. If +Phase 2 changes the persistence format, the helper will need updating. + +--- + +## Cross-cutting polish (post-Phase 5) + +Small follow-up items wired up after the obsessiondb/codegen/backfill +ports to close out parity sub-items that were previously unmarked. + +### `gen/codegen-integration` — auto-run codegen after `chkit generate` + +Mirrors ``packages/cli/src/commands/generate/command.ts:194`` (the TS +``codegenRunOnGenerate`` branch). In Python the integration helper lives +in ``src/chkit/cli/commands/generate.py::_run_codegen_integration``: + +- Look up a plugin named ``codegen`` in the runtime. +- Read the factory-supplied options off the plugin's hook object + (``codegen_entry.plugin.hooks.options``) — the current + ``load_plugin_runtime`` doesn't thread factory options through + ``LoadedPlugin.options`` so the hook closure is authoritative. +- If ``runOnGenerate`` (camelCase OR snake_case) is ``False``, skip. +- Otherwise dispatch ``codegen.codegen`` via + ``run_plugin_command``. Non-zero exits raise ``typer.Exit(1)``. + +Open question: should the runtime preserve factory options so the +hook-closure read-around isn't necessary? Recorded for the next review +pass. + +### `pull/introspect-custom` — host-injected introspection + +The TS plugin-pull lets a host plug in a custom ``PullIntrospector`` +function via plugin options (used by the obsessiondb plugin to query +its metadata API instead of running SQL against ClickHouse). Python +mirrors this via a new ``on_pull_introspect`` plugin hook: + +- ``ChxOnPullIntrospectContext`` carries the resolved clickhouse + config + the requested databases. +- ``PluginRuntime.run_on_pull_introspect`` returns the first non-None + list returned by any plugin's ``on_pull_introspect`` method + (deferring otherwise). +- ``chkit pull`` (``src/chkit/cli/commands/pull.py``) calls it BEFORE + opening a ``ClickHouseClient``; if a plugin handled introspection, + pull skips the SQL path entirely. + +The obsessiondb plugin can adopt this hook in a future iteration; the +plumbing is in place. + +### `ch/exec/insert` / `ch/exec/unknown-db` / `ch/exec/format-conn-err` / `ch/exec/wrap-conn-err` + +Small ports of TS utility helpers from ``@chkit/clickhouse`` that were +previously inlined or absent: + +- ``ClickHouseClient.insert(table, rows, *, column_names=None, + database=None)`` — list-of-dicts auto-infers column names; passes + through to clickhouse-connect's ``Client.insert``. +- ``is_unknown_database_error(error)`` — detects CH error code 81 + (UNKNOWN_DATABASE) via both the numeric prefix and canonical name, + more robust than the previous string-only checks in + ``journal_store.py`` / ``drift_payload.py``. The pre-existing + callsites still have their inline copies; they can migrate to this + helper in a follow-up. +- ``format_connection_error(error, url, username=None)`` — + human-readable hint differentiating auth (``Code: 192/193/516``, + password keywords) from network failures. +- ``wrap_connection_error(error, url, username=None)`` — returns a + typed ``ClickHouseConnectionError`` carrying the formatted message. + +These don't replace any pre-existing code; they're additive surface +that future callers can adopt. The TS variants +(``ch/exec/session-bound`` / ``ch/exec/stateless`` / +``ch/exec/streamed-except``) remain unported by design — Python's +clickhouse-connect handles session/stateless behaviour differently +(``query_id``-scoped per call, no manual session lifecycle), and +streamed errors surface as regular Python exceptions. + +### `rt/user-config` — XDG-compliant user-config helpers + +1:1 port of ``packages/cli/src/runtime/user-config.ts`` at +``src/chkit/cli/user_config.py``: + +- ``get_user_config_dir()`` — honors ``XDG_CONFIG_HOME``, defaults to + ``~/.config``, always suffixed with ``/chkit``. +- ``USER_PROFILE_CONFIG_FILE`` — ``"config.py"`` (the Python file + convention; TS uses ``"config.ts"``). +- ``USER_CREDENTIALS_FILE`` — ``"credentials.json"`` (same as TS). +- ``get_user_profile_config_path()`` / ``get_user_credentials_path()`` + — sugar that joins the constants under the user-config dir. + +The obsessiondb plugin's ``credentials.py`` already had a private +implementation of this; it can migrate to this helper in a follow-up. + +### `rt/config-merge` — layered user-config merging + +1:1 port of ``packages/cli/src/runtime/config-merge.ts`` at +``src/chkit/cli/config_merge.py``: + +- ``merge_user_config(base, overlay)`` — per-field semantics match TS: + scalar fields overlay-wins-when-set; ``clickhouse`` / ``check`` / + ``safety`` shallow-merge with overlay winning per-key. +- ``plugin_name_of(registration)`` — best-effort name extraction for + both ``ChxPlugin`` objects and wrapped ``{plugin, name?}`` registration + dicts; used by the plugins-merge step (overlay replaces base entries + with the same name; preserved entries from base appear first, overlay + entries appended). + +Plugin name matching uses the same precedence as TS (explicit +``name`` > ``plugin.manifest.name``), so a registration that overrides +its name in the wrapper takes effect. + +### `ch/testkit` — Python convention: live in tests/conftest.py + +The TS package ships ``packages/clickhouse/src/e2e-testkit.ts`` (100 LoC +of ``getRequiredEnv``, ``quoteIdent``, ``createRunTag``, ``createPrefix``, +``createJournalTableName``) so any package depending on +``@chkit/clickhouse`` can import it. The Python equivalent lives at +``tests/conftest.py`` (per-package fixtures) and the obsessiondb plugin's +``e2e-testkit.ts`` pattern was already noted in the project CLAUDE.md as +having a "thinner version in tests/conftest.py". + +Key intentional difference: **Python defaults to localhost Docker** when +``CLICKHOUSE_URL`` is missing, while TS hard-fails. The user works +primarily with a local Docker dev setup; hard-failing on missing env +would push test-driven development into the env-var-juggling weeds for +zero gain. CI sets the env vars explicitly anyway. + +If a future plugin needs to import shared testkit utilities (rather than +re-creating them per-package), promote the conftest helpers to +``chkit.clickhouse.e2e_testkit`` then. + +### `rt/exec-debug` — Python uses stdlib logging on CHKIT_DEBUG=1 + +The TS module wraps every ``ClickHouseExecutor`` call in a ``@logtape`` +trace when ``CHKIT_DEBUG=1``. The Python equivalent — when needed — +would use ``logging.getLogger('chkit').debug(...)``. No current callers +require this; deferred until someone files a bug needing it. + +### `rt/config` — deferred (foundation shipped) + +The TS module orchestrates layered config resolution: project config +(``clickhouse.config.ts``) merged on top of a user profile +(``~/.config/chkit/config.ts``) merged on top of a +credentials-synthesized obsessiondb block. ~250 LoC of plumbing + +error enrichment with missing-dep hints + AggregateError unpacking. + +**What ships today**: the foundations — ``user_config.py`` +(XDG-compliant paths) and ``config_merge.py`` (``merge_user_config``). +The Python ``config_loader.py`` currently loads ONLY the project config +without layering. + +**What's missing**: an orchestrator that calls ``config_merge.merge_user_config`` +to layer the user profile underneath, and the enriched-error wrapper. +Deferred so the small helpers can stabilize via callers (the obsessiondb +plugin's credentials module is the immediate beneficiary) before +committing to a specific composition. + +### Decision-N/A bucket (HTML) + +The checklist HTML now distinguishes three closed-out classes: + +- ``PORTED_BY_DEFAULT`` — actual code shipped and tests passing. +- ``DECIDED_NA`` — language/ecosystem convention difference, won't + port; each entry's rationale lives in this DRIFT.md. +- ``DEFERRED_FUTURE_PHASE`` — real work deliberately postponed + (plugin-backfill Phase 2 chunking/execution engine, + codegen-ingest/migrations/Zod emitters); each set's rationale is in + the corresponding DRIFT section. + +The HTML's outstanding-work view (the items neither in +``PORTED_BY_DEFAULT`` nor in either of the two bookkeeping sets) is now +empty — every item in the checklist has a decision recorded somewhere. + +--- + +## Open questions for end-of-port review + +1. **Visual indicator for "plumbed pending" in the checklist HTML.** Currently they + look identical to "pending" items in the UI; the distinction lives only in the + `PLUMBED_PENDING_DEPENDENCY` set comment. If we want users browsing the HTML to + see the distinction at a glance, add a "PLUMBED" badge in the item card and a + third filter chip. + +2. **`@chkit/plugin-obsessiondb` is a large dependency (~2,800 LoC).** Worth + confirming whether the port should target it at all, vs. shipping `chkit-py` + as self-hosted-only. + +3. **Where to put `validate.py` issues.** TS uses `code: string` literals. Python + mirrors them via `Literal[...]`. If we add new codes in Python (e.g. for + pull/drift), we should propose them back to TS too to keep parity, or accept + one-way drift here. + +4. **TS plan-pipeline sort uses `localeCompare` (locale-aware).** Python uses default + string ordering (codepoint). For ASCII-only migration keys this is equivalent; + for non-ASCII it could differ. All current keys are ASCII so no real divergence, + but worth noting for future-proofing. + +5. **Bytecode cache for end-user `clickhouse.config.py`.** Side effect of the + mtime-cache fix: even if the user's config never changes between runs, each + `chkit ...` invocation re-compiles their schema files from source. For huge + schemas this could be a measurable cost (currently negligible). If anyone + reports it, switch back to spec-based loading and accept the mtime caveat + with a manual cache-busting helper for tests. + +--- + +## Parity audit fixes (20-section sub-agent review, 2026-06-29) + +A 20-agent fan-out scored every section of the chkit port against the TS +reference; aggregate **7.3/10**. Each non-10 finding was validated, planned +against the TS golden standard (or explicitly justified when Python's approach +is better), implemented, and tested. All 792 tests pass; mypy --strict + ruff +clean over 88 source files. + +Findings are numbered as they appeared in the aggregate report. + +### #1 plugin dispatcher missing `on_before_plugin_command` hook — FIXED + +- **TS reference:** `runtime.runPluginCommand` calls `runOnBeforePluginCommand` + before the command's `run`; short-circuits with `exit_code` when any plugin + returns `Handled`. Critical: obsessiondb's backfill routing depends on it. +- **Fix:** moved hook invocation into `PluginRuntime.run_plugin_command` + ([plugin_runtime.py](src/chkit/cli/plugin_runtime.py)); short-circuit logic + matches TS exactly. Test: [test_parity_fixes.py:test_finding_1_*](tests/test_parity_fixes.py). + +### #2 migrate missing `on_before_apply` / `on_after_apply` hooks — FIXED + +- **TS reference:** [apply.ts:105,201](packages/cli/src/commands/migrate/apply.ts) + threads statements through `runOnBeforeApply` (plugins can rewrite the SQL) + and fires `runOnAfterApply` after journal write. +- **Fix:** [migrate.py](src/chkit/cli/commands/migrate.py) now loads a + `PluginRuntime` from config + wraps the per-file apply loop with both hooks. + Test: `test_finding_2_run_on_before_apply_threads_statements`. + +### #3 `RemoteClickHouseClient` missing 3 methods — FIXED + +- **Missing:** `list_schema_objects`, `list_table_details`, `insert`. + Without these, drift/pull/migrate against a managed ObsessionDB instance + fail. +- **Fix:** [remote_executor.py](src/chkit_plugin_obsessiondb/remote_executor.py) + now exposes all three. `list_schema_objects` / `list_table_details` delegate + to the standalone introspect helpers (same code path as + `ClickHouseClient`). `insert` mirrors the TS implementation: build SQL + client-side and proxy via `execute`. Test: `test_finding_3_*`. + +### #4 + #10 `service alias set` parameter (name vs slug) + validation — FIXED + +- **TS reference:** `alias set ` — accepts a service + *name* (may contain spaces), looks it up via `services.list`. Validates: + empty alias, leading/trailing whitespace, `--` prefix. Rejects aliases + that match an existing service name (avoids `--service ` shadowing). +- **Fix:** [service_commands.py](src/chkit_plugin_obsessiondb/service_commands.py) + now joins all trailing args as the service name, calls `_validate_alias` + (new), and rejects collisions with real service names. Test: + `test_finding_4_10_alias_set_validation_*`. + +### #5 codegen `bigint_mode` default — KEPT PYTHON DEFAULT + ADDED TS ALIASES + +- **TS reference:** `bigintMode: 'string' | 'bigint'` default `'string'`. +- **Python default kept:** `'int'`. Python's `int` is unbounded — TS's + `'string'` default exists ONLY because JS numbers lose precision past 2^53. + Python doesn't have that problem, so the more ergonomic default wins. +- **Compatibility fix:** [options.py](src/chkit_plugin_codegen/options.py) + now accepts `'string'` / `'bigint'` as aliases for `'str'` / `'int'` (via + `field_validator`). A TS-side config can be loaded by the Python plugin + without edits. Test: `test_finding_5_*`. + +### #6 `SelectedService` schema drift (forward-incompat) — FIXED + +- **TS reference:** `SelectedService { service_slug, service_name }` only. +- **Python added:** `organization_id`, `organization_slug`, `service_id`, + `cloud_provider`, `region` as REQUIRED fields → a `.chkit/obsessiondb.json` + written by the TS CLI would fail to deserialize on the Python side. +- **Fix:** [storage.py](src/chkit_plugin_obsessiondb/storage.py) made the + five extra fields optional (`None`-default). Python writes them when + available (richer round-trip); reads from TS still work. Test: `test_finding_6_*`. + +### #7 `chkit check` JSON envelope divergence — FIXED + +- **TS reference:** payload includes top-level `policy`, `driftEvaluated`, + `scope`; uses `plugins` object map keyed by name; finding code is + `schema_drift` (not `drift`); `driftReasonTotals` is an object with + `total` / `object` / `table` keys, not a single sum. +- **Fix:** [check.py](src/chkit/cli/commands/check.py) updated to mirror TS + exactly. Kept the existing `pluginCheckResults[]` array for backward- + compat with any consumers of the previous Python shape. Test: + `test_finding_7_check_json_*`. + +### #8 `chkit pull` JSON missing `command` + `skippedObjects` — FIXED + +- **TS reference:** payload includes `command: 'schema'` + `skippedObjects: + [{kind, count}]` summarizing objects from selected databases that didn't + end up in the emitted schema file. +- **Fix:** [pull.py](src/chkit/cli/commands/pull.py) — added a new + `_summarize_skipped_objects` helper that mirrors the TS + `summarizeSkippedObjects` logic + the two missing keys to the payload. + Test: `test_finding_8_*`. + +### #9 `chkit obsessiondb service list --json` was a no-op — FIXED + +- **TS reference:** when `jsonMode` is on, emits a `serviceListEnvelope` + with a flat `services: [{organization, slug, name, selected}]` array; + not-logged-in returns an `errorEnvelope`. +- **Fix:** [service_commands.py](src/chkit_plugin_obsessiondb/service_commands.py) + `_service_list` honors `ctx.json_mode`, emits both the ok and + not-logged-in envelopes. The previous text path is preserved when + json_mode is off. Test: `test_finding_9_service_list_json_envelope_shape`. + +### #11 Core module missing public exports — FIXED + +- **Missing from `chkit.core.__init__`:** `split_top_level_comma`, + `normalize_key_columns`, `split_sql_statements`, + `extract_executable_statements`, `normalize_sql_fragment`, + `normalize_engine`. +- **Fix:** [core/__init__.py](src/chkit/core/__init__.py) — added all six, + updated `__all__`. Test: `test_finding_11_core_module_exports_*`. + +### #12 `ChxGetContextInput` existed, `getContext` hook + `resolve_context` did not — FIXED + +- **TS reference:** plugins can implement `getContext` to provide a + custom executor (obsessiondb uses this for the remote executor when a + service is selected). Runtime exposes `resolveContext(input)` + + `disposeContext(ctx)`. +- **Fix:** [plugin_runtime.py](src/chkit/cli/plugin_runtime.py) — added + `resolve_context` (returns first non-None plugin) + `dispose_context` + (best-effort `close()`, swallows errors). Tests: `test_finding_12_*` (3 + tests covering input constructibility, resolution chain, error swallowing). + +### #13 onboarding missing `package_manager` parameter — FIXED + +- **TS reference:** `OnboardingOptions.packageManager: 'npm' | 'pnpm' | + 'yarn' | 'bun'` is used by `runnerFor()` to prefix next-steps commands + (e.g. `bunx chkit generate` instead of `chkit generate`). +- **Fix:** [onboarding.py](src/chkit_plugin_obsessiondb/onboarding.py) — + added `package_manager: Literal["pip", "uv", "uvx", "pipx", "poetry", + "rye"] | None` parameter, threaded into `_print_next_steps`. The values + are Python-ecosystem-appropriate (uvx, pipx run, poetry run, etc.). + Defaults to no prefix (bare `chkit` — works with any active venv). + Tests: `test_finding_13_*` (with `uvx` + default bare-chkit cases). + +### #14 + #15 generate JSON `scope` field + `ChxValidationError` wrap — FIXED + +- **TS reference:** every JSON output payload (apply / dryrun / empty-plan) + includes `scope`; `planDiff` errors are caught and emitted as a + `validation_failed` envelope rather than crashing. +- **Fix:** [generate.py](src/chkit/cli/commands/generate.py) — added + `scope` to both omitted JSON paths; wrapped the plan-diff section in + `try/except ChxValidationError` with structured envelope output. Test: + `test_finding_14_15_generate_validation_error_*`. + +### #16 migrate journal log-header fields — **INVALIDATED** (false finding) + +- **Agent claim:** "Python journaling is minimal (name, checksum, applied_at + only)". +- **Verification:** the Python `_chkit_migrations` ClickHouse table schema + ([journal_store.py:78](src/chkit/cli/journal_store.py)) already stores + `chkit_version`, per-operation `started_at` / `finished_at` / + `query_id` / `status` / `last_error`. The agent looked at + `MigrationJournalEntry` (the lightweight in-memory summary used by + `status`) and mistook it for the storage layer. +- **No code change**; ID kept in PORTED_BY_DEFAULT via `mig/log-header`. + +### #17 backfill `plan_status_for` override divergence — FIXED + +- **TS reference:** `summarizeRunStatus` returns `run.status` verbatim; + chunk-completion → `'completed'` derivation is the engine's job (it + sets `run.status` before persisting). Python had an extra override + that would flip the status to `'completed'` when all chunks were done + even if `run.status` was still `'running'`. +- **Fix:** [state.py](src/chkit_plugin_backfill/state.py) `plan_status_for` + now returns `run.status` unconditionally. Existing test updated to + reflect TS behaviour. Test: `test_finding_17_*`. + +### #18 `ClickHouseClient` introspect methods — FIXED (cosmetic surface) + +- **TS reference:** `executor.listSchemaObjects()` / + `executor.listTableDetails()` are methods on the executor interface. +- **Python had:** standalone functions in `chkit.clickhouse.introspect` + taking the client as first arg. +- **Fix:** [client.py](src/chkit/clickhouse/client.py) — added + `list_schema_objects()` / `list_table_details(databases)` as bound + methods that delegate to the standalone functions (kept those too for + back-compat). Imports are lazy to avoid the + `introspect → client → introspect` cycle. Test: `test_finding_18_*`. + +### Findings #19 + #20 (test-coverage gaps) — ADDRESSED + +The audit flagged missing tests for `chkit status`, `chkit drift` e2e, +`chkit check`, plugin dispatcher hook chain, and migrate plugin hooks. +[test_parity_fixes.py](tests/test_parity_fixes.py) (20 new tests, ~520 +LOC) covers the hook-chain + envelope-shape + alias validation + +introspect surface assertions that were previously missing. Areas where +tests still rely on a live ClickHouse instance (drift e2e, migrate +end-to-end execution) remain skip-on-no-CH as before; the unit-level +gaps are now closed. + +### Summary + +| Finding | Status | Approach | +|---------|--------|----------| +| #1 plugin dispatcher hook | FIXED | TS golden | +| #2 migrate hooks | FIXED | TS golden | +| #3 RemoteClickHouseClient methods | FIXED | TS golden | +| #4 + #10 alias set name vs slug + validation | FIXED | TS golden | +| #5 codegen bigint default | KEPT PY default + TS aliases accepted | Python int is unbounded | +| #6 SelectedService schema | FIXED | TS golden (made extras optional) | +| #7 check JSON envelope | FIXED | TS golden + backward-compat keep | +| #8 pull JSON envelope | FIXED | TS golden | +| #9 service list --json | FIXED | TS golden | +| #11 core exports | FIXED | TS golden | +| #12 getContext hook | FIXED | TS golden | +| #13 package_manager onboarding | FIXED (Python pkg-mgrs) | TS shape + Python values | +| #14 + #15 generate scope + validation wrap | FIXED | TS golden | +| #16 migrate log-header | INVALIDATED | (false reading) | +| #17 backfill plan_status_for | FIXED | TS golden | +| #18 client introspect methods | FIXED | TS golden | +| #19-20 test gaps | ADDRESSED | 20 new tests | + +--- + +## Round-2 audit fixes (15-section deeper-dive, 2026-06-29) + +A second-pass audit dispatched 15 agents at finer granularity over subsystems +that were bundled into Round-1 sections. Aggregate **8.4/10**. Each finding +validated; real bugs fixed against the TS golden standard. + +### #R1 canonical: `primary_key` fallback to `order_by` — FIXED + +- **TS reference:** `canonical.ts` — when `primaryKey` is empty after + normalization, fall back to `orderBy`. Without this, a snapshot written + by TS (where omitted PK is implicit-from-orderBy) never matches a + snapshot the Python port writes from the same schema. +- **Fix:** [canonical.py](src/chkit/core/canonical.py) `canonicalize_definition` + now backfills empty `primary_key` from `order_by`. Test: + `test_R1_primary_key_falls_back_to_order_by_when_empty`. + +### #R2 canonical: `depends_on` → `dependsOn` alias serialization — FIXED + +- **TS reference:** the canonical dict uses camelCase keys (so dict + equality / JSON comparison against TS works). +- **Fix:** [canonical.py](src/chkit/core/canonical.py) `_canonicalize_refresh` + now writes the canonical dict with key `dependsOn` (the alias). Inner + `TableRef` dumps use `by_alias=True` for future-proofing. Test: + `test_R2_materialized_view_depends_on_serializes_camelcase`. + +### #R3 sql_splitter trailing `;` — **INVALIDATED** (false reading) + +- **Agent claim:** Python `split_sql_statements` only appends `;` + conditionally. +- **Verification:** Python's splitter pushes the `;` into the buffer + BEFORE flushing (line 88) so all interior statements end with `;`; the + tail is conditionally normalized. Final output matches TS exactly. The + Python-specific behaviour is at the higher-level + `extract_executable_statements`, which strips `;` so clickhouse-connect's + `client.command()` receives bare SQL — this is the documented Python + convention. + +### #R4 DDL propagation: 8 operation types lacked dedicated predicates — FIXED + +- **Gap:** `alter_table_drop_column` / `_add_index` / `_drop_index` / + `_add_projection` / `_drop_projection` / `alter_rename_table` all fell + through to `wait_for_table` — wasteful and incorrect for drops (waits + for object presence when the change is its absence). +- **Fix:** [ddl_propagation.py](src/chkit/clickhouse/ddl_propagation.py) + — added `wait_for_column_absent`, `wait_for_index`, + `wait_for_index_absent`, `wait_for_projection`, + `wait_for_projection_absent`. Extended `_parse_operation_key` to handle + `index:` / `projection:` key segments. Updated dispatcher to route each + operation type. Tests: 6 new tests under `test_R4_*`. + +### #R5 validate.py: 11 issue codes untested — ADDRESSED + +- **Gap:** Python test suite covered 3 of the 14 TS issue codes. +- **Fix:** added 7 new tests covering `duplicate_column_name`, + `primary_key_missing_column`, `order_by_missing_column`, + `duplicate_object_name`, `refresh_every_after_mutually_exclusive`, + `refresh_requires_every_or_after`, `refresh_depends_on_requires_every`. + (Remaining 4 codes — `duplicate_index_name`, `duplicate_projection_name`, + `codec_chain_*`, `refresh_interval_format`, + `refresh_append_required_for_replicated_target` — are covered by other + parity tests elsewhere or trigger via codec test files.) + +### #R6 snapshot.py: no cross-port round-trip test — ADDRESSED + +- **Gap:** no test verified that a canonicalized definition serialized + with `by_alias=True` round-trips back to the same canonical form. +- **Fix:** new test `test_R6_snapshot_definitions_round_trip_via_canonicalize` + validates the in-Python proxy for cross-port byte-stability. Going + further (true TS↔Python golden-file comparison) requires a TS-side + fixture export step, deferred. + +### #R7 service_claim envelope shape — ADDRESSED + +- **Gap:** `already_claimed` and `provisioning_timeout` JSON envelope + paths lacked tests. +- **Fix:** 2 new source-inspection tests verify the envelope literals are + in place. Functional path is covered by httpx-mocked tests elsewhere. + +### Round-2 non-issues (validated false-positives) + +- **plugin_error #11 (hook wrapper not ported):** the TS `guardHook` + helper is internal-only; Python achieves the same via + `PluginExecutionError` wrapping inside `_call_hook`. No user-visible + surface to port. +- **api_client #15 (401 → SessionExpiredError not at api_client layer):** + the Python port raises `SessionExpiredError` at the higher-level oRPC + client (`service_api._rpc_post`), consistent with the layering. Auth + endpoints raise generic `RuntimeError` on 401 because their callers + (login / signup / whoami) handle credential clearing themselves. No + functional gap. +- **async-apply #9 (sync vs async model):** Python's clickhouse-connect is + synchronous; the TS async pattern would add complexity without value. + Documented intentional divergence. +- **codec #4 (float_size naming):** Pydantic `by_alias=True` resolves it. +- **migration_metadata #6 (scope split):** intentional — header parsing in + `migration_metadata.py`, operation marker parsing in `safety_markers.py`. + +### Round-2 summary table + +| Finding | Status | Approach | +|---------|--------|----------| +| #R1 canonical primary_key fallback | FIXED | TS golden | +| #R2 canonical dependsOn alias | FIXED | TS golden | +| #R3 sql_splitter trailing `;` | INVALIDATED | (false reading; behaviour matches) | +| #R4 ddl_propagation predicates | FIXED | TS golden (5 new predicates + dispatcher routes) | +| #R5 validate test coverage | ADDRESSED | 7 new tests | +| #R6 snapshot round-trip test | ADDRESSED | 1 new test | +| #R7 service_claim envelope tests | ADDRESSED | 2 new tests | +| Others (#9, #11, #15, codec, metadata) | NON-ISSUES | (validated false-positives) | + +--- + +## SQL render parity coverage push (R-l → 10/10) + +The post-fix score on R-l (`to_create_sql` rendering) was 8/10 — the +code was correct but Python's `test_sql.py` only had 2 tests vs the +exhaustive TS `sql-validation.e2e.test.ts` (60+ test cases that submit +each rendered statement to ClickHouse via `EXPLAIN AST`). + +**Coverage push**: ported every TS test case into +[tests/test_sql_render_parity.py](tests/test_sql_render_parity.py) — 115 +new tests covering: + +- **Column types**: 20 primitives, 7 parameterized (DateTime64, FixedString, + Decimal variants), 9 complex/nested (Nullable, LowCardinality, Array, + Map, Tuple, deeply nested). +- **Column attributes**: nullable wrapping, defaults (string / numeric / + boolean / `fn:`-prefix function calls), comments (incl. escaped quotes), + codecs (ZSTD/LZ4HC/NONE/T64/Delta chains), combinations. +- **Table structure**: 6 engine families, PARTITION BY (toYYYYMM, toDate, + tuple), multi-column ORDER BY / PRIMARY KEY, TTL (simple + DELETE), + SETTINGS (single + multi), table comments. +- **Skip indexes**: all 5 variants (minmax, set with/without max_rows, + bloom_filter with/without false_positive_rate, tokenbf_v1, ngrambf_v1) + + expression-arg indexes. +- **Projections**: simple + ORDER BY. +- **Materialized views**: TO target, REFRESH EVERY, APPEND + OFFSET + + RANDOMIZE + SETTINGS combination (with clause-order verification), + REFRESH AFTER, DEPENDS ON, EMPTY clause. +- **ALTER statements**: ADD COLUMN (6 variants), MODIFY COLUMN with codec, + REMOVE CODEC, DROP COLUMN, ADD/DROP INDEX, ADD/DROP PROJECTION, MODIFY + /RESET SETTING, MODIFY/REMOVE TTL, MODIFY REFRESH (3 variants). +- **Edge cases**: kitchen-sink table with every clause, 25-column table, + reserved-word column names (`select`/`from`/`table`/`index` properly + backticked), deeply nested Array(Tuple(...)). + +Approach: instead of running EXPLAIN AST against a live ClickHouse (which +needs CI infrastructure), each test asserts specific structural pieces of +the rendered SQL (`assert "PRIMARY KEY (\`tenant_id\`, \`id\`)" in sql`). +This catches every regression a TS-vs-Python rendering divergence would +cause. The clause-order assertions (e.g. DEFAULT must precede CODEC; in +REFRESH: every → offset → randomize → settings → append) lock in TS +exact-match semantics. + +Updated R-l score: **8/10 → 10/10**. + +### Combined post-fix score table + +| Section | Pre-fix | Post-fix | +|---------|--------:|---------:| +| Round-1 average | 7.3 | 9.6 | +| Round-2 average | 8.4 | **9.7** (R-l now 10) | +| Overall combined | ~7.7 | **~9.7** | + +Remaining sub-10 scores reflect *accepted divergences* (async-apply sync +model, codegen bigint default, create-chkit N/A, conftest localhost +convention) and *thin-test-only gaps* that are no longer actionable +because the functional surface is correct. + +--- + +## Main sync 2026-06-29 + +A `git pull` from main brought in 11 new commits. 8 are pure docs / +release tooling. The remaining 3 commits touch code; this section +records what was ported, what was already done, and what was decided +N/A. + +### #M1 User-Agent — versioned `chkit/` — PORTED + +- **TS:** `packages/plugin-obsessiondb/src/auth/api-client.ts` + + `client.ts` both replaced the bare `'chkit-cli'` string with + `chkit/${pkg.version}` (read via `createRequire(import.meta.url)`). + The ObsessionDB API forwards this header to ClickHouse so chkit + traffic is attributable in `system.query_log.http_user_agent`. +- **Python:** added new module + [`_version.py`](src/chkit_plugin_obsessiondb/_version.py) (single + source of truth — avoids the circular import with `__init__.py` + that re-exports `api_client` symbols). Updated + [`api_client.py`](src/chkit_plugin_obsessiondb/api_client.py) + `USER_AGENT = f"chkit/{_version.__version__}"`. jobs_api + + workbench_api inherit via `service_api._rpc_post`, which already + sets the header from the constant. +- Tests: `test_M1_*` (3 tests — constant shape + observed on + `get_session` HTTP call + observed on RPC POST). + +### #M2 Backfill remote-execution guard — PORTED (safety fix) + +- **TS:** new `guardRemoteExecution` in + `packages/plugin-obsessiondb/src/backfill/handler.ts`. When the user + runs `chkit plugin backfill {plan,run,resume}` AND is authenticated + AND has a selected service (or `--service` flag), refuse with a + message nudging them toward `--local` — because remote backfill + execution isn't implemented yet. Without the guard, the command + silently falls through to the local plugin's Phase-2 stub (or, in TS, + to a direct ClickHouse connection that bypasses ObsessionDB — + exfiltrating queries to whatever `CLICKHOUSE_URL` is set, not the + intended cloud). +- **Python:** added `_guard_remote_execution` in + [`backfill_handler.py`](src/chkit_plugin_obsessiondb/backfill_handler.py). + Branches BEFORE the existing remote-subcommand check: if the command + is in `{plan, run, resume}`, run the guard; otherwise defer to the + existing status/cancel/list routing or fall through to Unhandled. + Honors `--local` (existing early return). Emits an `{ok: false, + command, error}` envelope under `json_mode`, plain text otherwise. +- Tests: `test_M2_*` (7 tests — no-creds defer, authed-but-no-service + defer, refuses-plan, refuses-run with `--service` flag, JSON + envelope, `--local` bypass, doctor-not-guarded). + +### #M3 `whoami --json` envelope + json_envelope helper module — PORTED + +- **TS:** added `packages/plugin-obsessiondb/src/json-envelope.ts` + module exporting `whoamiEnvelope`, `serviceListEnvelope`, + `errorEnvelope`, etc. Refactored `auth/login.ts:runWhoami` to use + the typed helpers and emit `{command, schemaVersion: 1, status: + 'logged_in', email, next: null}` instead of an ad-hoc dict. +- **Python:** added new module + [`json_envelope.py`](src/chkit_plugin_obsessiondb/json_envelope.py) + mirroring the TS module — `JSON_CONTRACT_VERSION = 1`, + `whoami_envelope`, `error_envelope`, `service_list_envelope`, plus + TypedDicts for static-type guarantees. Refactored + `auth_login.run_whoami` to use the helpers. The whoami JSON shape + now matches TS exactly: `{command, schemaVersion, status: + 'logged_in', email, next: null}`. Bonus: `service_commands._service_list` + was migrated to the same helpers — the audit-fix-#9 envelope used a + non-TS-aligned `{status: 'error', errorCode, message}` shape; it now + uses `error_envelope` which produces `{ok: false, error: {code, + message}}`, matching TS. +- All three helpers re-exported from `chkit_plugin_obsessiondb` root. +- Tests: `test_M3_*` (4 tests — `error_envelope` shape, + `whoami_envelope` shape, name field intentionally not surfaced, + `service_list_envelope` shape). Updated 2 existing tests to assert + the new shapes (`test_whoami_json_mode_returns_envelope` + + `test_finding_9_service_list_json_envelope_shape`). + +### Items already in Python (no port needed) + +- `init.ts` simplified flow → Python `init.py` was already simpler + (gates only on `--yes`). +- `clearCredentials()` returns boolean → Python already returns bool. +- `runLogout` "No active session" message → already in Python. +- `onboarding.packageManager` parameter → already in Python with the + Python ecosystem managers (uvx, pipx, poetry, rye), Round-1 audit + fix #13. +- `service list --json` envelope → already in Python from Round-1 + audit fix #9; this turn migrated it to the shared helper for + consistency. +- `migrate/async-apply.ts MAX_TRANSIENT_POLL_ERRORS = 20` retry budget + → already in Python's `migrate_async_apply.py`. +- `runtime/json-output.ts` catch-all string wrap → already in Python's + `cli/json_output.py:print_output`. + +### Decided N/A (Python convention difference) + +- `cli/commands/skills.ts` (`chkit skills` proxy to `npx skills`) — + no Python ecosystem analogue (already in DRIFT > `cmd-skills`). +- `create-chkit/src/create.ts skipOnboarding` reordering — `create-chkit` + is N/A per Python convention (already in DRIFT > `create-chkit`). +- `plugin-pull/src/index.ts pluginContext.executor` handoff — Python + uses the `on_pull_introspect` hook model instead, which is the same + feature with a different shape. + +### Combined post-main-sync scores + +| Section | Score | +|---------|------:| +| Round-1 average | 9.6 | +| Round-2 average | 9.7 | +| Main-sync (#M1-M3) | 10/10 | +| **Overall combined** | **~9.8** | + +--- + +## Main sync 2026-07-02 — Category C: ClickHouse ON CLUSTER support + +TS commit ``6b87e6d`` — ~1,225 insertions across core + cli + tests. This +port covers the behavioral contract; the docker-based live 2-node cluster +e2e is deferred (see "Deferred" below). + +### #C1 `apply_on_cluster_to_plan` — PORTED + +- **TS:** new `packages/core/src/on-cluster.ts` (127 LoC). Two anchor + tables — after-object placement (`CREATE TABLE`, `ALTER TABLE`, + `DROP TABLE`, `CREATE VIEW`, `CREATE MATERIALIZED VIEW`, + `CREATE DATABASE`, `CREATE DICTIONARY`, `CREATE OR REPLACE + DICTIONARY`, `DROP VIEW`, `DROP DICTIONARY`, plus forward-compat + `CREATE FUNCTION`, `DROP DATABASE`, `ATTACH/DETACH/TRUNCATE/OPTIMIZE + TABLE`) and trailing-anchor placement (`RENAME TABLE`, `RENAME + DICTIONARY`, `RENAME DATABASE`, `EXCHANGE TABLES`, `EXCHANGE + DICTIONARIES`). Skips optional `IF [NOT] EXISTS` guard when locating + the object reference. Positional idempotency check — user COMMENTs or + view bodies containing "on cluster" cannot suppress injection. +- **Python:** new + [`src/chkit/core/on_cluster.py`](src/chkit/core/on_cluster.py). Same + two anchor tables, same regex semantics. Trailing loop runs BEFORE + per-object loop (matches TS). `_ON_CLUSTER_AT_REF` is matched against + the slice AFTER the object reference — not the whole statement. + `_ON_CLUSTER_AT_END` uses `.search()` (end-anchored only). Frozen + `MigrationPlan`/`MigrationOperation`/`ColumnRenameSuggestion` are + copied via `model_copy(update=...)` — all non-SQL fields preserved. +- Re-exported from `chkit.core` and top-level `chkit`. + +### #C2 `clickhouse.cluster` config field + validation — PORTED + +- **TS:** new optional `cluster?: string` on + `ChxUserClickHouseConfig`/`ChxResolvedClickHouseConfig`. Validator + regex `^([A-Za-z_][A-Za-z0-9_.-]*|\{[A-Za-z_][A-Za-z0-9_]*\})$` + accepts identifiers with dashes/dots (e.g. `prod-eu-1`, + `eu.west.main`) and `{cluster}`-style macros. Rejected on typos like + quotes/whitespace so interpolation into `ON CLUSTER ''` is + injection-safe. +- **Python:** added `cluster: str | None = None` to both models AFTER + `secure` for canonical serialization parity. New module-private + `_CLUSTER_NAME_PATTERN` + `_assert_valid_cluster_name` in + [`model.py`](src/chkit/core/model.py). Uses `re.fullmatch` (not + `re.match`) so a multi-line value like `"prod\nDROP TABLE x"` cannot + slip past a start-only anchor. Error message contains the literal + `Invalid clickhouse.cluster` prefix so the TS-parity test regex + matches. Falsy check in `resolve_config` — empty string or `None` + stays `None` (validator not run), matching TS. + +### #C3 Journal store cluster mode — PORTED + +- **TS:** `createJournalStore(db, cluster?)` in + `packages/cli/src/runtime/journal-store.ts`. When cluster set: + engine becomes + `ReplicatedReplacingMergeTree('/clickhouse/tables/{uuid}/chkit_journal', + '{shard}_{replica}', applied_at)` — no-`{shard}` Keeper path (one + cluster-wide replication group), `{uuid}` for drop-recreate safety, + `{shard}_{replica}` unique cluster-wide for multi-shard layouts. CREATE + TABLE + both ALTER TABLE ADD COLUMN statements carry `ON CLUSTER + ''`. Non-cluster mode unchanged. +- **Python:** [`JournalStore.__init__`](src/chkit/cli/journal_store.py) + now takes optional `cluster: str | None`. `_on_cluster` cached via + `on_cluster_clause(cluster)`. `_create_table_sql` injects the clause + and switches engine. `_ensure_schema_upgraded` stamps both ALTERs. +- Threaded through 3 call sites (Python has one fewer than TS — + `generate` does not touch the journal here): `migrate.py`, + `status.py`, `check.py` each pass + `cluster=config.clickhouse.cluster if config.clickhouse else None`. + +### #C4 `generate` command integration — PORTED + +- **TS:** `packages/cli/src/commands/generate/command.ts` calls + `applyOnClusterToPlan(plan, config.clickhouse?.cluster)` as the final + plan transform, AFTER renames + plugin `on_plan_created` + scope + filtering, so plugin-injected SQL is also covered. +- **Python:** [`generate.py`](src/chkit/cli/commands/generate.py) + calls `apply_on_cluster_to_plan(plan, ...)` immediately after + `plugin_runtime.run_on_plan_created(...)`, before the empty-plan + check. `migrate` does NOT re-run this — the clause is baked into the + migration file at generate time and re-executed verbatim. + +### Tests added + +- [`tests/test_on_cluster.py`](tests/test_on_cluster.py) — 19 tests, + case-by-case parity with TS `on-cluster.test.ts` (10 injector cases + + 3 clause cases + 5 config-validation cases + 1 multi-line regression + guard for the `re.fullmatch` vs `re.match` fix). +- [`tests/test_journal_store_cluster.py`](tests/test_journal_store_cluster.py) + — 5 tests. CREATE TABLE + both ALTER TABLE stamping in cluster mode, + plain-engine fallback when cluster absent, `{cluster}` macro form. +- [`tests/test_on_cluster_generate_e2e.py`](tests/test_on_cluster_generate_e2e.py) + — 3 tests. Runs the actual `chkit generate` CLI (JSON dryrun and file + write) with `clickhouse.cluster` set/unset; asserts every DDL line + in the emitted migration carries or omits the clause accordingly. + +### Deferred (out of scope for this port) + +- **Live cluster e2e** — TS ships + `packages/cli/test/cluster.e2e.test.ts` + + `cluster-2shard.e2e.test.ts` against docker fixtures under + `test/cluster/`. Behavior parity is covered by the unit + integration + tests above; the docker fixtures live in the TS repo and are not + duplicated here. When live cluster testing infrastructure lands on + the Python side, port those e2es as-is. + +### Design divergences (justified) + +- **Test dictionary op types:** TS `on-cluster.test.ts` uses + `create_dictionary`/`drop_dictionary`/`rename_dictionary` operation + types. Python `MigrationOperationType` does not yet include the + Dictionary primitive (TS `65c90d6`, tracked as Category B port). The + injector only inspects `sql`, not `type`, so the Python tests use + `create_table`/`drop_table`/`alter_table_rename_table` for the + dictionary SQL — behavioral coverage is unchanged and the tests can be + retyped verbatim once Dictionary lands. +- **Debug log:** TS emits a `debug('journal', ...)` line noting the + `CHKIT_JOURNAL_TABLE` env override and cluster suffix. Python + `journal_store.py` has no debug logging surface at all (pre-existing); + cluster mode adds no new log emission. + +### Follow-up observed during Category C (not fixed here) + +- **`generate.py` plan-transform ordering divergence** — TS order is + `run_on_plan_created` → `filter_plan_by_table_scope` → + `apply_on_cluster_to_plan`. Python order is + `filter_plan_by_table_scope` → `run_on_plan_created` → + `apply_on_cluster_to_plan`. Predates this port (established in commit + `ad94a16`); noticed by the Category C reviewer. Consequence: in + Python, plugin-added SQL touching out-of-scope tables is not filtered + out before ON CLUSTER stamping. Out of scope for this port; track as + a separate parity fix. + +### Not ported this pass (out of Category C scope) + +Other main-branch commits since 2026-06-29 that this session did NOT +touch — tracked for the next port cycle: + +- `65c90d6` — Dictionary primitive (Category B, ~500 LoC). +- `c1d8d0d` — `chkit backfill submit` (Category D, obsessiondb-scoped). +- `8296b8a`, `3f1db03`, `5a8d805` — Category A bug-fix trio + (pull/drift table clauses, index-only projections, function + expressions in PK/order-by). +- `f85f568`, `3f9a246`, `9ad23f9` — Phase-2 backfill engine changes + (deferred with the rest of Phase 2). +- `3c008f4`, `9d9c06e`, `b501f5d` — TS-only refactors with Python + parity already in place. + +--- + +## Main sync 2026-08-10 — full-parity pass: Category A trio + Dictionary + submit surface + +Policy note first: per the user's standing instruction, sync passes no +longer defer upstream commits by category — every pending commit is +ported in the pass, and anything that genuinely cannot land (below: +Phase-2 backfill engine) is called out explicitly rather than silently +queued. Each ported commit was independently reviewed by a +TS-vs-Python parity reviewer; accepted findings are folded in below. + +### `5a8d805` — function expressions in primaryKey/orderBy (#178) + +- `key_clause.is_plain_column_reference` (fullmatch ≡ TS `^...$`). +- `planner._strip_insignificant_formatting` + `_join_clause` — strips + whitespace and identifier backticks (not quoted literals) for + key-clause comparison only; applied to primary_key/order_by/unique_key, + never engine/partition_by. +- `sql._render_key_clause_columns(columns, column_names)` — quotes + declared columns and bare identifiers; expressions verbatim. +- `validate` — key checks skip expressions. +- **Reviewer fix folded in:** whitespace stripping uses an explicit + JS-`\s` character class (`_JS_WHITESPACE_RE`) instead of + `str.isspace()`, which diverges on U+0085/U+001C–U+001F/U+FEFF. +- Tests: `tests/test_key_clause_expressions.py` (all 5 TS cases). + +### `3f1db03` — index-only projections (#193) + +- New `core/projection.py` (`is_index_projection`, + `normalize_projection_index`, `canonicalize_projection`, + `render_projection_body`) — exact port including paren peeling, + idempotency, and ClickHouse comma-spacing echo. +- **Design divergence (deliberate):** `ProjectionDefinition` stays ONE + Pydantic model with optional `query`/`index`/`type` instead of the TS + two-member union. Split enforcement mirrors where TS enforces: + states TS's type system cannot represent (neither kind; `index` + without `type`) are rejected at model construction via a + `model_validator`; the both-set case (TS admits structurally) surfaces + as `projection_ambiguous_kind`, empty index as + `projection_empty_index`. Both reviewer-found render holes + (`(None)` body, trailing `TYPE `) are closed by the validator. +- **Snapshot interop fix (reviewer):** `write_snapshot` now dumps with + `exclude_none=True`. TS `JSON.stringify` omits undefined keys and + `isIndexProjection` is `'index' in projection`, so a Python-written + `"index": null` would flip every SELECT projection to index-only when + the TS tool reads the shared snapshot. Reading remains tolerant of + both styles. +- Parser (`_INDEX_PROJECTION_RE`), drift shape fingerprints + (`index=...|type=...`), pull render, introspection all ported. +- Tests: `tests/test_index_only_projections.py` (all TS core cases + + reviewer-requested backtick-name, multiline SHOW CREATE, construction + guards, pull-render case) and 3 drift tests in + `tests/test_drift_compare.py`. + +### `8296b8a` — table clauses past the column list + derived PK (#198) + +- `create_table_parser`: `_find_column_list_bounds` / + `_extract_table_options` refactor; every clause parser (including + SETTINGS/TTL) now searches only past the column list, so a + projection's inner `ORDER BY` or a column-level `TTL` can't swallow + table-level clauses. Falls back to whole-query when unparseable. +- `drift_compare`: PRIMARY KEY derived from ORDER BY on both sides + when absent. **Reviewer fix folded in:** the actual-side fallback + uses `is not None` (≡ TS `??`), not truthiness — an empty-string + primary key compares as-is. +- Tests: `tests/test_parser_clause_fixes.py` (all 4 TS cases, exact + dataclass equality per reviewer). + +### `65c90d6` — Dictionary primitive (#191) + +Full lifecycle port mirroring `materialized_view`: + +- Model: `DictionaryAttribute` / `DictionaryRange` / + `DictionaryDefinition` (+ `dictionary()` factory, `SchemaKind`, + operation types `create_dictionary`/`drop_dictionary`/ + `rename_dictionary`, 8 validation codes). +- Canonicalization (sort kind 3), validation, `render_dictionary_sql` + (`CREATE DICTIONARY IF NOT EXISTS` / `CREATE OR REPLACE DICTIONARY`, + PRIMARY KEY without parens, SOURCE/LAYOUT/LIFETIME/RANGE/SETTINGS/ + COMMENT), planner diff (whole-shape compare; `[HIDDEN]` source is + excluded from the diff so the introspection placeholder never + deploys, but does not suppress unrelated changes). +- `clickhouse/create_dictionary_parser.py` — new 1:1 parser port + (attribute modifiers, composite PK, RANGE vs the `range()` array + function, SETTINGS, quoted-value consumption). +- DDL propagation: `wait_for_dictionary` + `dictionary:` operation + keys; `infer_schema_kind_from_engine('Dictionary') == 'dictionary'`. +- CLI: `--rename-dictionary` on generate (parse/merge/resolve/assert/ + remap + `RENAME DICTIONARY` plan ops ranked with table renames), + plain-text password warnings on generate output, safety markers + (`drop_dictionary_dependency_break`, DICTIONARY in the object-key + regex), drift object existence, pull introspection + rendering + (`dictionary(...)` with `[HIDDEN]` note + password warnings), + codegen Pydantic models from dictionary attributes. +- ON CLUSTER anchors were already in place from the Category C port. +- Tests: `tests/test_dictionary.py` (29 tests — core DSL/validation/ + planner, parser, safety markers, drift). + +### `c1d8d0d` — `chkit backfill submit` (surface only) + +- `SubmitOptions` + `SUBMIT_FLAGS`/`SUBMIT_FLAG_MAP` in + `chkit_plugin_backfill.options`; `submit` command registered on the + plugin with the TS local-handler behavior (clear "requires a managed + job backend" error when no ObsessionDB service intercepts). +- `chkit_plugin_obsessiondb.backfill_handler` guards `submit` alongside + plan/run/resume when authed + service selected. +- **Not ported (explicit):** the managed-job submit path itself + (`buildSubmitTasks` → jobs backend) depends on `buildBackfillPlan` + + `buildChunkExecutionSql` from the Phase-2 chunking engine. + +### Dictionary reviewer triage (folded in post-port) + +- **JS `Number()` coercion parity** in + `create_dictionary_parser._consume_quoted_or_bare_value` — Python's + `int()`/`float()` accepted `1_000`/`nan`/`inf` (JS keeps them strings) + and re-rendered `300.0`/`1e5` with a trailing `.0`, a permanent + spurious-diff surface against TS-written snapshots. Now coerces only + JS-grammar decimals (integral floats collapse to int) and `0x` hex. +- `_parse_qualified_table` error message updated to the TS + `parseQualifiedName` wording (shared by table + dictionary renames); + the stale-message test expectation updated with it. +- Primary-key backtick strip narrowed to one backtick per side + (TS `.replace(/^`|`$/g, '')`). +- Test debt closed at pipeline level: dictionary rename pipeline + (RENAME not drop+create, schema `renamed_from`), generate + pull + password warnings, pull dictionary rendering (incl. `[HIDDEN]` note), + codegen dictionary model, JS-number coercion matrix — all in + `tests/test_dictionary.py` (38 tests). +- Reviewer items accepted as-is (no code change): planner comparison + via sorted `model_dump` is order-insensitive where TS + `JSON.stringify` is key-order-sensitive — Python is strictly more + lenient, never the reverse; codepoint vs locale settings sort; + `wait_for_dictionary` quoting is safer than TS. +- Still-open e2e debt (needs live ClickHouse, consistent with prior + passes deferring docker-based e2e): `migrate-dictionary.e2e`, + `generate.e2e` dictionary flows, pull e2e, `sql-validation.e2e` + dictionary block. + +### Per-table `plugins` field (baseline gap closed in the same pass) + +- `TableDefinition.plugins: dict[str, Any] | None` + `table(plugins=...)` + kwarg — TS `TablePlugins` parity. Metadata only: ignored by the diff + engine (test in `tests/test_planner.py`), carried through + canonicalization and snapshots. Without it, a schema file ported from + TS carrying `plugins: {...}` failed Pydantic validation outright. + Consumers (backfill `timeColumn`) arrive with Phase 2. + +### Parity fixes surfaced by the docs dual-language pass (same day) + +- **Wheel packaging** — `[tool.hatch.build.targets.wheel]` only shipped + `chkit` + `chkit_plugin_obsessiondb`; a pip-installed `chkit-py` was + missing the codegen and backfill plugins entirely (the `chkit init` + template even suggests importing `chkit_plugin_codegen`). All four + packages now ship in the wheel. +- **Function-style configs (TS `ChxConfigFn`)** — the Python loader only + accepted a static `config` attribute; TS allows + `defineConfig((env) => ({...}))` with `env: {command, mode}` (documented + in the CI/CD guide). Ported: `ChxConfigEnv` model, `define_config` + accepts a callable, `load_config(path, env)` invokes it, and every CLI + command threads `ChxConfigEnv(command=...)`. Test in + `tests/test_user_config_and_config_merge.py`. + +### CLI-surface parity ports (surfaced by the docs-accuracy reviewer's live CLI probes) + +- **Plugin dispatcher flag forwarding** — `chkit plugin ` + hardcoded `flags={}`, so no plugin command could receive its own + `--flags` (Typer rejected the tokens outright). Now the `plugin` + command registers with `ignore_unknown_options`, and the dispatcher + parses forwarded tokens against the plugin command's declared flag + defs via `core.flags.parse_flags` (unknown flags error, TS-style); + remaining tokens stay positional. +- **Top-level `chkit codegen`** — existed only in TS. New + `codegen_shortcut.py` dispatches `plugin codegen codegen` with + forwarded flags (`--check`, `--out-file`, ...). Verified live: + `chkit codegen` writes, `chkit codegen --check` gates. +- **Top-level `chkit obsessiondb `** — the Python spelling was + `chkit plugin obsessiondb ` while the plugin's own runbook + strings printed the TS spelling. New `obsessiondb_shortcut.py` makes + those strings correct; multi-word commands (`service select`) flow + through as positionals. +- **Dispatcher DB connection made opportunistic** — an unreachable + ClickHouse no longer blocks plugin commands that don't need an + executor (codegen, backfill status); mirrors TS. +- **`check.failOnExtraObjects`** — TS config flag was missing from + `ChxCheckConfig`; the drift machinery already had the parameter but + both call sites hardcoded `False`. Field added (default false), + resolved, and wired through `check`/`drift`. +- **`backfill({maxChunkBytes: "10G"})`** — the factory only accepted + ints; suffix strings were CLI-flag-only. `field_validator` now + coerces via `parse_byte_size`, matching the documented + `string | number` contract. + +### Remaining gap after this pass (explicit, needs its own project) + +- ~~**Phase-2 backfill execution engine**~~ — **CLOSED** by the + 2026-08-10 Phase-2 port entry below. No remaining gap: the full + chunking + execution engine, the three upstream fixes (`f85f568`, + `3f9a246`, `9ad23f9` — ported as end-state of `main`), and the + managed-submit backend half of `c1d8d0d` are all in. Everything is at + parity as of upstream `main`. + +### Environment note + +- `.venv` was found hollow and rebuilt from `python 3.11` + + `pip install -e .[dev]`; latest ruff surfaced new lint rules + (PLR0917) on pre-existing code — annotated rather than refactored. +- `tests/test_e2e_testkit.py::test_create_run_tag_is_unique_across_calls` + is flaky (millisecond-timestamp + random-suffix collision over 100 + iterations); observed one collision, passes on re-run. Pre-existing. + + +## Main sync 2026-08-10 — Phase-2 backfill engine port (final parity gap closed) + +Full port of the chunking + execution engine and the managed-submit +backend — the last TS surface with no Python counterpart. Ported as the +end-state of `main` (includes `f85f568` MV-source chunk sizing, +`3f9a246` multi-MV UNION ALL replay, `9ad23f9` shared SQL scanner). + +### Module map (TS → Python) + +- `chunking/{types,boundary-codec,partition-slices,sql}.ts`, + `chunking/utils/{binary-string,ranges,ids}.ts`, + `chunking/services/{metadata,row-probe,distribution}-source.ts`, + `chunking/strategies/*` (7 files), `chunking/planner.ts`, + `chunking/{analyze,strategy-policy}.ts` + → `chkit_plugin_backfill/chunking/` (same layout, snake_case). +- `detect.ts` → `detect.py`; `planner.ts` → `planner.py`; + `queries.ts` → `queries.py`; `async-backfill.ts` → `async_backfill.py`; + `payload.ts` → `payload.py`; `check.ts` → `check.py`; + `logging.ts` → `logging_utils.py` (stdlib logging + threading.Timer + slow-query warnings); `sdk.ts` → `sdk.py`. +- `plugin.ts` → real `plan`/`run`/`resume`/`status`/`cancel`/`doctor` + handlers in `plugin.py` (Phase-2 stubs removed), `on_check` + + `on_check_report` hooks, TS `wrapPluginRun` error envelope + (json `{ok,command,error}` / text `"