Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .changeset/drizzle-encrypted-indexes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
'@cipherstash/stack-drizzle': minor
---

New `encryptedIndexes` helper on the `/v3` entry: spread
`...encryptedIndexes(t)` in `pgTable`'s third-argument callback and it derives
the recommended functional indexes for every encrypted column in the table —
named `<table>_<column>_<capability>`, tracked by `drizzle-kit generate` like
any other index. The mapping comes from the same per-domain capability record
the operator layer gates on, so the emitted indexes and the operators that
engage them cannot drift: equality → btree on `eql_v3.eq_term`, ordering →
btree on `eql_v3.ord_term` (on the numeric/date/timestamp `_ord` domains one
index serves `=` and range — their injective ordering term answers equality
and no `eq_term` overload exists; the non-injective `text_ord` / `text_ord_ore`
also carry `hm` and get an `eq_term` index alongside), ORE ordering →
`eql_v3.ord_term_ore`, free-text →
GIN on `eql_v3.match_term`, encrypted JSON → GIN on
`(eql_v3.to_ste_vec_query(col)::jsonb) jsonb_path_ops`. Storage-only and
non-encrypted columns emit nothing. Closes the #753 gap where integrations
emitted query operators but no index DDL, so encrypted predicates
sequential-scanned by default.

Also fixed: `isEqlV3Column` / `getEqlV3Column` no longer blow the stack when
handed a column from `pgTable`'s extras callback — drizzle-orm ≤0.45's
`ExtraConfigColumn.getSQLType()` recurses into itself, so the domain is now
recovered from the column's custom-type params instead of calling it.
23 changes: 23 additions & 0 deletions .changeset/stash-indexing-skill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
'stash': minor
'@cipherstash/wizard': minor
---

New bundled agent skill: `stash-indexing` — how to index EQL v3 encrypted
columns. Integrations that were otherwise correct shipped with no index on any
encrypted predicate because nothing in the installed skills said encrypted
columns *can* be indexed (#753). The skill covers the functional-index recipes
over the term extractors (`eql_v3.eq_term` / `ord_term` / `ord_term_ore` /
`match_term` / `to_ste_vec_query`) mapped to the `types.*` domains, what works
without superuser on Supabase and managed Postgres versus the ORE opclass
restriction, which domains are storage-only by design, the query shapes that
engage an index (`ORDER BY` sort-key and `GROUP BY` traps), building indexes on
large tables, an `EXPLAIN` verification checklist, and when to create indexes
during an encryption rollout (after backfill, before switching reads).

`stash init` / `stash impl` handoffs — and the `@cipherstash/wizard` skills
prompt — now install it for **every** integration (Drizzle, Supabase, Prisma
Next, plain PostgreSQL) — the gap is cross-cutting.
The existing per-integration skills gained pointers to it (including the
missing `stash-prisma-next` one-line purpose in the setup prompt, which
previously rendered "(no description)").
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ If these variables are missing, tests that require live encryption will fail or
- `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README)
- `examples/*`: Working apps (basic, prisma, supabase-worker)
- `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo).
- `skills/*`: Agent skills (`stash-cli`, `stash-encryption`, `stash-drizzle`, `stash-dynamodb`, `stash-supabase`, `stash-prisma-next`, `stash-supply-chain-security`)
- `skills/*`: Agent skills (`stash-cli`, `stash-encryption`, `stash-indexing`, `stash-drizzle`, `stash-dynamodb`, `stash-supabase`, `stash-prisma-next`, `stash-supply-chain-security`)

## Agent Skills — these ship to customers

Expand All @@ -119,6 +119,7 @@ nothing type-checks them, and the damage lands in a customer's repo, not ours.
| `packages/stack` encryption API, schema builders, subpath exports | `skills/stash-encryption` |
| Drizzle / Supabase / Prisma Next / DynamoDB integrations | `skills/stash-drizzle`, `skills/stash-supabase`, `skills/stash-prisma-next`, `skills/stash-dynamodb` |
| The rollout/cutover lifecycle (`packages/migrate`, `stash encrypt *`) | `skills/stash-encryption` and `skills/stash-cli` |
| The `@cipherstash/eql` pin, `eql install`/`eql migration` behaviour, or index-related SQL guidance | `skills/stash-indexing` |
| pnpm config, CI workflows, dependency policy | `skills/stash-supply-chain-security` |
| The durable agent rules themselves | `packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md` |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,4 @@ The CipherStash setup skills carry the API details. Use them when you need speci
- `.codex/skills/<skill>/SKILL.md` (Codex)
- Under `## Skill references` at the bottom of this file (editor agents, or when a skills directory could not be written — if the section exists, it is the current copy)

Skills relevant to this project depend on the integration. Common ones: `stash-encryption` (encryption API), `stash-cli` (`stash` commands), and one of `stash-drizzle` / `stash-supabase` / `stash-dynamodb` for the chosen ORM.
Skills relevant to this project depend on the integration. Common ones: `stash-encryption` (encryption API), `stash-indexing` (indexes on encrypted columns), `stash-cli` (`stash` commands), and one of `stash-drizzle` / `stash-supabase` / `stash-prisma-next` / `stash-dynamodb` for the chosen ORM.
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ describe('buildAgentsMdBody', () => {
expect(out).toContain('# CipherStash')
expect(out).toContain('# Skill: stash-encryption')
expect(out).toContain('# Skill: stash-drizzle')
expect(out).toContain('# Skill: stash-indexing')
expect(out).toContain('# Skill: stash-cli')
// Frontmatter from individual skill files should be stripped — the
// `name: <skill>` line is part of YAML frontmatter and should not leak.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,16 @@ describe('SKILL_MAP', () => {
}
})

// #753: integrations shipped without a single CREATE INDEX example, so
// agent-built integrations left every encrypted predicate unindexed. The
// indexing skill is cross-cutting (Drizzle, Prisma Next, Supabase, and raw
// SQL all need it), so every integration must install it.
it('always includes stash-indexing for every integration', () => {
for (const [integration, skills] of Object.entries(SKILL_MAP)) {
expect(skills, integration).toContain('stash-indexing')
}
})

it('drizzle includes stash-drizzle', () => {
expect(SKILL_MAP.drizzle).toContain('stash-drizzle')
})
Expand All @@ -79,14 +89,15 @@ describe('skillsFor', () => {
expect(skillsFor('prisma-next')).toEqual([
'stash-encryption',
'stash-prisma-next',
'stash-indexing',
'stash-cli',
])
})

it('falls back to the base skills for an unmapped integration (never crashes)', () => {
// Simulate a future Integration variant with no SKILL_MAP entry.
const skills = skillsFor('mystery-orm' as Integration)
expect(skills).toEqual(['stash-encryption', 'stash-cli'])
expect(skills).toEqual(['stash-encryption', 'stash-indexing', 'stash-cli'])
})
})

Expand All @@ -104,7 +115,12 @@ describe('installSkills', () => {

it('copies the per-integration skills into destDir', () => {
const { copied, failed } = installSkills(tmp, '.claude/skills', 'drizzle')
expect(copied).toEqual(['stash-encryption', 'stash-drizzle', 'stash-cli'])
expect(copied).toEqual([
'stash-encryption',
'stash-drizzle',
'stash-indexing',
'stash-cli',
])
expect(failed).toEqual([])
for (const name of copied) {
expect(
Expand Down Expand Up @@ -141,7 +157,12 @@ describe('installSkills', () => {
}).not.toThrow()
expect(result).toEqual({
copied: [],
failed: ['stash-encryption', 'stash-drizzle', 'stash-cli'],
failed: [
'stash-encryption',
'stash-drizzle',
'stash-indexing',
'stash-cli',
],
})
expect(warnings()).toContain('Could not create .codex/skills/')
})
Expand All @@ -162,7 +183,7 @@ describe('installSkills', () => {
)

const { copied, failed } = installSkills(tmp, '.codex/skills', 'drizzle')
expect(copied).toEqual(['stash-encryption', 'stash-cli'])
expect(copied).toEqual(['stash-encryption', 'stash-indexing', 'stash-cli'])
expect(failed).toEqual(['stash-drizzle'])
expect(warnings()).toContain('Failed to install skill stash-drizzle')
})
Expand All @@ -177,7 +198,12 @@ describe('installSkills', () => {
const { copied, failed } = installSkills(tmp, '.codex/skills', 'drizzle')
expect(copied).toEqual([])
expect(failed).toEqual(availableSkills('drizzle'))
expect(failed).toEqual(['stash-encryption', 'stash-drizzle', 'stash-cli'])
expect(failed).toEqual([
'stash-encryption',
'stash-drizzle',
'stash-indexing',
'stash-cli',
])
})

it('is idempotent — re-running does not throw and yields the same result', () => {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { describe, expect, it } from 'vitest'
import { SKILL_MAP } from '../install-skills.js'
import { renderSetupPrompt, type SetupPromptContext } from '../setup-prompt.js'

const baseCtx: SetupPromptContext = {
Expand All @@ -12,7 +13,12 @@ const baseCtx: SetupPromptContext = {
handoff: 'claude-code',
mode: 'implement',
skills: {
installed: ['stash-encryption', 'stash-drizzle', 'stash-cli'],
installed: [
'stash-encryption',
'stash-drizzle',
'stash-indexing',
'stash-cli',
],
inlined: [],
failed: [],
},
Expand Down Expand Up @@ -134,13 +140,30 @@ describe('renderSetupPrompt — orient + route (implement mode)', () => {
const out = renderSetupPrompt(baseCtx)
expect(out).toContain('`stash-encryption`')
expect(out).toContain('`stash-drizzle`')
expect(out).toContain('`stash-indexing`')
expect(out).toContain('`stash-cli`')
// Each skill line should explain what the skill is for, not just name it.
// A skill missing from SKILL_PURPOSES silently renders with no purpose
// line, so every skill SKILL_MAP can install must assert one here.
expect(out).toMatch(/`stash-encryption`.*lifecycle/i)
expect(out).toMatch(/`stash-drizzle`.*Drizzle/i)
expect(out).toMatch(/`stash-indexing`.*index recipes/i)
expect(out).toMatch(/`stash-cli`.*command reference/i)
})

it('has a purpose line for every skill SKILL_MAP can install', () => {
// renderSkillIndex falls back to "(no description)" for a skill missing
// from SKILL_PURPOSES — a silent gap no per-skill assertion catches when
// a new skill lands (stash-prisma-next shipped that way). Render the
// union of every installable skill and require zero fallbacks.
const everyInstallable = [...new Set(Object.values(SKILL_MAP).flat())]
const out = renderSetupPrompt({
...baseCtx,
skills: { installed: everyInstallable, inlined: [], failed: [] },
})
expect(out).not.toContain('(no description)')
})

it('points each handoff at the right rule location', () => {
const claude = renderSetupPrompt({ ...baseCtx, handoff: 'claude-code' })
const codex = renderSetupPrompt({ ...baseCtx, handoff: 'codex' })
Expand Down
24 changes: 19 additions & 5 deletions packages/cli/src/commands/init/lib/install-skills.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,28 @@ import { findBundledDir } from './bundled-paths.js'
* `dist/skills/` at build time.
*/
export const SKILL_MAP: Record<Integration, readonly string[]> = {
drizzle: ['stash-encryption', 'stash-drizzle', 'stash-cli'],
supabase: ['stash-encryption', 'stash-supabase', 'stash-cli'],
'prisma-next': ['stash-encryption', 'stash-prisma-next', 'stash-cli'],
postgresql: ['stash-encryption', 'stash-cli'],
drizzle: ['stash-encryption', 'stash-drizzle', 'stash-indexing', 'stash-cli'],
supabase: [
'stash-encryption',
'stash-supabase',
'stash-indexing',
'stash-cli',
],
'prisma-next': [
'stash-encryption',
'stash-prisma-next',
'stash-indexing',
'stash-cli',
],
postgresql: ['stash-encryption', 'stash-indexing', 'stash-cli'],
}

/** The skills every integration gets — the safe fallback for an unmapped one. */
const BASE_SKILLS: readonly string[] = ['stash-encryption', 'stash-cli']
const BASE_SKILLS: readonly string[] = [
'stash-encryption',
'stash-indexing',
'stash-cli',
]

/**
* Skills for an integration, resilient to an unmapped one. `SKILL_MAP` is
Expand Down
4 changes: 4 additions & 0 deletions packages/cli/src/commands/init/lib/setup-prompt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,10 @@ const SKILL_PURPOSES: Record<string, string> = {
'Drizzle-specific patterns: declaring encrypted columns, query operators, the rollout/cutover walkthrough for an existing column',
'stash-supabase':
'Supabase-specific patterns: `encryptedSupabase` wrapper, encrypted query filters, transparent decryption, the rollout/cutover walkthrough',
'stash-prisma-next':
'Prisma Next-specific patterns: `cipherstash.*` field constructors, migration flow, encrypted query operators',
'stash-indexing':
'index recipes for encrypted columns — the `eql_v3` extractor functional indexes, Supabase/managed-Postgres constraints, EXPLAIN verification',
'stash-dynamodb':
'DynamoDB encryption: per-item encrypt/decrypt, HMAC attribute keys, audit logging',
'stash-cli':
Expand Down
31 changes: 31 additions & 0 deletions packages/stack-drizzle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,37 @@ comparisons and ordering at a scalar JSONPath leaf. For example,
`.orderBy(await ops.selector(users.profile, '$.age').asc())` lowers to
`ORDER BY eql_v3.ord_term(...)` over the selected encrypted entry.

### Indexing encrypted columns

Encrypted predicates only use an index if one exists over the matching
`eql_v3.*` term-extractor expression — otherwise every encrypted query
sequential-scans. `encryptedIndexes` derives the recommended indexes for
every encrypted column in a table; spread it into `pgTable`'s third-argument
callback and `drizzle-kit generate` picks the indexes up like any others:

```ts
import { integer, pgTable } from 'drizzle-orm/pg-core'
import { encryptedIndexes, types } from '@cipherstash/stack-drizzle/v3'

export const users = pgTable(
'users',
{
id: integer('id').primaryKey(),
email: types.TextEq('email'),
bio: types.TextSearch('bio'),
},
(t) => [...encryptedIndexes(t)],
)
```

Each column gets indexes matching its domain's capabilities, named
`<table>_<column>_<capability>` (equality btree, ordering btree, free-text
GIN, JSON containment GIN); storage-only and non-encrypted columns get none.
After the migration applies, run `ANALYZE <table>` — expression indexes have
no statistics until then. For custom names, subsets, or field-level selector
indexes on encrypted JSON, declare individual expression indexes instead;
the bundled `stash-indexing` agent skill has the full recipes.

## EQL v2 (package root) — legacy

The v2 integration predates the typed v3 domains and is kept for existing
Expand Down
1 change: 1 addition & 0 deletions packages/stack-drizzle/__tests__/v3/exports.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ const PUBLIC_SURFACE = [
'EncryptionOperatorError',
'EqlV3CodecError',
'createEncryptionOperatorsV3',
'encryptedIndexes',
'extractEncryptionSchemaV3',
'getEqlV3Column',
'isEqlV3Column',
Expand Down
Loading
Loading