Skip to content

feat(discord): encrypt the webhook URL at rest, keyed by (integrationId, kind) (TASK-124 part 2) - #1898

Merged
lilyshen0722 merged 3 commits into
mainfrom
kai/124-webhookurl-encryption
Sep 25, 2026
Merged

lilyshen0722 merged 3 commits into
mainfrom
kai/124-webhookurl-encryption

Conversation

@lilyshen0722

@lilyshen0722 lilyshen0722 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Encrypts discord_integrations.webhookUrl at rest and fixes the three blockers that made the encryption safe to switch on. Wren's ruling: option A, keyed with kind (74127). Gate: @vera.

The shape

connector_secrets is the envelope that already held Slack's bot token. A secret's identity becomes the pair (integrationId, kind); writers store only a ref (Integration.config.webhookUrlRef), readers resolve through one new module.

  • services/connectorSecretKinds.ts — the single definition: kind, provider, the row paths that hold that kind's ref, and the sentence written for the person reading the Connectors page. Four sites encode the pair (the index, the upsert filter, the 11000 retry's filter, the orphan sweep's keep-condition) and all four read this instead of literals.
  • utils/discordWebhookUrl.ts — the one precedence definition: encrypted ref wins, the two legacy plaintext stores (platformIntegration.webhookUrl, Integration.config.webhookUrl) are fallbacks. A ref that cannot be resolved throws rather than falling through to a plaintext copy: a wrong webhook URL does not fail authentication, it authenticates as a different connector (vera 74141 — a missing secret is loud, a substituted one is not).
  • Writers — the connect path (routes/integrations.ts) and the lazy backfill (discordService.connect) both put the URL and $set the ref; only webhookId, which is not a secret, stays on the platform document.
  • Strip list — webhookUrl and webhookUrlRef join SERVER_OWNED_CONFIG_KEYS (the URL is derived server-side on both writers; a planted ref would aim a row at another row's secret, and a planted URL would be picked up as the resolver's legacy fallback). webhookUrlRef also joins INTEGRATION_SECRET_CONFIG_KEYS so the pointer never rides out in a response.

The three blockers

# Defect Fix Witness
1 sweepOrphanedConnectorSecrets kept a secret only on a Slack botTokenRef pair → revoked a Discord secret 10 min after put keep-condition drawn from the registry, unrecognized kinds KEPT and logged (fail-closed) "keeps a Discord webhook secret past the grace window while its row still points at it"
2 sweepUnavailableSlackSecretKeys marked any row "Slack connector secret key is unavailable" — user-facing since #1878/#1881 kind-aware reason; Slack's sentence unchanged "names the Discord webhook, not a Slack key, when a Discord secret is unreadable"
3 upsertEncrypted filtered { integrationId } on both the upsert and the retry → second kind silently overwrote the first, first ref decrypting the second secret kind in the filter; one index on the pair "keys a secret by the PAIR…" + "declares exactly one index, on the pair"

Migration — separate gated step

backend/scripts/encrypt-discord-webhook-url (same shape as #1891): dry-run by default, --apply writes, digests instead of URLs, reads both plaintext stores (discord_integrations.webhookUrl and the legacy Integration.config.webhookUrl), and clears a row only after its own ref is stored — per row, put first, then both $unsets, so a failure of any kind leaves that row whole in both stores and a re-run reports it as still waiting. Corrected after vera's hold: the first draft also called listWithUnavailableKey() as a pre-flight and claimed it failed fast. That call does parse the key ring before it queries, so it would throw on an unusable ring — but so does every put, ahead of either $unset, so the pre-flight was redundant rather than protective and was removed rather than witnessed; the two suite witnesses below hold the ordering instead. (vera's correction at 74198: the first version of this note said the call “cannot fail”, which was false — listWithUnavailableKey's first statement is parseKeyRing().)

kubectl exec -n commonly-dev deploy/backend -- node -e "require('ts-node/register'); require('./scripts/encrypt-discord-webhook-url')"   # dry run first

Run it after this deploys: the current image reads the plaintext field directly.

Two things found while building, both disclosed

  1. config on Integration is a strict subdocument, so an undeclared path in $set is dropped in silence. The ref write was a no-op until webhookUrlRef: String was declared beside webhookUrl. The witness that caught it is the created-row assertion in integrations.discordTokenCopy.test.js (it read undefined while every mocked-model suite stayed green).
  2. The Discord connect path now depends on the connector-secret ring, exactly as the Slack bind already does. Without it a connect 500s instead of storing plaintext — fail-closed, and the ring is part of the deployment. Three route suites needed the ring in their setup; that is a real deployment dependency, not a test detail.

Verification

  • full backend tier: 446 suites / 4167 tests passed, 0 failed (24 skipped by their own guards);
  • tsc -p tsconfig.build.json --noEmit exit 0; eslint . --ext .ts 0 errors at the 1210-warning baseline;
  • mutation ledger, no survivors — 8 arms on the code: M1 filter loses kind (2 red), M2 a second unique index (2 red), M3 Slack-only keep-condition (1 red), M4 Slack-only reason (1 red), M5 fall back instead of throw (2 red), M6 connect writes plaintext again (1 red), M7 the strip entries removed (1 red), M8 the schema declaration removed (1 red). On the migration script: vera's mutation — both $unsets above the put — reddens 3 named tests (both ordering witnesses + the empty-ring one), removing the dry-run guard reddens 1, and re-adding the deleted pre-flight is a survivor by design, and the reason it survives is that it is redundant — both it and put reach the same parseKeyRing() before any write, so no test can distinguish them;
  • full backend tier re-run: 446 suites, the only two failures were the known oauthController total-count flake (TASK-134) and admin.analytics.human-filter, both green on re-run (24 passed).

Not changed, deliberately: installable/projectors/webhookProjector.ts still revokes only Slack refs on uninstall — the orphan sweep now covers a leftover Discord ref within the grace window, so a second revoke path would be duplicate machinery rather than a fix.

…Id, kind) (TASK-124 part 2)

`discord_integrations.webhookUrl` was the last connector credential stored in
plaintext outside the ESO path. The URL embeds the webhook's own token, so
holding it is posting AS that channel. Writers now put it in the
connector-secret envelope (AES-256-GCM, ring from CONNECTOR_SECRET_KEYS) and the
row keeps only an opaque ref at `Integration.config.webhookUrlRef`; readers go
through `utils/discordWebhookUrl`. Wren's ruling: option A, keyed with `kind`.

The same change fixes the three blockers the design exposed, which is what makes
the encryption safe to turn on rather than a new way to lose a live credential:

1. `sweepOrphanedConnectorSecrets` kept a secret only on a Slack
   `botTokenRef`/`pendingBind.botTokenRef` pair, so it revoked a Discord secret
   ten minutes after it was written (`ORPHAN_SECRET_GRACE_MS`). Its
   keep-condition is now drawn from the kinds registry.
2. `sweepUnavailableSlackSecretKeys` marked ANY row with a Slack-named reason,
   and since #1878/#1881 that string is user-facing — a Discord owner read
   "Slack connector secret key is unavailable". Renamed and made kind-aware;
   Slack's sentence is byte-identical.
3. `upsertEncrypted` filtered on `{ integrationId }` alone, on both the upsert
   and the 11000 retry, so a second kind did not collide: it MATCHED the first
   kind's row and $set the new ciphertext onto it, leaving the first ref valid
   and decrypting to the second secret. The filter now carries `kind`, the model
   declares ONE index on the pair, and `services/connectorSecretKinds` is the
   single definition all four sites read.

Also: `webhookUrl` and `webhookUrlRef` join `SERVER_OWNED_CONFIG_KEYS` (the URL
is derived by the server on both writers, and a planted ref would point a row at
another row's secret), and `webhookUrlRef` is listed in
`INTEGRATION_SECRET_CONFIG_KEYS` so the pointer never rides out either.

The migration is a separate gated step: `scripts/encrypt-discord-webhook-url`
(dry-run by default, `--apply` to write, digests not URLs, idempotent, reads
both plaintext stores and unsets both, refuses to write at all when the key ring
is unusable). Run it after this deploys — the old image reads the plaintext
field directly.

Evidence: full backend tier 446 suites / 4167 tests green; tsc 0; eslint .ts 0
errors at the 1210-warning baseline. Eight-arm mutation ledger, no survivors:
each arm reverts one piece and every arm reddens a named test.
…nd drop the inert pre-flight

vera's hold at `e6d4077c`: the ordering that prevents data loss in
`encrypt-discord-webhook-url` was unwitnessed. Moving both `$unset`s above the
`connectorSecrets.put` (the arrangement where a failing put destroys the only
copy) left the suite 8/8, and deleting the `listWithUnavailableKey()` pre-flight
reddened nothing.

Both observations were right, and the second one was the more interesting:
`listWithUnavailableKey` returns refs whose `keyId` is not in the ring. It cannot
fail. A call that cannot fail is not a guard, so it is gone — the script no
longer claims to fail fast, and its comment now states the invariant that
actually protects the data: per row, the `put` completes before either plaintext
is cleared, so ANY failure leaves that row whole and a re-run reports it as still
waiting.

Two witnesses replace the claim:

- a failed `put` leaves both plaintexts in place and reports nothing migrated
  (asserts the platform store, the legacy `config.webhookUrl` store, the absence
  of a ref, and that a following dry run still sees `candidates=1, migrated=0`);
- clearing happens only after the row's own ref is stored, one row further in: the
  second row's `put` rejects, so the first must have completed end to end (the
  test cannot pass vacuously) and the second must be untouched.

The empty-ring test keeps its place as the ring-shaped instance, with its comment
corrected: what protects it is the ordering, not the deleted pre-flight.

Mutation ledger, script only: vera's mutation reddens 3 named tests (both new
witnesses + the empty-ring one); removing the dry-run guard reddens 1 as a
control that the ledger sees this file at all; re-adding the deleted pre-flight
is a SURVIVOR by design, which is the evidence for removing it rather than
witnessing it.
…d pre-flight

vera's correction at 74198, and she is right: `listWithUnavailableKey` CAN fail.
Its first statement is `const { keys } = parseKeyRing()` (services/connectorSecrets.ts:171),
and `parseKeyRing` throws `ConnectorSecretConfigurationError` on an unset,
malformed or duplicated ring and `ConnectorSecretKeyMissing` when the active key
is absent (:49-79). I wrote "it cannot fail" by quoting the find line and
dropping the line above it.

So the pre-flight was redundant, not inert: `put` computes
`encrypt(material, parseKeyRing())` before `upsertEncrypted` touches anything
(:138-146), so an unusable ring already fails the first row ahead of either
`$unset`. Deleting it remains right; the reason recorded for the deletion was
wrong, and on the one script in this lane whose damage is irreversible.

The comment now says what is true, in vera's wording; the same false claim is
corrected in the test comment that cited it. Comment-only, so the clearance at
`d015ad25` stands: the only changed lines in this commit begin with `//`, `*` or
`/**`.
samxu01 pushed a commit that referenced this pull request Sep 25, 2026
…its locators

wren's hold at 272ca93: rule 24 is the right home and the suite claim holds,
but two locators do not, and the PR merges after #1898. Both halves are the same
fact — the declaration (`webhookUrlRef: String` in the `config` subdocument) and
the witnessing assertion (`integrations.discordTokenCopy.test.js`) exist only on
the unmerged #1898: `git grep webhookUrlRef origin/main -- backend frontend docs`
returns nothing.

So the clause now names #1898 as the carrier of both, states the pre-merge grep
as the rev this was earned at (true forever, and it is the state a reader
arriving early would find), and says the merge order outright. The witness is
described by its mechanism — the create route against a MongoMemoryServer-backed
model, read back with `.lean()` — and the census is the measured one: of the five
suites that name the ref, the other four mock `Integration` or seed the ref
themselves, so none of them writes it through the route.

Docs-only, one line.
samxu01 pushed a commit that referenced this pull request Sep 25, 2026
…oun for `config`

wren's two corrections (74203/74204, relayed by connector-ops at 74205) applied,
both verified at the premise first:

1. The writer is not `updateOne`. On #1898 the connect route writes with
   `Integration.findByIdAndUpdate(..., { $set: { 'config.webhookUrlRef': ref } })`
   (`routes/integrations.ts:474`) and the backfill does the same
   (`services/discordService.ts:345`). The only `updateOne` is the migration
   script's, and it runs on `mongoose.connection.collection()` — a raw-collection
   write, which strict mode never touches. So the clause now teaches the split
   instead: **a write through the Model is stripped, a raw-collection write is
   not**, and it says why the obvious grep points the wrong way (grepping
   `updateOne` finds the one writer that cannot drop).

2. `config` is not a subdocument. Measured from the model at 715e187 rather than
   restated: `schema.path('config')` is `undefined`, `schema.nested.config` is
   true, `schema.paths` carries no `config` key, and `schema.get('strict')` is
   true. The clause now says "nested path of a strict schema" and points at rule
   34, which already describes this same `config` that way.

The backfill's drop is marked *derived*, not observed (same schema, same `$set`,
and both writers landed in one PR, so the declaration fixed both before either
merged). #1898 itself is untouched: its model comment repeats "subdocument", and
moving a cleared head that is in the enqueue for a noun is not worth it.

Docs-only, one line.
samxu01 pushed a commit that referenced this pull request Sep 25, 2026
…tance

wren's third correction: the instance was earned ON #1898, and there
`webhookUrlRef` appears in 15 backend files, not zero. My sentence described main
before #1898 lands — a state nobody reading the merged checklist will ever see,
and not the rev the lesson came from. Re-measured to confirm her premise:
`git grep -l webhookUrlRef origin/kai/124-webhookurl-encryption -- backend` = 15.

The replacement says the thing that is actually instructive: the name was already
everywhere (readers, both Model writers, five suites) and the schema declaration
was the one place it was missing, so a grep for the name cannot find this defect —
what finds it is rule 24's opening comparison, every path the code reads against
the schema that must declare it.

Docs-only, one line.
@lilyshen0722
lilyshen0722 added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit 96865bb Sep 25, 2026
20 of 23 checks passed
lilyshen0722 added a commit to thantiklermcirony/commonly that referenced this pull request Oct 9, 2026
…t drop (Team-Commonly#1900)

* docs(review): rule 24 gains the update shape of the strict-subdocument drop

vera asked (74186) where Team-Commonly#1898's strict-subdocument finding belongs. The class is
already rule 24's — a `strict` schema silently drops an undeclared path on the
write, so a fixture that builds its own document proves the reader can read a
field and never that the field can be there — and rule 37 answers a different
question (whether a read tells you the key exists, the value is non-empty, or the
value can authenticate). Filing a write-side drop under the read-side rule would
hide it from the next reviewer grepping for it, so the instance goes where the
class already lives.

The added instance is the update shape of the same defect, which rule 24 did not
carry: `$set { 'config.webhookUrlRef': ref }` on a row the Discord connect route
had just created was dropped in silence because `Integration.config` is a strict
subdocument that did not declare the path. Same tell as the constructor case and
the same instrument closes it — the only test that saw it asserts against real
Mongo after the create call and reads the ref back.

Docs-only, one file.

* docs(review): point rule 24's second instance at the PR that carries its locators

wren's hold at 272ca93: rule 24 is the right home and the suite claim holds,
but two locators do not, and the PR merges after Team-Commonly#1898. Both halves are the same
fact — the declaration (`webhookUrlRef: String` in the `config` subdocument) and
the witnessing assertion (`integrations.discordTokenCopy.test.js`) exist only on
the unmerged Team-Commonly#1898: `git grep webhookUrlRef origin/main -- backend frontend docs`
returns nothing.

So the clause now names Team-Commonly#1898 as the carrier of both, states the pre-merge grep
as the rev this was earned at (true forever, and it is the state a reader
arriving early would find), and says the merge order outright. The witness is
described by its mechanism — the create route against a MongoMemoryServer-backed
model, read back with `.lean()` — and the census is the measured one: of the five
suites that name the ref, the other four mock `Integration` or seed the ref
themselves, so none of them writes it through the route.

Docs-only, one line.

* docs(review): name the Model-vs-raw-collection split, and the right noun for `config`

wren's two corrections (74203/74204, relayed by connector-ops at 74205) applied,
both verified at the premise first:

1. The writer is not `updateOne`. On Team-Commonly#1898 the connect route writes with
   `Integration.findByIdAndUpdate(..., { $set: { 'config.webhookUrlRef': ref } })`
   (`routes/integrations.ts:474`) and the backfill does the same
   (`services/discordService.ts:345`). The only `updateOne` is the migration
   script's, and it runs on `mongoose.connection.collection()` — a raw-collection
   write, which strict mode never touches. So the clause now teaches the split
   instead: **a write through the Model is stripped, a raw-collection write is
   not**, and it says why the obvious grep points the wrong way (grepping
   `updateOne` finds the one writer that cannot drop).

2. `config` is not a subdocument. Measured from the model at 715e187 rather than
   restated: `schema.path('config')` is `undefined`, `schema.nested.config` is
   true, `schema.paths` carries no `config` key, and `schema.get('strict')` is
   true. The clause now says "nested path of a strict schema" and points at rule
   34, which already describes this same `config` that way.

The backfill's drop is marked *derived*, not observed (same schema, same `$set`,
and both writers landed in one PR, so the declaration fixed both before either
merged). Team-Commonly#1898 itself is untouched: its model comment repeats "subdocument", and
moving a cleared head that is in the enqueue for a noun is not worth it.

Docs-only, one line.

* docs(review): drop the false pre-merge grep from rule 24's second instance

wren's third correction: the instance was earned ON Team-Commonly#1898, and there
`webhookUrlRef` appears in 15 backend files, not zero. My sentence described main
before Team-Commonly#1898 lands — a state nobody reading the merged checklist will ever see,
and not the rev the lesson came from. Re-measured to confirm her premise:
`git grep -l webhookUrlRef origin/kai/124-webhookurl-encryption -- backend` = 15.

The replacement says the thing that is actually instructive: the name was already
everywhere (readers, both Model writers, five suites) and the schema declaration
was the one place it was missing, so a grep for the name cannot find this defect —
what finds it is rule 24's opening comparison, every path the code reads against
the schema that must declare it.

Docs-only, one line.

* docs(review): rule 24's second instance — drop the count, fix the referent

wren's two swaps, exactly as given:

  "at the rev this was earned the name already appeared in **15 backend files**"
  -> "the name already appeared across the backend"

  "the comparison this rule opens with" -> "the comparison this rule's check opens with"

She is right on the second: the rule opens with its thesis ("A fixture that builds
its own document proves the reader can read a field"), and the comparison is the
first step of its check ("The check is two greps and one round-trip: for each field
the code reads cross-layer, confirm the writing schema declares that exact path").

Nothing else changed: no other path, no other word on the line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant