Skip to content

fix(content-freshness): works sandboxed, pages its report, forgets deleted entries - #12

Merged
danielmlr merged 13 commits into
mainfrom
fix/content-freshness-phase1
Sep 25, 2026
Merged

danielmlr merged 13 commits into
mainfrom
fix/content-freshness-phase1

Conversation

@danielmlr

Copy link
Copy Markdown
Member

Phase 1 of the content-freshness plan: the bugs that would have shipped. Stacked on #9, which is stacked on #8. Merge those first with merge commits, then delete their branches, and GitHub retargets this PR to main.

Nothing is published, so the storage layout and settings keys change without a migration. No changeset: a changeset would make the release workflow open a version PR and publish to npm. The first changeset comes with the release.

What was broken

The plugin had only ever run in-process. Running it through the sandboxed test host showed most of this.

  • Sandboxed, every admin surface with findings answered 502 (INVALID_BLOCK_RESPONSE).

    • The stats blocks sent stats instead of items.
    • The table had no page_action_id.
    • The settings form sent elements instead of fields and submit.

    EmDash validates only sandboxed responses. Blocks are now typed with @emdash-cms/blocks 0.39.0 (types only), and tests load every surface through the sandboxed host.

  • A cron run made 16 bridge calls against the sandbox's 10. Seven were KV reads for settings, then one per collection cursor, a cron.list and a log line. Settings and sweep state are now one KV key each. tests/budget.test.ts counts every invocation's worst case (measured below).

  • The report showed at most 100 findings, and not the most urgent ones. It asked storage for 200 rows, got at most 100 and sorted those. Findings are now one row per entry with an indexed rank, and the table pages through them most urgent first.

  • Rows of trashed or deleted entries, and of removed collections, stayed forever. A finished sweep removes every row it did not confirm, and content:afterDelete removes an entry's row at once.

  • A fix showed only after the next nightly audit. Publish, unpublish, schedule, unschedule and restore now re-check the entry. content:afterSave stays unhooked: it fires on every autosave, which for a sandboxed install means a new isolate every few seconds while someone types.

  • The nightly audit was never scheduled for plugins listed in astro.config.mjs. plugin:activate only fires on a runtime enable or install. The dashboard widget and the report now schedule it on first view.

  • A missed scheduled update of a published entry went unreported. The rule now also covers published entries, after an hour of grace for EmDash's own scheduler.

  • Two settings problems. "Entries per run" accepted 200 where content.list returns 100. An invalid cron expression was stored and only failed later; now the form keeps the old schedule and says why.

  • A malformed page cursor made the report route answer 400. It now falls back to the first page.

Findings now store the rule and its values instead of an English sentence, so the report can be localised (Phase 2) without touching stored data.

Code review

/code-review high ran on the first commit before this PR was opened. It reported 10 findings.

Fixed in the second commit, each with a test that fails when its fix is removed (checked one by one):

  1. A failing content.list wiped a collection's findings. The error counted as "collection deleted", so cleanup removed every finding of a collection that was never audited. Now only a collection that schema.getCollection no longer knows is skipped. Any other error propagates before the state is written, and the next scheduled run retries the page.
  2. The audit was never scheduled again after a reinstall. An uninstall keeps KV but deletes cron tasks, so a stale record blocked scheduling. Activation now always schedules.
  3. An admin request could overwrite a cron run's progress by writing the whole sweep state back. The registered schedule is now recorded in the settings key, which only admin requests write. (No test: it's a race.)
  4. A scheduling error took down every admin page. Page loads now log it and render. The settings form reports the scheduler's own reason instead of calling every failure an invalid expression.
  5. "Load more" after the cursor's row was removed returned an empty page. It now falls back to the first page.
  6. The status route's counts had no names. They now come as entriesBySeverity: { high, medium, low, total }.

Not changed:

  • Migrating rows and settings from the old format (2 findings). Never published or deployed; the only installs were local development databases.
  • A trashed entry rewritten by a run that listed it moments before. The window is the milliseconds between content.list and putMany inside one run, and the next sweep removes the row.
  • No changeset. See above.

The review also looked at whether dropping the cron.list guard allows parallel sweeps, and concluded it doesn't: the cron runner processes due tasks one at a time, and a doubled chain merges back after one run.

Verified

  • 65 tests (was 38), emdash-plugin validate, typecheck and build. The bundle is 12.7 KB.

  • scripts/compat-matrix.sh 0.39.0 0.39.1 0.40.0 on top of the 0.40.1 baseline: content-freshness 65 and analytics 288 on every version.

  • Bridge calls per invocation:

    Invocation Calls
    Run that starts a sweep 8
    Follow-up 7
    Cleanup 6
    Content hook 3
    First dashboard view 8
    First report view 9
    Paging, Audit now, saving settings at most 7

    The limit is 10.

… plugins

Copied unchanged from the proof-of-concept repository at ef4884d, where
the content plugins use them: `chain.ts` carries a sweep across cron runs
with alternating one-shot follow-ups, and `listCollections()` finds every
collection through `schema:read`. `parseCollections()` stays for now.

The floor rises to 0.39.0 because `ctx.schema` first exists there; the
analytics plugin already requires 0.39.0.
Imported from the proof-of-concept repository at ef4884d as fresh files.
A scheduled, network-free audit: overdue schedules, missing or badly
sized SEO descriptions, stale entries, forgotten drafts, and routable
collections without a URL pattern. Capabilities `content:read` and
`schema:read`; it reports and never edits.

Aligned with the workspace on the way in: `registry:publish` and
`registry:login` instead of the generated `publish` and `login`,
`prepublishOnly` builds, repository metadata for trusted publishing, a
peer range with a floor and no ceiling, and the analytics plugin's vitest
limits (30 s timeouts, one test file at a time). Toolchain unchanged from
the proof of concept: plugin-cli 0.12.0, plugin-test 0.2.1, EmDash 0.39.1.
The full-audit test pinned the host's scheduler clock to 04:00:05 on
2026-09-25, but the plugin schedules its follow-up runs from
`Date.now()`. While that night lay ahead of the real time, follow-ups
fell due; since then they land after the host's clock, never run, and
the sweep never finishes. The test now uses tomorrow's night, which is
always ahead of the wall clock. It still fails if a follow-up is lost.
plugin-test 0.2.3 pins EmDash 0.40.1, so the suite's baseline moves from
0.39.1 to the current release. The peer floor stays at 0.39.0: every API
the plugin uses exists there.

`shared` takes its EmDash types from its own dev dependency. Left on
0.39.1, its `PluginContext` rejected the plugin's 0.40.1 one, because 0.40
added the `blocks` field type, so it resolves 0.40.1 as well. `shared`
imports EmDash for types only; analytics keeps its own toolchain and
still typechecks. pnpm added the 0.40.1 packages to
minimumReleaseAgeExclude itself, as it did for 0.39.0.
The gating job ran each suite on its harness's own EmDash plus 0.39.1 and
0.40.0. That covered the floor only while every harness pinned 0.39.0.
content-freshness now runs on plugin-test 0.2.3, which pins 0.40.1, so
0.39.0, the floor both plugins declare, joins the list.
…leted entries

Found while preparing the first release; the plugin had only ever run
in-process.

- Sandboxed, every admin surface with findings answered 502: `stats` used
  `stats` instead of `items`, the table had no `page_action_id`, and the
  settings form sent `elements` instead of `fields` and `submit`. Blocks
  are now typed with @emdash-cms/blocks (types only) and the tests load
  every surface through the sandboxed host, which validates them.
- A cron run made 16 bridge calls against the sandbox's 10: seven KV
  reads for settings, one per collection cursor, a `cron.list`, and a
  log line. Settings and sweep state are one KV key each now, a start
  while a sweep runs continues it instead of checking the task list,
  and a run makes 6 to 8 calls. A budget test counts every invocation.
- The report asked storage for 200 rows, got at most 100 and sorted
  those, so an urgent finding could be missing from the table while the
  stats counted it. Findings are one row per entry with an indexed rank,
  and the table pages through them most urgent first.
- Rows of trashed or deleted entries, and of removed collections, stayed
  forever. A finished sweep removes every row it did not confirm, and
  `content:afterDelete` removes an entry's row at once.
- A fix showed only after the next nightly audit. Publishing,
  unpublishing, scheduling, unscheduling and restoring re-check the
  entry. `content:afterSave` stays unhooked because it fires on every
  autosave.
- A missed scheduled update of a published entry went unreported; the
  rule required an unpublished entry. It now covers both, after an hour
  of grace for EmDash's own scheduler.
- `plugin:activate` never fires for a plugin listed in astro.config.mjs,
  so the nightly audit was never scheduled there. The dashboard widget
  and the report schedule it on first view.
- "Entries per run" accepted 200 where `content.list` returns 100, and
  an invalid cron expression was stored and failed later; the form now
  keeps the old schedule and says why.

Findings store a rule and its values instead of an English sentence, so
the report can be localised without touching stored data.
- A failing `content.list` counted as "collection deleted": the sweep
  skipped the collection, and its cleanup then removed every finding of
  a collection that was never audited. Only a collection that
  `schema.getCollection` no longer knows is skipped now. Any other error
  propagates before the state is written, and the next scheduled run
  retries the page.
- An uninstall that keeps data deletes the plugin's cron tasks but not
  its KV, so a stale "already scheduled" record kept the audit from ever
  being registered again. Activation always schedules.
- Admin requests wrote the whole sweep state back after awaiting the
  scheduler, which could overwrite a cron run's progress. The record of
  the registered schedule lives in the settings key now, which only
  admin requests write.
- A scheduling error took down every admin page, the settings page
  included. Page loads log it and render; the settings form reports the
  scheduler's own reason instead of calling every failure an invalid
  expression.
- "Load more" after the cursor's row was removed returned an empty page,
  because storage continues from that row's current rank. An empty later
  page falls back to the first.
- The status route names its counts: `entriesBySeverity` with high,
  medium, low and total.
…ness-emdash-0.40

# Conflicts:
#	pnpm-lock.yaml
Base automatically changed from chore/content-freshness-emdash-0.40 to main September 25, 2026 21:02
…-phase1

Keeps this branch's versions of the content-freshness files and adds the private flag from main. tests/sweep.test.ts, which this branch replaced, came back through the merge as a file main added and is removed again.
@danielmlr
danielmlr merged commit 1e71b76 into main Sep 25, 2026
2 checks passed
@danielmlr
danielmlr deleted the fix/content-freshness-phase1 branch September 25, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant