feat(content-freshness): unpublished changes and expired entries

packages/content-freshness/README.md

@@ -10,9 +10,25 @@ What it looks for:
 | Scheduled to publish, or scheduled changes to a published entry, and the date passed without them going live | urgent |
 | Published without an SEO description | should fix; nice to fix when the entry has an excerpt-like field |
 | Published and untouched for longer than your threshold | should fix |
+| Changes saved to a published entry and not published for longer than your threshold | should fix |
+| Published although the date in its expiry field has passed (opt-in per collection) | should fix |
 | SEO description too short or too long | nice to fix |
 | Draft abandoned for longer than your threshold | nice to fix |
 
+Unpublished changes are counted in collections that keep revisions, and not
+while the changes are scheduled to go live. An entry someone saved recently
+is not reported as stale: someone is working on it, and if they stop, the
+unpublished changes are reported instead.
+
+Expiry is opt-in per collection, because a site that keeps past events online
+as an archive would otherwise get a finding for each of them. When a
+collection has a `datetime` field named like `valid_until`, `end_date`,
+`expires_at`, `valid_through` or `deadline`, the report suggests it; choose
+it, another field, or "Never expires" in the settings. EmDash stores a date
+without a time as midnight in the site's timezone, and the finding names that
+instant in UTC. An ignored expiry comes back when the date is moved and
+passes again.
+
 Many sites render a description of their own when the SEO panel is empty,
 usually from an excerpt or intro field. An entry with a non-empty `excerpt`,
 `description`, `summary`, `intro`, `teaser`, `lead`, `subtitle`, or a field
@@ -27,10 +43,12 @@ templates always render a description can switch the check off.
   priority. Each entry links straight into its editor.
 - **The Freshness panel** in the entry editor shows that entry's findings. It
   checks the entry each time it opens, and it is where a finding is set aside:
-  **Mark as reviewed** holds a stale entry or a forgotten draft back until its
-  threshold runs out again, so a page that is still correct needs no edit to
-  quiet the report. **Ignore** holds a description finding back for good. A
-  missed schedule cannot be set aside. **Undo** brings a finding back.
+  **Mark as reviewed** holds a stale entry, a forgotten draft or forgotten
+  unpublished changes back until the threshold runs out again, so a page that
+  is still correct needs no edit to quiet the report. **Ignore** holds a
+  description finding, or an expiry on a page kept online as an archive, back
+  for good. A missed schedule cannot be set aside. **Undo** brings a finding
+  back.
   Authors see the panel on their own entries; the report is for editors and
   above.
 - **The dashboard widget** counts the entries by priority and says how many
@@ -90,11 +108,12 @@ entry. Without one, the audit is scheduled and never runs.
 | --- | --- | --- |
 | Stale after | 12 months | For published entries. |
 | Draft forgotten after | 6 months | For drafts. |
+| Unpublished changes forgotten after | 14 days | For changes saved to a published entry. |
 | SEO description | 50–160 characters | Outside this range is a low-priority finding. |
 | Report entries without an SEO description | on | Off for a site whose templates always render a description. The length check still runs. |
 | Entries per run | 50 | At most 98: D1 binds at most 100 parameters per statement, and storage adds two of its own. The number of calls per run does not grow with it. |
 | Schedule | `0 4 * * *` | A cron expression, in UTC. One the scheduler rejects is not saved. |
-| Per collection | none | Months for stale entries and forgotten drafts of one collection (empty: the site's value, 0: never), or leave the collection out of the audit. |
+| Per collection | none | Months for stale entries and forgotten drafts of one collection (empty: the site's value, 0: never), its expiry field, or leave the collection out of the audit. |
 
 ## Develop
 
@@ -109,10 +128,10 @@ pnpm build
 
 | EmDash | Plugin test suite |
 | ------ | ----------------- |
-| 0.40.1 | 99 passed |
-| 0.40.0 | 99 passed |
-| 0.39.1 | 99 passed |
-| 0.39.0 | 99 passed |
+| 0.40.1 | 119 passed |
+| 0.40.0 | 119 passed |
+| 0.39.1 | 119 passed |
+| 0.39.0 | 119 passed |
 
 Declared range: `emdash >=0.39.0`, with no upper bound. The table covers
 **every EmDash release since the floor**. 0.39.0 is the floor because the

packages/content-freshness/src/check.ts

@@ -7,7 +7,7 @@ import type { PluginContext } from "emdash/plugin";
 
 import {
 	dismissalFor,
-	isSetAside,
+	hasRunOut,
 	partition,
 	readDismissals,
 	writeDismissals,
@@ -16,8 +16,8 @@ import {
 } from "./dismissals.js";
 import { evaluateEntries, forget, storeOne, type EntryFindings, type EntryRef } from "./findings.js";
 import { findingId } from "./ids.js";
-import { evaluateEntry, type Hit, type Rule } from "./rules.js";
-import { readSettings, thresholdsFor } from "./settings.js";
+import { datetimeFields, evaluateEntry, type Hit, type Rule } from "./rules.js";
+import { contextFor, readSettings, thresholdsFor } from "./settings.js";
 
 export interface PanelAction {
 	kind: "set-aside" | "undo";
@@ -34,16 +34,17 @@ export interface PanelView {
 
 /**
  * Apply `action`, check the entry, store its row and return what the panel
- * shows. Four `ctx` calls; one more for a changed row, one more for an
+ * shows. Five `ctx` calls; one more for a changed row, one more for an
  * action or an expired review.
  */
 export async function checkEntry(ctx: PluginContext, ref: EntryRef, action?: PanelAction): Promise<PanelView> {
 	if (!ctx.content) return { active: [], setAside: [] };
-	const [settings, entry, stored, existing] = await Promise.all([
+	const [settings, entry, stored, existing, schema] = await Promise.all([
 		readSettings(ctx),
 		ctx.content.get(ref.collection, ref.id),
 		readDismissals(ctx, ref.collection, ref.id),
 		ctx.storage.findings.get(findingId(ref.collection, ref.id)) as Promise<EntryFindings | null>,
+		ctx.schema ? ctx.schema.getCollection(ref.collection) : Promise.resolve(null),
 	]);
 	const thresholds = thresholdsFor(settings, ref.collection);
 	if (!entry || thresholds.skip) {
@@ -52,11 +53,18 @@ export async function checkEntry(ctx: PluginContext, ref: EntryRef, action?: Pan
 	}
 
 	const now = new Date();
+	const context = contextFor(
+		settings,
+		ref.collection,
+		schema ? { dateFields: datetimeFields(schema.fields), revisions: schema.supports.includes("revisions") } : undefined,
+	);
+	const hits = evaluateEntry(entry, thresholds, now, context);
+
 	const rules = { ...stored?.rules };
 	// Reviews that have run out are dropped whenever the entry is checked.
 	let changed = false;
-	for (const rule of Object.keys(rules) as Rule[]) {
-		if (!isSetAside({ collection: ref.collection, entryId: ref.id, rules }, rule, now)) {
+	for (const [rule, dismissal] of Object.entries(rules) as Array<[Rule, Dismissal]>) {
+		if (hasRunOut(dismissal, now)) {
 			delete rules[rule];
 			changed = true;
 		}
@@ -65,7 +73,8 @@ export async function checkEntry(ctx: PluginContext, ref: EntryRef, action?: Pan
 		if (action.kind === "undo") {
 			delete rules[action.rule];
 		} else {
-			const dismissal = dismissalFor(action.rule, thresholds, now, action.by);
+			const hit = hits.find((candidate) => candidate.rule === action.rule);
+			const dismissal = dismissalFor(action.rule, thresholds, now, action.by, hit);
 			if (dismissal) rules[action.rule] = dismissal;
 		}
 		changed = true;
@@ -74,8 +83,9 @@ export async function checkEntry(ctx: PluginContext, ref: EntryRef, action?: Pan
 	if (changed) await writeDismissals(ctx, dismissals);
 
 	const keyed = new Map([[findingId(ref.collection, ref.id), dismissals]]);
-	await storeOne(ctx, evaluateEntries(ref.collection, [entry], thresholds, now, now.toISOString(), keyed), existing);
-	const { active, setAside } = partition(evaluateEntry(entry, thresholds, now), dismissals, now);
+	const evaluated = evaluateEntries(ref.collection, [entry], thresholds, now, now.toISOString(), keyed, context);
+	await storeOne(ctx, evaluated, existing);
+	const { active, setAside } = partition(hits, dismissals, now);
 	return { active, setAside: setAside.map((hit) => ({ hit, dismissal: dismissals.rules[hit.rule]! })) };
 }
 

packages/content-freshness/src/describe.ts

@@ -23,6 +23,10 @@ export function describeHit(lang: Lang, hit: Hit): string {
 			return t(lang, "hitStale", params);
 		case "stale-draft":
 			return t(lang, "hitStaleDraft", params);
+		case "unpublished-changes":
+			return t(lang, "hitUnpublishedChanges", params);
+		case "expired":
+			return t(lang, "hitExpired", params);
 	}
 }
 

packages/content-freshness/src/dismissals.ts

@@ -4,14 +4,15 @@
  *
  * "Mark as reviewed" holds a time-based finding back until the threshold has
  * run out again: an imprint that is still correct should not be edited just
- * to make the report quiet. "Ignore" holds a description finding back for
- * good. A missed schedule cannot be set aside.
+ * to make the report quiet. "Ignore" holds a description finding, or an
+ * expiry on a page kept online as an archive, back for good. A missed
+ * schedule cannot be set aside.
  */
 
 import type { PluginContext } from "emdash/plugin";
 
 import { findingId } from "./ids.js";
-import { subtractMonths, type Hit, type Rule, type Thresholds } from "./rules.js";
+import { DAY_MS, subtractMonths, type Hit, type Rule, type Thresholds } from "./rules.js";
 
 export type DismissKind = "review" | "ignore";
 
@@ -20,13 +21,21 @@ export const DISMISSIBLE: Partial<Record<Rule, DismissKind>> = {
 	"stale-draft": "review",
 	"missing-description": "ignore",
 	"description-length": "ignore",
+	"unpublished-changes": "review",
+	"expired": "ignore",
 };
 
 export interface Dismissal {
 	/** When the finding may come back; `null` for never. */
 	until: string | null;
 	by: string | null;
 	at: string;
+	/**
+	 * What exactly was set aside, for a finding that can recur with other
+	 * values: an ignored expiry names its date, so moving the date forward
+	 * and letting it pass again is reported.
+	 */
+	subject?: string;
 }
 
 export interface EntryDismissals {
@@ -35,25 +44,46 @@ export interface EntryDismissals {
 	rules: Partial<Record<Rule, Dismissal>>;
 }
 
-export function isSetAside(dismissals: EntryDismissals | undefined, rule: Rule, now: Date): boolean {
-	const dismissal = dismissals?.rules[rule];
-	if (!dismissal) return false;
-	return dismissal.until === null || Date.parse(dismissal.until) > now.getTime();
+/** The part of a hit a dismissal must match, for rules whose finding can recur with other values. */
+function subjectOf(hit: Hit): string | undefined {
+	return hit.rule === "expired" ? String(hit.params.date) : undefined;
+}
+
+export function hasRunOut(dismissal: Dismissal, now: Date): boolean {
+	return dismissal.until !== null && Date.parse(dismissal.until) <= now.getTime();
+}
+
+export function isSetAside(dismissals: EntryDismissals | undefined, hit: Hit, now: Date): boolean {
+	const dismissal = dismissals?.rules[hit.rule];
+	if (!dismissal || hasRunOut(dismissal, now)) return false;
+	return dismissal.subject === undefined || dismissal.subject === subjectOf(hit);
 }
 
 /** `hits` split into those that stand and those set aside. */
 export function partition(hits: Hit[], dismissals: EntryDismissals | undefined, now: Date) {
 	const active: Hit[] = [];
 	const setAside: Hit[] = [];
-	for (const hit of hits) (isSetAside(dismissals, hit.rule, now) ? setAside : active).push(hit);
+	for (const hit of hits) (isSetAside(dismissals, hit, now) ? setAside : active).push(hit);
 	return { active, setAside };
 }
 
-/** A dismissal for `rule`, or null when the rule cannot be set aside. */
-export function dismissalFor(rule: Rule, thresholds: Thresholds, now: Date, by: string | null): Dismissal | null {
+/** A dismissal for `rule`, or null when the rule cannot be set aside. `hit` is the finding being set aside. */
+export function dismissalFor(
+	rule: Rule,
+	thresholds: Thresholds,
+	now: Date,
+	by: string | null,
+	hit?: Hit,
+): Dismissal | null {
 	const kind = DISMISSIBLE[rule];
 	if (!kind) return null;
-	if (kind === "ignore") return { until: null, by, at: now.toISOString() };
+	if (kind === "ignore") {
+		const subject = hit ? subjectOf(hit) : undefined;
+		return { until: null, by, at: now.toISOString(), ...(subject !== undefined ? { subject } : {}) };
+	}
+	if (rule === "unpublished-changes") {
+		return { until: new Date(now.getTime() + thresholds.pendingDays * DAY_MS).toISOString(), by, at: now.toISOString() };
+	}
 	const months = rule === "stale" ? thresholds.staleMonths : thresholds.draftMonths;
 	return { until: subtractMonths(now, -months).toISOString(), by, at: now.toISOString() };
 }

packages/content-freshness/src/findings.ts

@@ -12,7 +12,7 @@ import type { PluginContext } from "emdash/plugin";
 
 import { partition, type EntryDismissals } from "./dismissals.js";
 import { findingId } from "./ids.js";
-import { evaluateEntry, worstRank, type Hit, type Thresholds } from "./rules.js";
+import { evaluateEntry, NO_CONTEXT, worstRank, type EntryContext, type Hit, type Thresholds } from "./rules.js";
 
 
 export interface EntryFindings {
@@ -48,11 +48,12 @@ export function evaluateEntries(
 	now: Date,
 	seenIn: string,
 	dismissals: Map<string, EntryDismissals> = new Map(),
+	context: EntryContext = NO_CONTEXT,
 ): Evaluated {
 	const result: Evaluated = { writes: [], clears: [] };
 	for (const entry of entries) {
 		const id = findingId(collection, entry.id);
-		const hits = partition(evaluateEntry(entry, thresholds, now), dismissals.get(id), now).active;
+		const hits = partition(evaluateEntry(entry, thresholds, now, context), dismissals.get(id), now).active;
 		if (hits.length === 0) {
 			result.clears.push(id);
 			continue;

packages/content-freshness/src/i18n.ts

@@ -62,6 +62,15 @@ const en = {
 	reportMissingHelp: "Switch off if your templates always render a description of their own.",
 	pageSize: "Entries per run",
 	schedule: "Schedule (cron, UTC)",
+	pendingDays: "Days before unpublished changes to a published entry count as forgotten",
+	collectionExpiry: "{collection}: expires after the date in",
+	expiryUnset: "Not checked",
+	expirySuggested: "{field} (suggested)",
+	expiryOff: "Never expires",
+	expirySuggestTitle: "Expiry dates not checked: {names}",
+	expirySuggestText:
+		"These collections have a date field that looks like an expiry date. Choose it in the Freshness settings to report entries still published after it, or choose “Never expires”.",
+	openSettings: "Open settings",
 	collectionStale: "{collection}: months before an entry is stale (empty: {months}, 0: never)",
 	collectionDraft: "{collection}: months before a draft is forgotten (empty: {months}, 0: never)",
 	collectionSkip: "{collection}: leave out of the audit",
@@ -77,6 +86,8 @@ const en = {
 	hitDescriptionLength: "SEO description is {length} characters; aim for {min}–{max}.",
 	hitStale: "Not touched since {since}.",
 	hitStaleDraft: "Draft untouched since {since}.",
+	hitUnpublishedChanges: "Changes saved on {since} are not published yet.",
+	hitExpired: "{field} was {date}, and the entry is still published.",
 } satisfies Record<string, Message>;
 
 export type MessageKey = keyof typeof en;
@@ -133,6 +144,15 @@ const de: Record<MessageKey, Message> = {
 	reportMissingHelp: "Schalte das ab, wenn deine Templates immer eine eigene Beschreibung ausgeben.",
 	pageSize: "Einträge pro Lauf",
 	schedule: "Zeitplan (Cron, UTC)",
+	pendingDays: "Tage, bis unveröffentlichte Änderungen an einem veröffentlichten Eintrag als vergessen gelten",
+	collectionExpiry: "{collection}: läuft ab nach dem Datum in",
+	expiryUnset: "Nicht geprüft",
+	expirySuggested: "{field} (vorgeschlagen)",
+	expiryOff: "Läuft nie ab",
+	expirySuggestTitle: "Ablaufdaten nicht geprüft: {names}",
+	expirySuggestText:
+		"Diese Kollektionen haben ein Datumsfeld, das nach einem Ablaufdatum aussieht. Wähle es in den Einstellungen zur Aktualität, um Einträge zu melden, die danach noch veröffentlicht sind, oder wähle „Läuft nie ab“.",
+	openSettings: "Einstellungen öffnen",
 	collectionStale: "{collection}: Monate, bis ein Eintrag veraltet ist (leer: {months}, 0: nie)",
 	collectionDraft: "{collection}: Monate, bis ein Entwurf vergessen ist (leer: {months}, 0: nie)",
 	collectionSkip: "{collection}: von der Prüfung ausnehmen",
@@ -148,6 +168,8 @@ const de: Record<MessageKey, Message> = {
 	hitDescriptionLength: "Die SEO-Beschreibung hat {length} Zeichen; empfohlen sind {min}–{max}.",
 	hitStale: "Seit {since} nicht bearbeitet.",
 	hitStaleDraft: "Entwurf seit {since} nicht bearbeitet.",
+	hitUnpublishedChanges: "Am {since} gespeicherte Änderungen sind noch nicht veröffentlicht.",
+	hitExpired: "{field} war am {date}, der Eintrag ist aber noch veröffentlicht.",
 };
 
 const catalogues = { en, de } as const;

packages/content-freshness/src/plugin.ts

@@ -24,7 +24,7 @@ import {
 } from "./report.js";
 import { renderPanel, ruleOf, SET_ASIDE_ACTION, UNDO_ACTION } from "./panel.js";
 import { ensureScheduled } from "./schedule.js";
-import { applyForm, readStoredSettings, writeStoredSettings } from "./settings.js";
+import { applyForm, readSettings, readStoredSettings, writeStoredSettings } from "./settings.js";
 import { readState } from "./state.js";
 import { AUDIT_NOW_TASK, runAudit } from "./sweep.js";
 
@@ -133,7 +133,7 @@ const plugin = {
 					const { settings, state } = await loadAdmin(ctx);
 					if (interaction.page === "widget:summary") return buildWidget(ctx, state, lang, routeCtx.user?.id);
 					if (interaction.page === "/settings") return buildSettingsPage(settings, lang, await listCollections(ctx));
-					return buildReportPage(ctx, state, lang);
+					return buildReportPage(ctx, state, settings, lang);
 				}
 
 				if (interaction.type === "form_submit" && interaction.action_id === SAVE_SETTINGS_ACTION) {
@@ -142,13 +142,14 @@ const plugin = {
 
 				if (interaction.type === "block_action") {
 					if (interaction.action_id === VIEW_ACTION) {
-						return buildReportPage(ctx, await readState(ctx), lang, viewFrom(interaction.value));
+						const [state, settings] = await Promise.all([readState(ctx), readSettings(ctx)]);
+						return buildReportPage(ctx, state, settings, lang, viewFrom(interaction.value));
 					}
 					if (interaction.action_id === AUDIT_NOW_ACTION) {
 						await ctx.cron?.schedule(AUDIT_NOW_TASK, { schedule: new Date(Date.now() + 1000).toISOString() });
-						const state = await readState(ctx);
+						const [state, settings] = await Promise.all([readState(ctx), readSettings(ctx)]);
 						return {
-							...(await buildReportPage(ctx, state, lang, viewFrom(interaction.value))),
+							...(await buildReportPage(ctx, state, settings, lang, viewFrom(interaction.value))),
 							toast: { message: t(lang, state.sweep ? "auditContinues" : "auditStarts"), type: "info" },
 						};
 					}