💡 Golden-set evaluation harness for gmail-ai-classifier (gate model migrations and rule changes on measured accuracy)

[github-actions[bot]]

Summary

Add a versioned, anonymised fixture set of labelled emails (sender, subject, snippet, expected domain, sub-label and category) and an eval runner. The runner reports per-domain precision and recall for the deterministic layer in CI, and optionally for the live model when a maintainer runs it manually. Model changes and rule PRs then come with an accuracy diff instead of anecdotes.

Market Signal

Golden-set regression testing with baseline diffs and release gates is now standard practice for LLM features: rerun the golden set on every prompt, model or rule change. Research on LLM email classification shows that spam-vs-legitimate is the hardest boundary, so per-class tracking matters. Google has published shutdown dates for gemini-2.5-flash and gemini-2.5-pro (2026-12-01, earliest possible), with automatic rerouting to gemini-3.5-flash-lite and gemini-3.8-flash. Those are different model tiers, so classification behaviour will change without any code change.

User Signal

Five classifier PRs in 30 days each added rules with no measured effect on other categories. Collisions between shields (for example Eventbrite appearing in both a promo rule and a ticket rule) can only be spotted by reading code. The test suite (code.test.js, review-fixes.test.js and others) checks behaviour case by case but has no aggregate accuracy metric. config.gs defaults to gemini-2.5-flash, and code.gs hardcodes 2.5-generation fallback endpoints.

Technical Opportunity

The deterministic layer (DETERMINISTIC_RULE_SPECS_ and the shields) is pure logic that already runs under Jest, so a fixtures/*.jsonl corpus plus a Jest-driven scorer can run in CI with no network access, which keeps to the repo's 'tests must not access external networks' rule. A separate npm script (eval:live, never part of CI) can replay the same corpus against a chosen model with a key from the maintainer's environment and write a markdown scorecard. Fixtures can be built from the existing test cases, which already contain realistic sender and subject pairs.

Assessment

Dimension	Score	Rationale
Feasibility	high	Grounded in existing seams described above
Impact	med	See Market and User Signal
Urgency	high	See Market Signal (dated platform changes)

Adversarial Review

Strongest objection: A synthetic or anonymised corpus may not reflect real inboxes, and keeping it up to date is extra work for a one-maintainer project.

Rebuttal: The corpus starts from cases the tests already encode, and it grows from production misses: each 'add a shield' PR adds the triggering email as a fixture, which costs nothing extra. Even a 200-row corpus catches cross-category regressions that case-by-case tests miss. The upcoming forced model reroute is the immediate reason to have a baseline before 2026-12-01.

Suggested Next Step

Turn the existing classifier test cases into src/gmail-ai-classifier/tests/fixtures/golden.jsonl (about 150 rows covering all 7 domains plus Promotions and Updates). Add a Jest scorer that prints a per-domain confusion summary and fails if deterministic-layer accuracy drops more than 2 points from a committed baseline.

---

Generated by the weekly feature-ideation agent (Mary, BMAD Analyst). Sources: Gemini API terms · Gemini deprecations · Apps Script release notes · Vertex AI advanced service · Gmail Gemini era · SaneBox review · Workspace Studio