Skip to content

docs: define component test sufficiency architecture - #6140

Merged
cixzhang merged 1 commit into
mainfrom
spec-component-test-sufficiency
Sep 9, 2026
Merged

cixzhang merged 1 commit into
mainfrom
spec-component-test-sufficiency

Conversation

@cixzhang

@cixzhang cixzhang commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • defines cross-component test sufficiency without creating product behavior or mandating test counts/internal structure
  • derives critical promises and distinct risk partitions from every applicable current component, module, family, architecture, system, design, and theme record plus objective standards
  • requires mutation-sensitive evidence at the capable layer: compile-time, DOM, browser behavior, pixels, accessibility tree, or real assistive technology
  • explicitly defines when not to add a new automated test
  • rejects tautology, declaration-copy tests, implementation coupling, private-class proxies for pixels, frozen prose, snapshot noise, same-partition duplicates, and open-ended defensive matrices
  • treats existing non-credited tests as optional hardening or cleanup debt, not an activation blocker unless they cause concrete maintenance or confidence harm

What must be covered

Coverage follows applicable current promises and distinct risks, including primary behavior, accessibility, public callbacks/state, compatibility, data integrity, cleanup/lifetime, documented composition boundaries whose failure harms consumers, and previously fixed observable defects.

Separate cases are required only when behavior, ownership, platform semantics, failure mode, or known regression risk differs.

When no new test is warranted

  • Docs-only correction: use existing parser/type/schema guards and source review; recurring deterministic drift gets one generalized validator.
  • Value-only visual change: use real-browser before/after and stable visual evidence; private class assertions cannot prove pixels.
  • Private refactor: run existing mutation-sensitive behavior evidence; do not freeze the new private structure.
  • Already-covered promise: cite the existing test and failure signal; do not duplicate it.
  • Public machine contract: use focused schema/type/inventory evidence for public classes, targets, APIs, or generated schemas.

A docs correction that exposes a real behavior defect still gets the smallest regression test for that behavior—not for the wording that revealed it.

Current-repo calibration

A stratified 12-case sample against current main classified:

  • 3 correct required evidence cases
  • 3 unnecessary/non-credited tests
  • 2 wrong-layer cases
  • 4 duplicate/no-new-test cases
  • 0 untestable

The initial sample found one false-positive wording risk and two false-negative risks. After narrowing private composition, admitting AST-029 established-interface evidence, and preserving named crash-free compatibility promises, the identical rerun produced 0 false positives and 0 false negatives.

Independent review then found that system, design, and theme records were omitted from several authority enumerations. The corrected authority matrix now passes with 0 false positives and 0 false negatives, including a current system-owned AST-005 navigation-sink case and a current theme-application provider-precedence case.

Existing non-credited tests do not block this specification. They become cleanup-worthy only when they freeze private implementation, duplicate maintenance, or replace capable evidence.

Overlap receipt

  • Terms searched: test sufficiency, test quality rubric, testing architecture, verification maps, Behavior rubric, B1–B16, AST-029, mutation-sensitive evidence, and no-test boundaries.
  • Canonical owner/path checked: current component, module, family, architecture, system, design, and theme records plus docs/specs/AST-029/spec.md.
  • Open work inspected: open specification and component PRs touching verification maps, tests, docs, visual evidence, AST-029, and rubric migration.
  • Result: no current record or other open PR owns the shared sufficiency/no-test boundary. AST-029 explicitly excludes the general component-test quality rubric; local verification maps own their promises and evidence relationships, not the cross-component sufficiency rule.

Future B1–B16 migration handoff

A separate rubric activation after this record lands must retire B1–B16 as generic product-behavior IDs, preserve Behavior scoring from applicable authority, score Testing from this record, reserve B12, move B16 to PR/audit-run evidence, and bump the rubric minor version.

This PR does not edit the wiki rubric, scoring code, audit prompt, tests, tooling, or CI.

Validation

  • node scripts/check-knowledge.mjs
  • pnpm check:sync
  • repository pre-commit checks
  • Prettier and git diff --check
  • 18-case authority/equivalence calibration: 0 FP / 0 FN after correction
  • 16-case no-test boundary calibration: 0 FP / 0 FN after correction
  • 12-case current-repo sample: 0 FP / 0 FN after correction
  • explicit system/design/theme authority-selection audit: 0 FP / 0 FN after correction

Length rationale

The 331-line record intentionally exceeds the 200-line soft ceiling because it contains three durable review surfaces that would otherwise become ambiguous or drift apart: the no-new-test decision table, the explicit B1–B16 rubric migration handoff, and the mutation/failure verification matrix. Removing them would move the actual decision rules back into PR context or future implementation inference.

One architecture record only. No implementation, tests, tooling, wiki, audit-table, rubric, or Changeset edits.

@vercel

vercel Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
astryx Ready Ready Preview Sep 9, 2026 6:59am UTC

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 8, 2026
github-actions Bot added a commit that referenced this pull request Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

PR Analysis Report

Preview availability: Storybook and Sandbox were not published for this CI run.

No new or modified components detected.

Bundle Size Summary

No component packages changed.

Accessibility Audit

Status: No accessibility violations detected.


Generated by PR Enrichment workflow | View full report

@cixzhang
cixzhang force-pushed the spec-component-test-sufficiency branch from 9e5bc20 to 695cf21 Compare September 8, 2026 00:48
@cixzhang

cixzhang commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Exact-head current-code audit at 695cf212a704708961b89a826b24ad251cc5d0ad: 0 false positives, 0 false negatives, 17 correct classifications, 1 untestable across the same bounded 18-case matrix. The three prior fixes hold: established-public-interface evidence does not create authority; equivalence is observable rather than dependent on private implementation identity; and Behavior, Testing sufficiency, and PR-only red/green evidence remain separate. #6116, #6122, and #6129 are integrated ancestors. No human decision remains from the audit.

github-actions Bot added a commit that referenced this pull request Sep 8, 2026
@cixzhang
cixzhang force-pushed the spec-component-test-sufficiency branch from 695cf21 to 23b0639 Compare September 8, 2026 16:13
github-actions Bot added a commit that referenced this pull request Sep 8, 2026
@cixzhang
cixzhang force-pushed the spec-component-test-sufficiency branch from 23b0639 to 54d4132 Compare September 9, 2026 06:39
@cixzhang
cixzhang marked this pull request as ready for review September 9, 2026 06:39
@github-actions github-actions Bot added the needs:spec-owner-review Current knowledge records await owner approval label Sep 9, 2026

@cixzhang cixzhang left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Semantic verdict: request changes

A current cross-component requirement can fall out of this test plan: the authority lists omit system, design, and theme records. Please cover every applicable current record kind and rerun the authority matrix with a system-owned case such as AST-005’s navigation-sink requirement.

The PR summary also needs the required overlap-search receipt (terms, canonical owner/path, open work inspected) and the rationale for this 331-line record.

[Reviewed by Robohands]

@cixzhang
cixzhang force-pushed the spec-component-test-sufficiency branch from 54d4132 to b232e5d Compare September 9, 2026 06:54
@cixzhang

cixzhang commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

/approve-spec b232e5d

@github-actions github-actions Bot removed the needs:spec-owner-review Current knowledge records await owner approval label Sep 9, 2026

@cixzhang cixzhang left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Semantic verdict: approve

Thanks, this now covers every authority owner, keeps the AST-029 and B1–B16 boundaries, and clears the prior calibration gap.

[Reviewed by Robohands]

@cixzhang
cixzhang enabled auto-merge (squash) September 9, 2026 07:01
@cixzhang
cixzhang merged commit 5123978 into main Sep 9, 2026
30 of 31 checks passed
@github-actions
github-actions Bot deleted the spec-component-test-sufficiency branch September 10, 2026 06:23

This branch was successfully deployed

1 active deployment
Preview — b232e5da Deployed Sep 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant