Skip to content

Add audit manifest contract - #7

Open
bhdicaire wants to merge 1 commit into
mainfrom
audit-manifest-json
Open

Add audit manifest contract#7
bhdicaire wants to merge 1 commit into
mainfrom
audit-manifest-json

Conversation

@bhdicaire

Copy link
Copy Markdown
Member

Summary

  • make manifest.json part of the lean audit root layout
  • add a canonical schema 1.0.0 manifest template for report identity, subject, evidence, execution, results, and relationships
  • instruct synthesis to update manifest.json with only evidence-supported values
  • extend validate_audit_structure.py with manifest checks for top-level keys, placeholders, full Git SHAs, reviewer shape/status, and conclusion IDs/confidence/severity
  • update README and contract tests

Impact

  • wgo-audit/code will emit the manifest contract in generated audit bundles
  • wgo-audit/reports can store this manifest now, but its ingest/index scripts still need a follow-up to read subject.id/evidence.cutoff/audit.type/results.conclusions from schema 1.0.0 instead of legacy asset/evidenceCutoff/conclusions fields
  • wgo-audit/website can later consume the same normalized contract for publishing and comparisons

Tests

  • python3 -m unittest discover -s tests -p 'test_*.py'

@Patrick-Loves-Espresso

Copy link
Copy Markdown
Contributor

The namespaced structure is much cleaner than the original flat manifest, and capturing source SHAs, reviewer versions, generator provenance, conclusions, and audit relationships will materially improve repeatability and downstream publishing.

I ran the PR’s exact commit: all 72 tests pass.

Before merging, I recommend tightening the implementation in these areas:

  1. Keep production lean and avoid a handwritten Python schema engine.
    Please move the JSON template out of control-templates.md into a standalone machine-readable template. If formal validation is required, prefer a static JSON Schema or validation in the downstream reports ingestion process rather than adding substantial custom logic to validate_audit_structure.py.

    Why: WGO’s production behavior should remain instruction-focused and lean. Maintaining the same contract independently in a Markdown example, Python validator, tests, and downstream consumer creates multiple places that can diverge.

  2. Populate everything known during onboarding.
    Subject identity, mode, depth, cutoff, platform/model, selected reviewer IDs and versions, accepted version variances, and baseline are already known after onboarding approval. Populate them immediately. Synthesis should update only final statuses, results, generated time, and relationships that depend on completed work.

    Why: An interrupted audit should still have a useful, repeatable manifest. Recording known values as unknown unnecessarily loses information.

  3. Define the nested machine contracts, not only the top-level keys.
    Please define required fields and types for at least:

    • sources: kind, repository or URL, ref, and full commit SHA;
    • reviewers: ID, version, and status;
    • conclusions: stable ID, concise summary, severity, effort, confidence, originating reviewer, and report link;
    • relationship references: one consistent report ID or object shape.

    Reuse WGO’s existing checklist states instead of introducing partial, failed, and not-applicable, or document an exact mapping. Align severity with WGO’s High/Medium/Low contract and include its S/M/L effort classification. Also define date/time formats and enforce the supported schema version.

    Why: A machine-readable file is valuable only when consumers can rely on nested field shapes and controlled values. Top-level consistency alone is insufficient.

  4. Remove or reject unresolved <…> placeholders.
    The template currently contains values such as <subject-id>, but the validator rejects only TODO and TBD.

    Why: An untouched template can currently pass validation while containing invalid production identifiers.

  5. Provide backward compatibility for existing audit roots.
    Making manifest.json unconditionally part of BASE_REQUIRED causes every audit created before this schema to fail structural validation. Please add an explicit migration/backfill path or preserve legacy-root validity.

    Why: Repeatability requires old audits to remain readable and comparable after WGO evolves.

  6. Clarify the relationship fields.
    Under WGO’s current single-baseline model, baseline and comparesTo represent the same relationship. Please remove comparesTo for schema 1.0, or define baseline as the primary comparison and comparesTo as a broader multi-report set.

    Why: Intentional denormalization is useful, but two fields with unclear overlapping meaning invite inconsistent producer and consumer behavior.

I would not remove useful duplication such as report title, subject identity, reviewer entries, generator provenance, headline, or normalized conclusions. The manifest should let tools identify and summarize an audit without parsing Markdown. That duplication is appropriate; the important requirement is that it remain concise, normalized, and deterministically derived from the canonical audit records.

With those adjustments, the overall model looks suitable for a schema 1.0 contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants