Skip to content

Add standard CodeceptJS conformance tests - #27

Merged
Ma11hewThomas merged 2 commits into
mainfrom
add-codeceptjs-conformance
Oct 24, 2026
Merged

Ma11hewThomas merged 2 commits into
mainfrom
add-codeceptjs-conformance

Conversation

@Ma11hewThomas

@Ma11hewThomas Ma11hewThomas commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Add CTRF 0.2.0 unit and fresh installed-CodeceptJS assessments, independent checker controls and native regression tests. Real plugin/runtime scenarios verify outcomes, metadata, retries and worker execution with independent native observations. CI publishes three separate ordinary, contract and checker scopes.

Validation: npm run all and npm run test:standard pass on Node 24.21.0, including existing integration and packed ESM/CommonJS/declaration consumers. Unsupported optional capabilities remain visible with reasons.

CodeceptJS 4.2.2 supplies native actor-step names, allowing full report emission. The obsolete #26 emission exceptions are replaced with positive/corruption regressions: each contract level now has 83 passed, 102 unsupported skips and no expected or unexpected failures. Native success-status normalization remains a separate follow-up. A fresh peer consumer without the checkout overrides reports four moderate propagated UUID findings; the checkout audits below include its tested overrides.

Dependency refresh: @biomejs/biome 2.5.15 → 2.6.2; @vitest/coverage-v8 5.0.3 → 5.0.5; codeceptjs 4.2.0 → 4.2.2; tsup 8.5.1 → 9.0.0; vitest 5.0.3 → 5.0.5; override uuid 11.1.1 → 14.0.3; override diff 8.0.3 → 9.0.0; override serialize-javascript 7.1.2 → 7.1.4. Refreshed compatible transitive resolutions and retained the reviewed shared conformance snapshot. Biome keeps the existing Markdown formatting policy.

Hosted verification: CI passes on Node 22/24/26 at the updated head. Local build, existing tests and both conformance levels pass; the three scoped comments updated in place and per-runtime artifacts are retained.

Dependency audit: full tree zero findings; production zero findings.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Reporter conformance — contract families (Node 24)

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
370 166 0 204 0 0 20.3s

Suites

166 passed, 0 failed, and 204 other

Suite Passed Failed Other Duration
✅ Unit conformance > Schema properties
        ✅ unit / conformance/property/reportFormat
        ✅ unit / conformance/property/specVersion
        ✅ unit / conformance/property/reportId
        ✅ unit / conformance/property/runId
        ✅ unit / conformance/property/timestamp
        ✅ unit / conformance/property/generatedBy
        ⏭️ unit / conformance/property/extra
        ✅ unit / conformance/property/results
        ✅ unit / conformance/property/results.tool
        ✅ unit / conformance/property/results.tool.name
        ⏭️ unit / conformance/property/results.tool.version
        ⏭️ unit / conformance/property/results.tool.extra
        ✅ unit / conformance/property/results.summary
        ✅ unit / conformance/property/results.summary.tests
        ✅ unit / conformance/property/results.summary.passed
        ✅ unit / conformance/property/results.summary.failed
        ✅ unit / conformance/property/results.summary.skipped
        ✅ unit / conformance/property/results.summary.pending
        ✅ unit / conformance/property/results.summary.other
        ⏭️ unit / conformance/property/results.summary.flaky
        ⏭️ unit / conformance/property/results.summary.suites
        ✅ unit / conformance/property/results.summary.start
        ✅ unit / conformance/property/results.summary.stop
        ⏭️ unit / conformance/property/results.summary.duration
        ⏭️ unit / conformance/property/results.summary.extra
        ✅ unit / conformance/property/results.tests
        ⏭️ unit / conformance/property/results.tests[].id
        ✅ unit / conformance/property/results.tests[].testId
        ✅ unit / conformance/property/results.tests[].executionId
        ✅ unit / conformance/property/results.tests[].attemptId
        ✅ unit / conformance/property/results.tests[].name
        ✅ unit / conformance/property/results.tests[].status
        ✅ unit / conformance/property/results.tests[].duration
        ✅ unit / conformance/property/results.tests[].start
        ✅ unit / conformance/property/results.tests[].stop
        ✅ unit / conformance/property/results.tests[].suite
        ✅ unit / conformance/property/results.tests[].message
        ✅ unit / conformance/property/results.tests[].trace
        ⏭️ unit / conformance/property/results.tests[].snippet
        ⏭️ unit / conformance/property/results.tests[].ai
        ⏭️ unit / conformance/property/results.tests[].line
        ✅ unit / conformance/property/results.tests[].rawStatus
        ✅ unit / conformance/property/results.tests[].tags
        ⏭️ unit / conformance/property/results.tests[].labels
        ✅ unit / conformance/property/results.tests[].type
        ✅ unit / conformance/property/results.tests[].filePath
        ✅ unit / conformance/property/results.tests[].retries
        ✅ unit / conformance/property/results.tests[].retryAttempts
        ✅ unit / conformance/property/results.tests[].retryAttempts[].attempt
        ✅ unit / conformance/property/results.tests[].retryAttempts[].attemptId
        ✅ unit / conformance/property/results.tests[].retryAttempts[].status
        ✅ unit / conformance/property/results.tests[].retryAttempts[].duration
        ✅ unit / conformance/property/results.tests[].retryAttempts[].message
        ✅ unit / conformance/property/results.tests[].retryAttempts[].trace
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].line
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].snippet
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].stdout
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].stderr
        ✅ unit / conformance/property/results.tests[].retryAttempts[].start
        ✅ unit / conformance/property/results.tests[].retryAttempts[].stop
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].attachments
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].attachments[].attachmentId
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].attachments[].name
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].attachments[].contentType
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].attachments[].path
        ⏭️ unit / conformance/property/results.tests[].retryAttempts[].attachments[].extra
        ✅ unit / conformance/property/results.tests[].retryAttempts[].extra
        ✅ unit / conformance/property/results.tests[].flaky
        ⏭️ unit / conformance/property/results.tests[].stdout
        ⏭️ unit / conformance/property/results.tests[].stderr
        ⏭️ unit / conformance/property/results.tests[].threadId
        ⏭️ unit / conformance/property/results.tests[].browser
        ⏭️ unit / conformance/property/results.tests[].device
        ⏭️ unit / conformance/property/results.tests[].screenshot
        ⏭️ unit / conformance/property/results.tests[].attachments
        ⏭️ unit / conformance/property/results.tests[].attachments[].attachmentId
        ⏭️ unit / conformance/property/results.tests[].attachments[].name
        ⏭️ unit / conformance/property/results.tests[].attachments[].contentType
        ⏭️ unit / conformance/property/results.tests[].attachments[].path
        ⏭️ unit / conformance/property/results.tests[].attachments[].extra
        ⏭️ unit / conformance/property/results.tests[].parameters
        ✅ unit / conformance/property/results.tests[].steps
        ✅ unit / conformance/property/results.tests[].steps[].name
        ✅ unit / conformance/property/results.tests[].steps[].status
        ⏭️ unit / conformance/property/results.tests[].steps[].extra
        ⏭️ unit / conformance/property/results.tests[].insights
        ⏭️ unit / conformance/property/results.tests[].insights.passRate
        ⏭️ unit / conformance/property/results.tests[].insights.passRate.current
        ⏭️ unit / conformance/property/results.tests[].insights.passRate.baseline
        ⏭️ unit / conformance/property/results.tests[].insights.passRate.change
        ⏭️ unit / conformance/property/results.tests[].insights.failRate
        ⏭️ unit / conformance/property/results.tests[].insights.failRate.current
        ⏭️ unit / conformance/property/results.tests[].insights.failRate.baseline
        ⏭️ unit / conformance/property/results.tests[].insights.failRate.change
        ⏭️ unit / conformance/property/results.tests[].insights.flakyRate
        ⏭️ unit / conformance/property/results.tests[].insights.flakyRate.current
        ⏭️ unit / conformance/property/results.tests[].insights.flakyRate.baseline
        ⏭️ unit / conformance/property/results.tests[].insights.flakyRate.change
        ⏭️ unit / conformance/property/results.tests[].insights.averageTestDuration
        ⏭️ unit / conformance/property/results.tests[].insights.averageTestDuration.current
        ⏭️ unit / conformance/property/results.tests[].insights.averageTestDuration.baseline
        ⏭️ unit / conformance/property/results.tests[].insights.averageTestDuration.change
        ⏭️ unit / conformance/property/results.tests[].insights.p95TestDuration
        ⏭️ unit / conformance/property/results.tests[].insights.p95TestDuration.current
        ⏭️ unit / conformance/property/results.tests[].insights.p95TestDuration.baseline
        ⏭️ unit / conformance/property/results.tests[].insights.p95TestDuration.change
        ⏭️ unit / conformance/property/results.tests[].insights.executedInRuns
        ⏭️ unit / conformance/property/results.tests[].insights.extra
        ✅ unit / conformance/property/results.tests[].extra
        ✅ unit / conformance/property/results.environment
        ⏭️ unit / conformance/property/results.environment.reportName
        ⏭️ unit / conformance/property/results.environment.appName
        ✅ unit / conformance/property/results.environment.appVersion
        ⏭️ unit / conformance/property/results.environment.buildId
        ✅ unit / conformance/property/results.environment.buildName
        ✅ unit / conformance/property/results.environment.buildNumber
        ✅ unit / conformance/property/results.environment.buildUrl
        ✅ unit / conformance/property/results.environment.repositoryName
        ✅ unit / conformance/property/results.environment.repositoryUrl
        ✅ unit / conformance/property/results.environment.commit
        ✅ unit / conformance/property/results.environment.branchName
        ✅ unit / conformance/property/results.environment.osPlatform
        ✅ unit / conformance/property/results.environment.osRelease
        ✅ unit / conformance/property/results.environment.osVersion
        ✅ unit / conformance/property/results.environment.testEnvironment
        ✅ unit / conformance/property/results.environment.shardId
        ⏭️ unit / conformance/property/results.environment.healthy
        ✅ unit / conformance/property/results.environment.extra
        ⏭️ unit / conformance/property/results.extra
        ⏭️ unit / conformance/property/insights
        ⏭️ unit / conformance/property/insights.passRate
        ⏭️ unit / conformance/property/insights.passRate.current
        ⏭️ unit / conformance/property/insights.passRate.baseline
        ⏭️ unit / conformance/property/insights.passRate.change
        ⏭️ unit / conformance/property/insights.failRate
        ⏭️ unit / conformance/property/insights.failRate.current
        ⏭️ unit / conformance/property/insights.failRate.baseline
        ⏭️ unit / conformance/property/insights.failRate.change
        ⏭️ unit / conformance/property/insights.flakyRate
        ⏭️ unit / conformance/property/insights.flakyRate.current
        ⏭️ unit / conformance/property/insights.flakyRate.baseline
        ⏭️ unit / conformance/property/insights.flakyRate.change
        ⏭️ unit / conformance/property/insights.averageRunDuration
        ⏭️ unit / conformance/property/insights.averageRunDuration.current
        ⏭️ unit / conformance/property/insights.averageRunDuration.baseline
        ⏭️ unit / conformance/property/insights.averageRunDuration.change
        ⏭️ unit / conformance/property/insights.p95RunDuration
        ⏭️ unit / conformance/property/insights.p95RunDuration.current
        ⏭️ unit / conformance/property/insights.p95RunDuration.baseline
        ⏭️ unit / conformance/property/insights.p95RunDuration.change
        ⏭️ unit / conformance/property/insights.averageTestDuration
        ⏭️ unit / conformance/property/insights.averageTestDuration.current
        ⏭️ unit / conformance/property/insights.averageTestDuration.baseline
        ⏭️ unit / conformance/property/insights.averageTestDuration.change
        ⏭️ unit / conformance/property/insights.runsAnalyzed
        ⏭️ unit / conformance/property/insights.extra
        ⏭️ unit / conformance/property/baseline
        ⏭️ unit / conformance/property/baseline.reportId
        ⏭️ unit / conformance/property/baseline.timestamp
        ⏭️ unit / conformance/property/baseline.source
        ⏭️ unit / conformance/property/baseline.buildNumber
        ⏭️ unit / conformance/property/baseline.buildName
        ⏭️ unit / conformance/property/baseline.buildUrl
        ⏭️ unit / conformance/property/baseline.commit
        ⏭️ unit / conformance/property/baseline.extra
✅ 65 ❌ 0 ⏭️ 100 40ms
✅ Unit conformance > Specification relationships
        ✅ unit / conformance/report/utf8-without-bom
        ✅ unit / conformance/report/single-json-object
        ✅ unit / conformance/report/declares-supported-contract
        ✅ unit / conformance/report/summary-matches-final-results
        ⏭️ unit / conformance/report/summary-matches-flaky-results
        ✅ unit / conformance/report/orders-start-and-stop
        ✅ unit / conformance/report/history-requires-retry-count
        ✅ unit / conformance/report/positive-retries-require-history
        ✅ unit / conformance/report/zero-retries-omit-history
        ✅ unit / conformance/report/retry-count-matches-history-length
        ✅ unit / conformance/report/history-orders-contiguous-attempts
        ✅ unit / conformance/report/history-excludes-final-attempt
        ✅ unit / conformance/report/flaky-matches-attempt-outcomes
        ✅ unit / conformance/report/case-identity-stable-across-runs
        ✅ unit / conformance/report/execution-identity-distinguishes-executions
        ✅ unit / conformance/report/attempt-identity-scoped-to-execution
        ⏭️ unit / conformance/report/attachment-identity-scoped-to-parent
        ✅ unit / conformance/report/document-identity-follows-emission
        ✅ unit / conformance/report/coordinates-run-and-shard-identity
        ✅ unit / conformance/report/respects-reserved-extension-ownership
✅ 18 ❌ 0 ⏭️ 2 5ms
✅ Integration conformance > Schema properties
        ✅ integration / conformance/property/reportFormat
        ✅ integration / conformance/property/specVersion
        ✅ integration / conformance/property/reportId
        ✅ integration / conformance/property/runId
        ✅ integration / conformance/property/timestamp
        ✅ integration / conformance/property/generatedBy
        ⏭️ integration / conformance/property/extra
        ✅ integration / conformance/property/results
        ✅ integration / conformance/property/results.tool
        ✅ integration / conformance/property/results.tool.name
        ⏭️ integration / conformance/property/results.tool.version
        ⏭️ integration / conformance/property/results.tool.extra
        ✅ integration / conformance/property/results.summary
        ✅ integration / conformance/property/results.summary.tests
        ✅ integration / conformance/property/results.summary.passed
        ✅ integration / conformance/property/results.summary.failed
        ✅ integration / conformance/property/results.summary.skipped
        ✅ integration / conformance/property/results.summary.pending
        ✅ integration / conformance/property/results.summary.other
        ⏭️ integration / conformance/property/results.summary.flaky
        ⏭️ integration / conformance/property/results.summary.suites
        ✅ integration / conformance/property/results.summary.start
        ✅ integration / conformance/property/results.summary.stop
        ⏭️ integration / conformance/property/results.summary.duration
        ⏭️ integration / conformance/property/results.summary.extra
        ✅ integration / conformance/property/results.tests
        ⏭️ integration / conformance/property/results.tests[].id
        ✅ integration / conformance/property/results.tests[].testId
        ✅ integration / conformance/property/results.tests[].executionId
        ✅ integration / conformance/property/results.tests[].attemptId
        ✅ integration / conformance/property/results.tests[].name
        ✅ integration / conformance/property/results.tests[].status
        ✅ integration / conformance/property/results.tests[].duration
        ✅ integration / conformance/property/results.tests[].start
        ✅ integration / conformance/property/results.tests[].stop
        ✅ integration / conformance/property/results.tests[].suite
        ✅ integration / conformance/property/results.tests[].message
        ✅ integration / conformance/property/results.tests[].trace
        ⏭️ integration / conformance/property/results.tests[].snippet
        ⏭️ integration / conformance/property/results.tests[].ai
        ⏭️ integration / conformance/property/results.tests[].line
        ✅ integration / conformance/property/results.tests[].rawStatus
        ✅ integration / conformance/property/results.tests[].tags
        ⏭️ integration / conformance/property/results.tests[].labels
        ✅ integration / conformance/property/results.tests[].type
        ✅ integration / conformance/property/results.tests[].filePath
        ✅ integration / conformance/property/results.tests[].retries
        ✅ integration / conformance/property/results.tests[].retryAttempts
        ✅ integration / conformance/property/results.tests[].retryAttempts[].attempt
        ✅ integration / conformance/property/results.tests[].retryAttempts[].attemptId
        ✅ integration / conformance/property/results.tests[].retryAttempts[].status
        ✅ integration / conformance/property/results.tests[].retryAttempts[].duration
        ✅ integration / conformance/property/results.tests[].retryAttempts[].message
        ✅ integration / conformance/property/results.tests[].retryAttempts[].trace
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].line
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].snippet
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].stdout
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].stderr
        ✅ integration / conformance/property/results.tests[].retryAttempts[].start
        ✅ integration / conformance/property/results.tests[].retryAttempts[].stop
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].attachments
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].attachments[].attachmentId
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].attachments[].name
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].attachments[].contentType
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].attachments[].path
        ⏭️ integration / conformance/property/results.tests[].retryAttempts[].attachments[].extra
        ✅ integration / conformance/property/results.tests[].retryAttempts[].extra
        ✅ integration / conformance/property/results.tests[].flaky
        ⏭️ integration / conformance/property/results.tests[].stdout
        ⏭️ integration / conformance/property/results.tests[].stderr
        ⏭️ integration / conformance/property/results.tests[].threadId
        ⏭️ integration / conformance/property/results.tests[].browser
        ⏭️ integration / conformance/property/results.tests[].device
        ⏭️ integration / conformance/property/results.tests[].screenshot
        ⏭️ integration / conformance/property/results.tests[].attachments
        ⏭️ integration / conformance/property/results.tests[].attachments[].attachmentId
        ⏭️ integration / conformance/property/results.tests[].attachments[].name
        ⏭️ integration / conformance/property/results.tests[].attachments[].contentType
        ⏭️ integration / conformance/property/results.tests[].attachments[].path
        ⏭️ integration / conformance/property/results.tests[].attachments[].extra
        ⏭️ integration / conformance/property/results.tests[].parameters
        ✅ integration / conformance/property/results.tests[].steps
        ✅ integration / conformance/property/results.tests[].steps[].name
        ✅ integration / conformance/property/results.tests[].steps[].status
        ⏭️ integration / conformance/property/results.tests[].steps[].extra
        ⏭️ integration / conformance/property/results.tests[].insights
        ⏭️ integration / conformance/property/results.tests[].insights.passRate
        ⏭️ integration / conformance/property/results.tests[].insights.passRate.current
        ⏭️ integration / conformance/property/results.tests[].insights.passRate.baseline
        ⏭️ integration / conformance/property/results.tests[].insights.passRate.change
        ⏭️ integration / conformance/property/results.tests[].insights.failRate
        ⏭️ integration / conformance/property/results.tests[].insights.failRate.current
        ⏭️ integration / conformance/property/results.tests[].insights.failRate.baseline
        ⏭️ integration / conformance/property/results.tests[].insights.failRate.change
        ⏭️ integration / conformance/property/results.tests[].insights.flakyRate
        ⏭️ integration / conformance/property/results.tests[].insights.flakyRate.current
        ⏭️ integration / conformance/property/results.tests[].insights.flakyRate.baseline
        ⏭️ integration / conformance/property/results.tests[].insights.flakyRate.change
        ⏭️ integration / conformance/property/results.tests[].insights.averageTestDuration
        ⏭️ integration / conformance/property/results.tests[].insights.averageTestDuration.current
        ⏭️ integration / conformance/property/results.tests[].insights.averageTestDuration.baseline
        ⏭️ integration / conformance/property/results.tests[].insights.averageTestDuration.change
        ⏭️ integration / conformance/property/results.tests[].insights.p95TestDuration
        ⏭️ integration / conformance/property/results.tests[].insights.p95TestDuration.current
        ⏭️ integration / conformance/property/results.tests[].insights.p95TestDuration.baseline
        ⏭️ integration / conformance/property/results.tests[].insights.p95TestDuration.change
        ⏭️ integration / conformance/property/results.tests[].insights.executedInRuns
        ⏭️ integration / conformance/property/results.tests[].insights.extra
        ✅ integration / conformance/property/results.tests[].extra
        ✅ integration / conformance/property/results.environment
        ⏭️ integration / conformance/property/results.environment.reportName
        ⏭️ integration / conformance/property/results.environment.appName
        ✅ integration / conformance/property/results.environment.appVersion
        ⏭️ integration / conformance/property/results.environment.buildId
        ✅ integration / conformance/property/results.environment.buildName
        ✅ integration / conformance/property/results.environment.buildNumber
        ✅ integration / conformance/property/results.environment.buildUrl
        ✅ integration / conformance/property/results.environment.repositoryName
        ✅ integration / conformance/property/results.environment.repositoryUrl
        ✅ integration / conformance/property/results.environment.commit
        ✅ integration / conformance/property/results.environment.branchName
        ✅ integration / conformance/property/results.environment.osPlatform
        ✅ integration / conformance/property/results.environment.osRelease
        ✅ integration / conformance/property/results.environment.osVersion
        ✅ integration / conformance/property/results.environment.testEnvironment
        ✅ integration / conformance/property/results.environment.shardId
        ⏭️ integration / conformance/property/results.environment.healthy
        ✅ integration / conformance/property/results.environment.extra
        ⏭️ integration / conformance/property/results.extra
        ⏭️ integration / conformance/property/insights
        ⏭️ integration / conformance/property/insights.passRate
        ⏭️ integration / conformance/property/insights.passRate.current
        ⏭️ integration / conformance/property/insights.passRate.baseline
        ⏭️ integration / conformance/property/insights.passRate.change
        ⏭️ integration / conformance/property/insights.failRate
        ⏭️ integration / conformance/property/insights.failRate.current
        ⏭️ integration / conformance/property/insights.failRate.baseline
        ⏭️ integration / conformance/property/insights.failRate.change
        ⏭️ integration / conformance/property/insights.flakyRate
        ⏭️ integration / conformance/property/insights.flakyRate.current
        ⏭️ integration / conformance/property/insights.flakyRate.baseline
        ⏭️ integration / conformance/property/insights.flakyRate.change
        ⏭️ integration / conformance/property/insights.averageRunDuration
        ⏭️ integration / conformance/property/insights.averageRunDuration.current
        ⏭️ integration / conformance/property/insights.averageRunDuration.baseline
        ⏭️ integration / conformance/property/insights.averageRunDuration.change
        ⏭️ integration / conformance/property/insights.p95RunDuration
        ⏭️ integration / conformance/property/insights.p95RunDuration.current
        ⏭️ integration / conformance/property/insights.p95RunDuration.baseline
        ⏭️ integration / conformance/property/insights.p95RunDuration.change
        ⏭️ integration / conformance/property/insights.averageTestDuration
        ⏭️ integration / conformance/property/insights.averageTestDuration.current
        ⏭️ integration / conformance/property/insights.averageTestDuration.baseline
        ⏭️ integration / conformance/property/insights.averageTestDuration.change
        ⏭️ integration / conformance/property/insights.runsAnalyzed
        ⏭️ integration / conformance/property/insights.extra
        ⏭️ integration / conformance/property/baseline
        ⏭️ integration / conformance/property/baseline.reportId
        ⏭️ integration / conformance/property/baseline.timestamp
        ⏭️ integration / conformance/property/baseline.source
        ⏭️ integration / conformance/property/baseline.buildNumber
        ⏭️ integration / conformance/property/baseline.buildName
        ⏭️ integration / conformance/property/baseline.buildUrl
        ⏭️ integration / conformance/property/baseline.commit
        ⏭️ integration / conformance/property/baseline.extra
✅ 65 ❌ 0 ⏭️ 100 66ms
✅ Integration conformance > Specification relationships
        ✅ integration / conformance/report/utf8-without-bom
        ✅ integration / conformance/report/single-json-object
        ✅ integration / conformance/report/declares-supported-contract
        ✅ integration / conformance/report/summary-matches-final-results
        ⏭️ integration / conformance/report/summary-matches-flaky-results
        ✅ integration / conformance/report/orders-start-and-stop
        ✅ integration / conformance/report/history-requires-retry-count
        ✅ integration / conformance/report/positive-retries-require-history
        ✅ integration / conformance/report/zero-retries-omit-history
        ✅ integration / conformance/report/retry-count-matches-history-length
        ✅ integration / conformance/report/history-orders-contiguous-attempts
        ✅ integration / conformance/report/history-excludes-final-attempt
        ✅ integration / conformance/report/flaky-matches-attempt-outcomes
        ✅ integration / conformance/report/case-identity-stable-across-runs
        ✅ integration / conformance/report/execution-identity-distinguishes-executions
        ✅ integration / conformance/report/attempt-identity-scoped-to-execution
        ⏭️ integration / conformance/report/attachment-identity-scoped-to-parent
        ✅ integration / conformance/report/document-identity-follows-emission
        ✅ integration / conformance/report/coordinates-run-and-shard-identity
        ✅ integration / conformance/report/respects-reserved-extension-ownership
✅ 18 ❌ 0 ⏭️ 2 6ms

Github Test Reporter by CTRF 💚

🔄 This comment has been updated

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Shared checker controls (Node 24)

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
34 34 0 0 0 0 206ms

Suites

34 passed, 0 failed, and 0 other

Suite Passed Failed Other Duration
✅ Shared checker controls > schema
        ✅ checker / checker/schema/catalogue-covers-expanded-properties-and-relationships
        ✅ checker / checker/schema/accepts-independent-valid-document
        ✅ checker / checker/schema/rejects-unknown-property
        ✅ checker / checker/schema/rejects-wrong-type
        ✅ checker / checker/schema/rejects-missing-required-field
        ✅ checker / checker/schema/requires-positive-evidence
        ✅ checker / checker/schema/required-child-is-checked-when-parent-exists
        ✅ checker / checker/schema/capabilities-require-entry-and-reason
        ✅ checker / checker/schema/unsupported-declaration-cannot-hide-emitted-invalid-field
        ✅ checker / checker/schema/shared-snapshot-matches-declared-files
✅ 10 ❌ 0 ⏭️ 0 10ms
✅ Shared checker controls > specification
        ✅ checker / checker/specification/accepts-independent-retry-history
        ✅ checker / checker/specification/rejects-utf8-without-bom-mutation
        ✅ checker / checker/specification/rejects-summary-matches-final-results-mutation
        ✅ checker / checker/specification/rejects-history-orders-contiguous-attempts-mutation
        ✅ checker / checker/specification/rejects-retry-count-matches-history-length-mutation
        ✅ checker / checker/specification/rejects-history-excludes-final-attempt-mutation
        ✅ checker / checker/specification/rejects-flaky-matches-attempt-outcomes-mutation
        ✅ checker / checker/specification/rejects-attempt-identity-scoped-to-execution-mutation
        ✅ checker / checker/specification/rejects-respects-reserved-extension-ownership-mutation
        ✅ checker / checker/specification/rejects-source-duration-unit-mismatch
        ✅ checker / checker/specification/missing-contextual-evidence-fails
        ✅ checker / checker/specification/timing-recommendation-warns
        ✅ checker / checker/specification/total-count-recommendation-warns-but-bounds-fail
✅ 13 ❌ 0 ⏭️ 0 6ms
✅ Shared checker controls > expected-failure
        ✅ checker / checker/expected-failure/accepts-only-recorded-mismatch
        ✅ checker / checker/expected-failure/rejects-unexpected-pass
        ✅ checker / checker/expected-failure/rejects-partial-fix
        ✅ checker / checker/expected-failure/rejects-different-value
        ✅ checker / checker/expected-failure/rejects-missing-sample
        ✅ checker / checker/expected-failure/rejects-duplicate-sample
        ✅ checker / checker/expected-failure/rejects-unrecorded-broken-sample
        ✅ checker / checker/expected-failure/rejects-infrastructure-error
        ✅ checker / checker/expected-failure/does-not-apply-at-unrecorded-level
        ✅ checker / checker/expected-failure/allows-optional-absence-outside-defect-samples
        ✅ checker / checker/expected-failure/rejects-unrelated-error-in-later-sample
✅ 11 ❌ 0 ⏭️ 0 1ms

Github Test Reporter by CTRF 💚

🔄 This comment has been updated

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Reporter tests (Node 24)

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
37    ↓12 37    ↓12 0 0 0 0 1.2s    ↓42ms

Test Results

passed 37 passed ↓12

Details

tests 37 tests ↓12
clock 1.2s ↓42ms
tool vitest
build CI arrow-right test-build link #31
pull-request Add standard CodeceptJS conformance tests link #27

Previous Results

Build 🏗️ Result 🧪 Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
#31 ✅ 37 37 0 0 0 0 1.2s
#30 ✅ 49 49 0 0 0 0 1.2s

Insights

Average Tests per Run Total Flaky Tests Total Failed Slowest Test (p95)
43 0 0 196ms

Tests

View All Tests
Test Name Status Flaky Duration
reporter / conformance-tooling.test.ts / Codecept tooling / reviewed shared snapshot remains exact ✅ 3ms
reporter / conformance-tooling.test.ts / Codecept tooling / capabilities cover exactly the shared inventory ✅ 2ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/report/single-json-object accepts native serialized output ✅ 9ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/report/single-json-object rejects corrupted native output ✅ 14ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/property/results.tests[].steps accepts native serialized output ✅ 4ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/property/results.tests[].steps rejects corrupted native output ✅ 4ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/property/results.tests[].steps[].name accepts native serialized output ✅ 2ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/property/results.tests[].steps[].name rejects corrupted native output ✅ 2ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/property/results.tests[].steps[].status accepts native serialized output ✅ 2ms
reporter / conformance-tooling.test.ts / Codecept actor steps / conformance/property/results.tests[].steps[].status rejects corrupted native output ✅ 2ms
reporter / conformance-tooling.test.ts / Codecept oracle / catches corrupted structured runtime metadata ✅ 2ms
reporter / conformance-tooling.test.ts / Codecept oracle / catches inclusion of the final attempt in history ✅ 1ms
reporter / conformance-tooling.test.ts / Codecept oracle / catches a schema-valid native status mapping change ✅ 2ms
reporter / conformance-tooling.test.ts / Codecept actor steps / retains actual bytes and native name evidence ✅ 1ms
reporter / conformance-tooling.test.ts / Reporter results / completed records preserve pass, failure, pending and skip ✅ 1ms
reporter / conformance-tooling.test.ts / Reporter results / module and unhandled failures stay visible ✅ 1ms
reporter / conformance-tooling.test.ts / Codecept oracle / omitted zero retry count still checks absence of history ✅ 1ms
reporter / generate-report.test.ts / CTRF reports > writes strict 0.2.0 reports with canonical root fields and scenario details ✅ 178ms
reporter / generate-report.test.ts / CTRF reports > counts final outcomes once and captures failure errors ✅ 89ms
reporter / generate-report.test.ts / CTRF reports > replaces retries with the final result and retains only prior attempts ✅ 82ms
reporter / generate-report.test.ts / CTRF reports > marks exhausted retries as failures rather than flaky passes ✅ 75ms
reporter / generate-report.test.ts / CTRF reports > deep merges metadata, concatenates arrays and isolates scenarios and retries ✅ 73ms
reporter / generate-report.test.ts / CTRF reports > rejects unsafe metadata keys ✅ 32ms
reporter / generate-report.test.ts / CTRF reports > supports minimal output and custom filenames ✅ 105ms
reporter / generate-report.test.ts / CTRF reports > normalizes environment values and retains appName in extra ✅ 79ms
reporter / generate-report.test.ts / CTRF reports > rejects invalid buildNumber ✅ 2ms
reporter / generate-report.test.ts / CTRF reports > rejects invalid buildNumber no ✅ 1ms
reporter / generate-report.test.ts / CTRF reports > rejects invalid buildNumber 1.5 ✅ 1ms
reporter / generate-report.test.ts / CTRF reports > rejects invalid buildNumber -1 ✅ 1ms
reporter / generate-report.test.ts / CTRF reports > rejects invalid buildNumber NaN ✅ 1ms
reporter / generate-report.test.ts / CTRF reports > rejects invalid buildNumber Infinity ✅ 1ms
reporter / generate-report.test.ts / CTRF reports > resets reports between runs and detaches listeners ✅ 31ms
reporter / generate-report.test.ts / CTRF reports > handles missing optional data and unknown step statuses ✅ 77ms
reporter / identity.test.ts / identity semantics > normalizes paths but preserves suite component boundaries ✅ 4ms
reporter / identity.test.ts / identity semantics > includes only explicitly supplied run identity ✅ 3ms
reporter / identity.test.ts / identity semantics > supports an explicit case resolver without allowing empty identity ✅ 1ms
reporter / identity.test.ts / retains execution identity through retries and separates attempt identity ✅ 1ms

± Comparison with run #30 at 3e30af3 | 🍂 No flaky tests detected across all runs. | ⏱️ Measured over 2 runs.

Github Test Reporter by CTRF 💚

🔄 This comment has been updated

@Ma11hewThomas
Ma11hewThomas merged commit a44c8ff into main Oct 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant