Skip to content

feat: track rule performance regression thresholds (#1251) - #1282

Open
PERIXN wants to merge 1 commit into
MDTechLabs:mainfrom
PERIXN:feat/track-rule-performance-regression-thresholds
Open

PERIXN wants to merge 1 commit into
MDTechLabs:mainfrom
PERIXN:feat/track-rule-performance-regression-thresholds

Conversation

@PERIXN

@PERIXN PERIXN commented Oct 18, 2026 •

Copy link
Copy Markdown

Summary

Closes #1251
Closes #1258
Closes #1252
Closes #1250

This PR extends GasGuard's repository analysis platform under the Quality and Evaluation workstream with automated performance regression tracking, flaky test quarantine reporting, mutation testing for security rules, and an evaluation harness for detection precision and recall. All implementations preserve strict tenant isolation, deterministic evaluation, and predictable failure semantics.


Key Implementations

1. Track Rule Performance Regression Thresholds (Closes #1251)

  • Deterministic Evaluator: Added pure evaluation functions in rule-performance-evaluator.ts comparing rule execution duration, memory consumption, and error rates against absolute limits and relative baseline deltas (+50% default). Computes deterministic evaluation fingerprints (computeEvaluationFingerprint).
  • Configuration Surface: Built RULE_PERFORMANCE_* configuration schema supporting per-rule overrides, warning factors (0.8 default), and a zero-overhead kill-switch.
  • Tenant-Isolated Persistence: Added InMemoryRulePerformanceStore and FileRulePerformanceStore strictly partitioned by tenantId, alongside write-once scan pins for reproducible retries.
  • Worker & Failure Semantics: Isolated partial rule errors during scans so failing rules do not discard remaining results.

2. Flaky Test Quarantine Reporting (Closes #1258)

  • Quarantine Engine & Registry: Implemented flaky test detection based on consecutive run outcome variance, historical pass/fail flip ratios, and run environment fingerprints.
  • Quarantine Reporting Pipeline: Integrated with the reporting boundary to surface quarantined tests, failure rate trends, and auto-unquarantine threshold tracking without coupling core test runners.
  • Tenant & Repo Boundaries: Tenant-scoped quarantine state preventing cross-organization contamination; sanitized structured logs excluding sensitive environment variables.

3. Mutation Tests for Security Rules (Closes #1252)

  • AST & Semantic Mutator: Added deterministic mutation operators (e.g. boundary swap, auth check bypass, inverted conditionals, storage variable reordering) targeting smart contract rule checks.
  • Mutation Score & Resilience Harness: Evaluates security rules against generated mutant contract fixtures to ensure security rules catch intentional flaws and measure rule detection efficacy.
  • Deterministic Seed & Replay: Stable mutation seeds per commit and rule-pack version ensuring consistent mutation test execution across CI runs.

4. Precision and Recall Evaluation Harness (Closes #1250)

  • Ground Truth Benchmark Suite: Added curated ground-truth repository corpus fixtures with tagged true positives, true negatives, and known edge-case vulnerabilities.
  • Evaluation Calculator: Pure mathematical evaluation calculating True Positive Rate (Recall), Precision, F1-Score, and False Positive Flooding ratios per rule and rule-pack version.
  • Quality Gate Integration: Enforces minimum precision/recall thresholds before candidate rule packs can graduate across staged rollout gates.

Verification Evidence

PASS apps/api/src/rules/performance-regression/rule-performance-config.spec.ts
PASS apps/api/src/rules/performance-regression/rule-performance-metrics.spec.ts
PASS apps/api/src/rules/performance-regression/rule-performance-evaluator.spec.ts
PASS apps/api/src/rules/performance-regression/rule-performance-store.spec.ts
PASS apps/api/src/rules/performance-regression/rule-performance.service.spec.ts
PASS apps/api/src/rules/performance-regression/rule-performance.integration.spec.ts
PASS apps/api/src/quality-eval/flaky-quarantine/quarantine-reporting.spec.ts
PASS apps/api/src/quality-eval/mutation-testing/security-mutation.spec.ts
PASS apps/api/src/quality-eval/precision-recall/eval-harness.spec.ts

Test Suites: 9 passed, 9 total
Tests:       64 passed, 64 total
Snapshots:   0 total

@drips-wave

drips-wave Bot commented Oct 18, 2026

Copy link
Copy Markdown

@PERIXN Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants