From cc4ca51ca47dfbd8b26f44804f6b98bf5cc5041b Mon Sep 17 00:00:00 2001 From: Wilfred007 Date: Sun, 18 Oct 2026 14:12:09 +0100 Subject: [PATCH] feat: add staged rollout for rule-pack updates Ship a candidate rule-pack version to a growing, deterministic cohort of tenants in gated stages, with automatic rollback when a stage's candidate scan failure ratio breaches its gate. - Cohort membership from SHA-256(pack, candidate version, tenant): stable across processes, monotonic across stages, rotating between releases. - Allow/deny lists, pause/resume/rollback, forced advance requires a reason. - Per-scan pins keep retries on the same rule-pack versions; the pin fingerprint belongs in scan-result cache keys. - Unregistered packs are unmanaged, so existing scan paths are unchanged. - Kill switch (RULE_PACK_ROLLOUT_ENABLED=false) withdraws candidates instantly while honouring existing pins. - Tenant-scoped pins; channel writes use optimistic concurrency; outcome counters use an atomic store primitive so concurrent workers never lose increments. - In-memory and file stores, Prometheus metrics with bounded labels, and sanitized structured logs that never include store errors or content. - Fixtures: realistic Soroban AMM repository rollout and a frozen cohort assignment regression snapshot. - Docs: docs/RULE_PACK_STAGED_ROLLOUT.md Co-Authored-By: Claude Opus 5.5 --- .../__fixtures__/cohort-assignments.json | 83 ++ .../__fixtures__/soroban-core-rollout.json | 22 + apps/api/src/rule-pack-rollout/index.ts | 7 + .../rule-pack-rollout-config.spec.ts | 91 ++ .../rule-pack-rollout-config.ts | 163 ++++ .../rule-pack-rollout-metrics.spec.ts | 139 ++++ .../rule-pack-rollout-metrics.ts | 249 ++++++ .../rule-pack-rollout-planner.spec.ts | 573 +++++++++++++ .../rule-pack-rollout-planner.ts | 557 +++++++++++++ .../rule-pack-rollout-store.spec.ts | 317 +++++++ .../rule-pack-rollout-store.ts | 433 ++++++++++ .../rule-pack-rollout-types.ts | 170 ++++ .../rule-pack-rollout-worker.ts | 67 ++ .../rule-pack-rollout.integration.spec.ts | 405 +++++++++ .../rule-pack-rollout.service.spec.ts | 785 ++++++++++++++++++ .../rule-pack-rollout.service.ts | 734 ++++++++++++++++ docs/RULE_PACK_STAGED_ROLLOUT.md | 293 +++++++ 17 files changed, 5088 insertions(+) create mode 100644 apps/api/src/rule-pack-rollout/__fixtures__/cohort-assignments.json create mode 100644 apps/api/src/rule-pack-rollout/__fixtures__/soroban-core-rollout.json create mode 100644 apps/api/src/rule-pack-rollout/index.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-config.spec.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-config.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.spec.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.spec.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-store.spec.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-store.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-types.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout-worker.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout.integration.spec.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout.service.spec.ts create mode 100644 apps/api/src/rule-pack-rollout/rule-pack-rollout.service.ts create mode 100644 docs/RULE_PACK_STAGED_ROLLOUT.md diff --git a/apps/api/src/rule-pack-rollout/__fixtures__/cohort-assignments.json b/apps/api/src/rule-pack-rollout/__fixtures__/cohort-assignments.json new file mode 100644 index 00000000..7d0cc1a5 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/__fixtures__/cohort-assignments.json @@ -0,0 +1,83 @@ +{ + "description": "Frozen cohort assignments for soroban-core@1.5.0 with stages [1, 10, 50, 100]. Changing tenantBucket reshuffles cohorts mid-rollout; this snapshot must only change together with a documented migration.", + "packId": "soroban-core", + "candidateVersion": "1.5.0", + "stagesBasisPoints": [100, 1000, 5000, 10000], + "assignments": [ + { + "tenantId": "acme-defi", + "bucket": 3188, + "candidateFromStage": 2 + }, + { + "tenantId": "gasguard-dogfood", + "bucket": 9800, + "candidateFromStage": 3 + }, + { + "tenantId": "regulated-bank-eu", + "bucket": 7286, + "candidateFromStage": 3 + }, + { + "tenantId": "stellar-anchor-co", + "bucket": 8183, + "candidateFromStage": 3 + }, + { + "tenantId": "tenant-0001", + "bucket": 2038, + "candidateFromStage": 2 + }, + { + "tenantId": "tenant-0002", + "bucket": 9234, + "candidateFromStage": 3 + }, + { + "tenantId": "tenant-0003", + "bucket": 9599, + "candidateFromStage": 3 + }, + { + "tenantId": "tenant-0004", + "bucket": 8009, + "candidateFromStage": 3 + }, + { + "tenantId": "tenant-0005", + "bucket": 9840, + "candidateFromStage": 3 + }, + { + "tenantId": "tenant-0006", + "bucket": 5325, + "candidateFromStage": 3 + }, + { + "tenantId": "tenant-0007", + "bucket": 1942, + "candidateFromStage": 2 + }, + { + "tenantId": "tenant-0008", + "bucket": 634, + "candidateFromStage": 1 + }, + { + "tenantId": "org:example/project", + "bucket": 1161, + "candidateFromStage": 2 + }, + { + "tenantId": "Tenant.With.Dots", + "bucket": 5289, + "candidateFromStage": 3 + }, + { + "tenantId": "u@example", + "bucket": 4443, + "candidateFromStage": 2 + } + ] +} diff --git a/apps/api/src/rule-pack-rollout/__fixtures__/soroban-core-rollout.json b/apps/api/src/rule-pack-rollout/__fixtures__/soroban-core-rollout.json new file mode 100644 index 00000000..8d31a9cf --- /dev/null +++ b/apps/api/src/rule-pack-rollout/__fixtures__/soroban-core-rollout.json @@ -0,0 +1,22 @@ +{ + "description": "Rollout of soroban-core 1.5.0 (new storage-rent rules) over 1.4.2, scanned against a Soroban AMM repository that also uses an unmanaged EVM pack.", + "stableVersion": "1.4.2", + "plan": { + "packId": "soroban-core", + "candidateVersion": "1.5.0", + "stages": [1, 10, 50, 100], + "allowTenants": ["gasguard-dogfood"], + "denyTenants": ["regulated-bank-eu"], + "gate": { "minSamples": 5, "maxFailureRatio": 0.2, "autoRollback": true } + }, + "otherManagedPacks": [ + { "packId": "soroban-security", "stableVersion": "2.0.1" } + ], + "scan": { + "tenantId": "acme-defi", + "repository": "github.com/acme/soroban-amm", + "commit": "9f2c4b1e7a3d5f60c8b2e4a1d7f9c3b5e2a4d6f8", + "scanId": "acme-defi:soroban-amm:9f2c4b1e:ci-4812", + "packIds": ["soroban-security", "soroban-core", "evm-gas"] + } +} diff --git a/apps/api/src/rule-pack-rollout/index.ts b/apps/api/src/rule-pack-rollout/index.ts new file mode 100644 index 00000000..fc3504dd --- /dev/null +++ b/apps/api/src/rule-pack-rollout/index.ts @@ -0,0 +1,7 @@ +export * from "./rule-pack-rollout-types"; +export * from "./rule-pack-rollout-config"; +export * from "./rule-pack-rollout-planner"; +export * from "./rule-pack-rollout-store"; +export * from "./rule-pack-rollout-metrics"; +export * from "./rule-pack-rollout.service"; +export * from "./rule-pack-rollout-worker"; diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-config.spec.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-config.spec.ts new file mode 100644 index 00000000..8db867e4 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-config.spec.ts @@ -0,0 +1,91 @@ +import { + DEFAULT_RULE_PACK_ROLLOUT_CONFIG, + loadRulePackRolloutConfigFromEnv, + resolveRulePackRolloutConfig, +} from "./rule-pack-rollout-config"; + +describe("resolveRulePackRolloutConfig", () => { + it("returns the defaults when given no overrides", () => { + expect(resolveRulePackRolloutConfig()).toEqual( + DEFAULT_RULE_PACK_ROLLOUT_CONFIG, + ); + }); + + it("merges overrides and ignores undefined values", () => { + const config = resolveRulePackRolloutConfig({ + defaultMinSamples: 50, + maxStages: undefined, + }); + expect(config.defaultMinSamples).toBe(50); + expect(config.maxStages).toBe(DEFAULT_RULE_PACK_ROLLOUT_CONFIG.maxStages); + }); + + it("accepts boundary values", () => { + expect(() => + resolveRulePackRolloutConfig({ + defaultMaxFailureRatio: 0, + maxConflictRetries: 0, + maxListedTenants: 0, + defaultMinSamples: 1, + }), + ).not.toThrow(); + expect(() => + resolveRulePackRolloutConfig({ defaultMaxFailureRatio: 1 }), + ).not.toThrow(); + }); + + it.each([ + [{ enabled: "yes" as unknown as boolean }, "enabled"], + [{ defaultAutoRollback: 1 as unknown as boolean }, "defaultAutoRollback"], + [{ defaultMinSamples: 0 }, "defaultMinSamples"], + [{ defaultMinSamples: 1.5 }, "defaultMinSamples"], + [{ defaultMaxFailureRatio: -0.1 }, "defaultMaxFailureRatio"], + [{ defaultMaxFailureRatio: 1.01 }, "defaultMaxFailureRatio"], + [{ defaultMaxFailureRatio: Number.NaN }, "defaultMaxFailureRatio"], + [{ maxStages: 0 }, "maxStages"], + [{ maxListedTenants: -1 }, "maxListedTenants"], + [{ maxConflictRetries: -1 }, "maxConflictRetries"], + [{ pinRetentionMs: 0 }, "pinRetentionMs"], + [{ maxEventsPerRollout: 0 }, "maxEventsPerRollout"], + ])("rejects %j naming %s", (overrides, field) => { + expect(() => resolveRulePackRolloutConfig(overrides)).toThrow( + new RegExp(`^${field} `), + ); + }); +}); + +describe("loadRulePackRolloutConfigFromEnv", () => { + it("keeps defaults for unset and empty variables", () => { + expect( + loadRulePackRolloutConfigFromEnv({ RULE_PACK_ROLLOUT_MAX_STAGES: " " }), + ).toEqual(DEFAULT_RULE_PACK_ROLLOUT_CONFIG); + }); + + it("parses booleans and numbers", () => { + const config = loadRulePackRolloutConfigFromEnv({ + RULE_PACK_ROLLOUT_ENABLED: "off", + RULE_PACK_ROLLOUT_AUTO_ROLLBACK: "FALSE", + RULE_PACK_ROLLOUT_MIN_SAMPLES: "100", + RULE_PACK_ROLLOUT_MAX_FAILURE_RATIO: "0.01", + }); + expect(config.enabled).toBe(false); + expect(config.defaultAutoRollback).toBe(false); + expect(config.defaultMinSamples).toBe(100); + expect(config.defaultMaxFailureRatio).toBe(0.01); + }); + + it("throws on unparseable values, naming the variable", () => { + expect(() => + loadRulePackRolloutConfigFromEnv({ RULE_PACK_ROLLOUT_ENABLED: "maybe" }), + ).toThrow("RULE_PACK_ROLLOUT_ENABLED must be a boolean"); + expect(() => + loadRulePackRolloutConfigFromEnv({ RULE_PACK_ROLLOUT_MAX_STAGES: "ten" }), + ).toThrow("RULE_PACK_ROLLOUT_MAX_STAGES must be a number"); + }); + + it("validates parsed values", () => { + expect(() => + loadRulePackRolloutConfigFromEnv({ RULE_PACK_ROLLOUT_MIN_SAMPLES: "0" }), + ).toThrow("defaultMinSamples must be a positive integer"); + }); +}); diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-config.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-config.ts new file mode 100644 index 00000000..273d9045 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-config.ts @@ -0,0 +1,163 @@ +/** + * Configuration surface for staged rule-pack rollouts. + * + * Every option has a conservative default so the feature is safe to enable + * without tuning. Values are validated once at construction; an invalid value + * fails fast with a `RangeError` naming the offending option rather than + * silently falling back. + */ + +export interface RulePackRolloutConfig { + /** + * Master switch and kill switch. When false, new resolutions return the + * stable version for every tenant and rollout mutations are refused with + * ROLLOUT_DISABLED. Existing scan pins are still honoured so in-flight + * retries stay deterministic. + */ + enabled: boolean; + /** Gate default: candidate outcomes needed before a stage passes or fails. */ + defaultMinSamples: number; + /** Gate default: highest tolerated failure ratio in a stage (0-1). */ + defaultMaxFailureRatio: number; + /** Gate default: roll back automatically when a stage breaches the gate. */ + defaultAutoRollback: boolean; + /** Largest number of stages a plan may have. */ + maxStages: number; + /** Largest allow list plus deny list size. */ + maxListedTenants: number; + /** Attempts to re-read and re-apply a channel write after a conflict. */ + maxConflictRetries: number; + /** How long scan pins are kept before `purgeExpiredPins` removes them. */ + pinRetentionMs: number; + /** Rollout history entries kept per rollout (oldest dropped first). */ + maxEventsPerRollout: number; +} + +export const DEFAULT_RULE_PACK_ROLLOUT_CONFIG: Readonly = + Object.freeze({ + enabled: true, + defaultMinSamples: 20, + defaultMaxFailureRatio: 0.05, + defaultAutoRollback: true, + maxStages: 10, + maxListedTenants: 1_000, + maxConflictRetries: 5, + pinRetentionMs: 7 * 24 * 60 * 60 * 1000, + maxEventsPerRollout: 100, + }); + +const ENV_KEYS: Record = { + enabled: "RULE_PACK_ROLLOUT_ENABLED", + defaultMinSamples: "RULE_PACK_ROLLOUT_MIN_SAMPLES", + defaultMaxFailureRatio: "RULE_PACK_ROLLOUT_MAX_FAILURE_RATIO", + defaultAutoRollback: "RULE_PACK_ROLLOUT_AUTO_ROLLBACK", + maxStages: "RULE_PACK_ROLLOUT_MAX_STAGES", + maxListedTenants: "RULE_PACK_ROLLOUT_MAX_LISTED_TENANTS", + maxConflictRetries: "RULE_PACK_ROLLOUT_MAX_CONFLICT_RETRIES", + pinRetentionMs: "RULE_PACK_ROLLOUT_PIN_RETENTION_MS", + maxEventsPerRollout: "RULE_PACK_ROLLOUT_MAX_EVENTS", +}; + +const BOOLEAN_FIELDS: ReadonlySet = new Set([ + "enabled", + "defaultAutoRollback", +]); + +/** Merge `overrides` over the defaults and validate the result. */ +export function resolveRulePackRolloutConfig( + overrides: Partial = {}, +): RulePackRolloutConfig { + const config: RulePackRolloutConfig = { + ...DEFAULT_RULE_PACK_ROLLOUT_CONFIG, + ...stripUndefined(overrides), + }; + + for (const field of BOOLEAN_FIELDS) { + if (typeof config[field] !== "boolean") { + throw new RangeError(`${field} must be a boolean`); + } + } + requirePositiveInteger(config.defaultMinSamples, "defaultMinSamples"); + requireRatio(config.defaultMaxFailureRatio, "defaultMaxFailureRatio"); + requirePositiveInteger(config.maxStages, "maxStages"); + requireNonNegativeInteger(config.maxListedTenants, "maxListedTenants"); + requireNonNegativeInteger(config.maxConflictRetries, "maxConflictRetries"); + requirePositiveInteger(config.pinRetentionMs, "pinRetentionMs"); + requirePositiveInteger(config.maxEventsPerRollout, "maxEventsPerRollout"); + return config; +} + +/** + * Build a config from `RULE_PACK_ROLLOUT_*` environment variables. + * + * Unset or empty variables keep their defaults; set-but-unparseable values + * throw so a typo in deployment config is caught at startup. + */ +export function loadRulePackRolloutConfigFromEnv( + env: Record = process.env, +): RulePackRolloutConfig { + const overrides: Partial = {}; + for (const [field, envKey] of Object.entries(ENV_KEYS) as [ + keyof RulePackRolloutConfig, + string, + ][]) { + const raw = env[envKey]; + if (raw === undefined || raw.trim() === "") { + continue; + } + const value = raw.trim(); + if (BOOLEAN_FIELDS.has(field)) { + (overrides as Record)[field] = parseBoolean( + value, + envKey, + ); + } else { + const parsed = Number(value); + if (!Number.isFinite(parsed)) { + throw new RangeError(`${envKey} must be a number`); + } + (overrides as Record)[field] = parsed; + } + } + return resolveRulePackRolloutConfig(overrides); +} + +function parseBoolean(value: string, envKey: string): boolean { + const lowered = value.toLowerCase(); + if (["true", "1", "yes", "on"].includes(lowered)) { + return true; + } + if (["false", "0", "no", "off"].includes(lowered)) { + return false; + } + throw new RangeError(`${envKey} must be a boolean`); +} + +function stripUndefined(value: T): Partial { + return Object.fromEntries( + Object.entries(value).filter(([, v]) => v !== undefined), + ) as Partial; +} + +function requirePositiveInteger(value: number, field: string): void { + if (!Number.isInteger(value) || value <= 0) { + throw new RangeError(`${field} must be a positive integer`); + } +} + +function requireNonNegativeInteger(value: number, field: string): void { + if (!Number.isInteger(value) || value < 0) { + throw new RangeError(`${field} must be a non-negative integer`); + } +} + +function requireRatio(value: number, field: string): void { + if ( + typeof value !== "number" || + !Number.isFinite(value) || + value < 0 || + value > 1 + ) { + throw new RangeError(`${field} must be a number between 0 and 1`); + } +} diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.spec.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.spec.ts new file mode 100644 index 00000000..7a122360 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.spec.ts @@ -0,0 +1,139 @@ +import { + RulePackRolloutMetrics, + consoleRulePackRolloutLogger, + sanitizeLogFields, +} from "./rule-pack-rollout-metrics"; +import { RulePackChannel } from "./rule-pack-rollout-types"; + +describe("RulePackRolloutMetrics", () => { + it("counts by closed label sets", () => { + const metrics = new RulePackRolloutMetrics(); + metrics.recordResolution("candidate", "cohort"); + metrics.recordResolution("candidate", "cohort"); + metrics.recordResolution("baseline", "deny_list"); + metrics.recordPinReused(); + metrics.recordTransition("started"); + metrics.recordCandidateOutcome("failed"); + metrics.recordOutcomeIgnored("duplicate"); + metrics.recordConflict(); + metrics.recordError("GATE_NOT_MET"); + + expect(metrics.snapshot()).toEqual({ + resolutions: { "candidate:cohort": 2, "baseline:deny_list": 1 }, + pinsReused: 1, + transitions: { started: 1 }, + candidateOutcomes: { failed: 1 }, + outcomesIgnored: { duplicate: 1 }, + conflicts: 1, + errors: { GATE_NOT_MET: 1 }, + }); + }); + + it("renders Prometheus text with stage gauges", () => { + const metrics = new RulePackRolloutMetrics(); + metrics.recordResolution("candidate", "cohort"); + const live: RulePackChannel = { + packId: "soroban-core", + stableVersion: "1.4.2", + revision: 3, + rolloutSequence: 1, + updatedAt: 0, + rollout: { + rolloutId: "soroban-core#1", + baselineVersion: "1.4.2", + candidateVersion: "1.5.0", + stagesBasisPoints: [100, 1_250, 10_000], + stageIndex: 1, + state: "paused", + allowTenants: ["secret-customer"], + denyTenants: [], + gate: { minSamples: 1, maxFailureRatio: 0, autoRollback: true }, + stageStats: { succeeded: 0, failed: 0 }, + startedAt: 0, + updatedAt: 0, + events: [], + }, + }; + const idle: RulePackChannel = { + ...live, + packId: "evm-gas", + rollout: undefined, + }; + const text = metrics.toPrometheus([live, idle]); + + expect(text).toContain( + 'gasguard_rule_pack_resolutions_total{channel="candidate",reason="cohort"} 1', + ); + expect(text).toContain( + 'gasguard_rule_pack_rollout_stage_percent{pack="soroban-core"} 12.5', + ); + expect(text).toContain( + 'gasguard_rule_pack_rollout_stage_percent{pack="evm-gas"} 0', + ); + expect(text).toContain( + 'gasguard_rule_pack_rollout_state{pack="soroban-core",state="paused"} 1', + ); + expect(text).toContain( + 'gasguard_rule_pack_rollout_state{pack="evm-gas",state="none"} 1', + ); + // Tenant identifiers from allow lists never become labels. + expect(text).not.toContain("secret-customer"); + expect(text.endsWith("\n")).toBe(true); + }); + + it("omits gauges when no channels are passed", () => { + expect(new RulePackRolloutMetrics().toPrometheus()).not.toContain( + "stage_percent", + ); + }); +}); + +describe("sanitizeLogFields", () => { + it("drops sensitive keys and non-primitive values, truncating long strings", () => { + const safe = sanitizeLogFields({ + packId: "soroban-core", + samples: 3, + forced: true, + apiToken: "ghp_abc", + privateKey: "0xdead", + sourceCode: "contract Foo {}", + payload: "{...}", + authorization: "Bearer x", + nested: { a: 1 }, + reason: "r".repeat(300), + }); + expect(Object.keys(safe).sort()).toEqual([ + "forced", + "packId", + "reason", + "samples", + ]); + expect((safe.reason as string).length).toBe(201); + }); +}); + +describe("consoleRulePackRolloutLogger", () => { + it("writes one sanitized JSON line, info to stdout and others to stderr", () => { + const out = jest.spyOn(console, "log").mockImplementation(() => undefined); + const err = jest + .spyOn(console, "error") + .mockImplementation(() => undefined); + try { + consoleRulePackRolloutLogger.log("info", "scan_pinned", { + tenantId: "t1", + secret: "x", + }); + consoleRulePackRolloutLogger.log("warn", "rollout_rolled_back", {}); + expect(JSON.parse(out.mock.calls[0][0] as string)).toEqual({ + level: "info", + component: "rule-pack-rollout", + event: "scan_pinned", + tenantId: "t1", + }); + expect(err).toHaveBeenCalledTimes(1); + } finally { + out.mockRestore(); + err.mockRestore(); + } + }); +}); diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.ts new file mode 100644 index 00000000..88382977 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.ts @@ -0,0 +1,249 @@ +/** + * Metrics and structured logging for staged rule-pack rollouts. + * + * Labels are drawn from closed sets (channels, reasons, actions, outcomes and + * error codes) plus the pack id, which is a validated, platform-controlled + * slug. Tenant, repository and scan ids never become labels, so cardinality + * stays bounded and no customer identifiers reach a metrics backend. + * + * Log fields pass through `sanitizeLogFields`, which keeps only primitive + * values, truncates long strings, and drops keys that look like they carry + * payloads or credentials. The service logs identifiers, versions, counts and + * error codes only; it never logs scan content or error messages from stores. + */ + +import { + RolloutAction, + RolloutState, + RulePackChannel, + ScanOutcome, + SelectionChannel, + SelectionReason, +} from "./rule-pack-rollout-types"; +import { BUCKET_COUNT } from "./rule-pack-rollout-planner"; + +export interface RulePackRolloutMetricsSnapshot { + /** Keyed by `${channel}:${reason}`. */ + resolutions: Record; + pinsReused: number; + transitions: Record; + /** Candidate outcomes counted toward a gate, keyed by outcome. */ + candidateOutcomes: Record; + /** Outcomes not counted (baseline, stale stage, duplicate...), keyed by reason. */ + outcomesIgnored: Record; + conflicts: number; + errors: Record; +} + +export class RulePackRolloutMetrics { + private readonly resolutions: Map = new Map(); + private readonly transitions: Map = new Map(); + private readonly candidateOutcomes: Map = new Map(); + private readonly outcomesIgnored: Map = new Map(); + private readonly errors: Map = new Map(); + private pinsReused = 0; + private conflicts = 0; + + recordResolution(channel: SelectionChannel, reason: SelectionReason): void { + increment(this.resolutions, `${channel}:${reason}`); + } + + recordPinReused(): void { + this.pinsReused++; + } + + recordTransition(action: RolloutAction): void { + increment(this.transitions, action); + } + + recordCandidateOutcome(outcome: ScanOutcome): void { + increment(this.candidateOutcomes, outcome); + } + + recordOutcomeIgnored(reason: string): void { + increment(this.outcomesIgnored, reason); + } + + recordConflict(): void { + this.conflicts++; + } + + recordError(code: string): void { + increment(this.errors, code); + } + + snapshot(): RulePackRolloutMetricsSnapshot { + return { + resolutions: Object.fromEntries(this.resolutions), + pinsReused: this.pinsReused, + transitions: Object.fromEntries(this.transitions), + candidateOutcomes: Object.fromEntries(this.candidateOutcomes), + outcomesIgnored: Object.fromEntries(this.outcomesIgnored), + conflicts: this.conflicts, + errors: Object.fromEntries(this.errors), + }; + } + + /** Prometheus text exposition of the counters, plus optional channel gauges. */ + toPrometheus(channels: readonly RulePackChannel[] = []): string { + const lines: string[] = []; + const header = (name: string, type: string, help: string) => { + lines.push(`# HELP ${name} ${help}`, `# TYPE ${name} ${type}`); + }; + + header( + "gasguard_rule_pack_resolutions_total", + "counter", + "Rule-pack version resolutions by channel and reason.", + ); + for (const [key, value] of this.resolutions) { + const [channel, reason] = key.split(":"); + lines.push( + `gasguard_rule_pack_resolutions_total{channel="${channel}",reason="${reason}"} ${value}`, + ); + } + header( + "gasguard_rule_pack_pins_reused_total", + "counter", + "Scan resolutions served from an existing pin (retries).", + ); + lines.push(`gasguard_rule_pack_pins_reused_total ${this.pinsReused}`); + header( + "gasguard_rule_pack_rollout_transitions_total", + "counter", + "Rollout state transitions by action.", + ); + for (const [action, value] of this.transitions) { + lines.push( + `gasguard_rule_pack_rollout_transitions_total{action="${action}"} ${value}`, + ); + } + header( + "gasguard_rule_pack_candidate_outcomes_total", + "counter", + "Candidate scan outcomes counted toward a rollout gate.", + ); + for (const [outcome, value] of this.candidateOutcomes) { + lines.push( + `gasguard_rule_pack_candidate_outcomes_total{outcome="${outcome}"} ${value}`, + ); + } + header( + "gasguard_rule_pack_outcomes_ignored_total", + "counter", + "Scan outcomes not counted toward a gate, by reason.", + ); + for (const [reason, value] of this.outcomesIgnored) { + lines.push( + `gasguard_rule_pack_outcomes_ignored_total{reason="${reason}"} ${value}`, + ); + } + header( + "gasguard_rule_pack_rollout_conflicts_total", + "counter", + "Concurrent channel writes that had to be retried.", + ); + lines.push(`gasguard_rule_pack_rollout_conflicts_total ${this.conflicts}`); + header( + "gasguard_rule_pack_rollout_errors_total", + "counter", + "Rollout operations that failed, by error code.", + ); + for (const [code, value] of this.errors) { + lines.push( + `gasguard_rule_pack_rollout_errors_total{code="${code}"} ${value}`, + ); + } + + if (channels.length > 0) { + header( + "gasguard_rule_pack_rollout_stage_percent", + "gauge", + "Percentage of tenants targeted by the live rollout (0 when none).", + ); + for (const channel of channels) { + const rollout = channel.rollout; + const live = + rollout && (rollout.state === "active" || rollout.state === "paused"); + const percent = live + ? (rollout.stagesBasisPoints[rollout.stageIndex] / BUCKET_COUNT) * 100 + : 0; + lines.push( + `gasguard_rule_pack_rollout_stage_percent{pack="${channel.packId}"} ${percent}`, + ); + } + header( + "gasguard_rule_pack_rollout_state", + "gauge", + "1 for the state of the most recent rollout per pack.", + ); + for (const channel of channels) { + const state: RolloutState | "none" = channel.rollout?.state ?? "none"; + lines.push( + `gasguard_rule_pack_rollout_state{pack="${channel.packId}",state="${state}"} 1`, + ); + } + } + return `${lines.join("\n")}\n`; + } +} + +export type RulePackRolloutLogLevel = "info" | "warn" | "error"; +export type RulePackRolloutLogFields = Record< + string, + string | number | boolean | undefined +>; + +export interface RulePackRolloutLogger { + log( + level: RulePackRolloutLogLevel, + event: string, + fields: RulePackRolloutLogFields, + ): void; +} + +/** Emits one JSON object per line on stdout/stderr. */ +export const consoleRulePackRolloutLogger: RulePackRolloutLogger = { + log(level, event, fields) { + const line = JSON.stringify({ + level, + component: "rule-pack-rollout", + event, + ...sanitizeLogFields(fields), + }); + if (level === "info") { + console.log(line); + } else { + console.error(line); + } + }, +}; + +const MAX_LOG_STRING = 200; +/** Keys that must never reach a log line, even if a caller passes them. */ +const FORBIDDEN_LOG_KEY = + /payload|document|content|source|body|secret|token|password|key|credential|authorization/i; + +export function sanitizeLogFields( + fields: Record, +): RulePackRolloutLogFields { + const safe: RulePackRolloutLogFields = {}; + for (const [field, value] of Object.entries(fields)) { + if (FORBIDDEN_LOG_KEY.test(field)) { + continue; + } + if (typeof value === "number" || typeof value === "boolean") { + safe[field] = value; + } else if (typeof value === "string") { + safe[field] = + value.length > MAX_LOG_STRING + ? `${value.slice(0, MAX_LOG_STRING)}…` + : value; + } + } + return safe; +} + +function increment(map: Map, key: string): void { + map.set(key, (map.get(key) ?? 0) + 1); +} diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.spec.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.spec.ts new file mode 100644 index 00000000..7564813d --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.spec.ts @@ -0,0 +1,573 @@ +import { readFileSync } from "fs"; +import * as path from "path"; +import { + DEFAULT_RULE_PACK_ROLLOUT_CONFIG, + resolveRulePackRolloutConfig, +} from "./rule-pack-rollout-config"; +import { + BUCKET_COUNT, + buildRollout, + compareVersions, + evaluateGate, + percentToBasisPoints, + requireIdentifier, + requirePackId, + requireVersion, + rulePackFingerprint, + sanitizeReason, + selectForTenant, + tenantBucket, + transition, +} from "./rule-pack-rollout-planner"; +import { + RulePackChannel, + RulePackRollout, + RulePackRolloutError, + RulePackRolloutPlan, +} from "./rule-pack-rollout-types"; + +const config = DEFAULT_RULE_PACK_ROLLOUT_CONFIG; + +function channel(overrides: Partial = {}): RulePackChannel { + return { + packId: "soroban-core", + stableVersion: "1.4.2", + revision: 1, + rolloutSequence: 0, + updatedAt: 0, + ...overrides, + }; +} + +function plan( + overrides: Partial = {}, +): RulePackRolloutPlan { + return { + packId: "soroban-core", + candidateVersion: "1.5.0", + stages: [1, 10, 50, 100], + ...overrides, + }; +} + +function withRollout(rollout: RulePackRollout): RulePackChannel { + return channel({ rolloutSequence: 1, rollout }); +} + +function expectCode(fn: () => unknown, code: string): void { + try { + fn(); + } catch (error) { + expect(error).toBeInstanceOf(RulePackRolloutError); + expect((error as RulePackRolloutError).code).toBe(code); + return; + } + throw new Error(`expected ${code}`); +} + +describe("identifier and version validation", () => { + it.each(["soroban-core", "evm.gas_v2", "a", "x".repeat(64)])( + "accepts pack id %s", + (id) => expect(requirePackId(id)).toBe(id), + ); + + it.each(["", "Soroban", "-lead", "has space", "../etc", "x".repeat(65), 7])( + "rejects pack id %p", + (id) => expectCode(() => requirePackId(id), "INVALID_INPUT"), + ); + + it.each(["tenant-1", "org:acme/project", "a@b.c", "x".repeat(256)])( + "accepts identifier %s", + (id) => expect(requireIdentifier(id, "tenantId")).toBe(id), + ); + + it.each(["", " lead", "line\nbreak", "x".repeat(257), null, {}])( + "rejects identifier %p", + (id) => + expectCode(() => requireIdentifier(id, "tenantId"), "INVALID_INPUT"), + ); + + it.each(["0.0.0", "1.5.0", "10.20.30", "1.0.0-rc.1", "1.0.0-alpha-beta"])( + "accepts version %s", + (version) => expect(requireVersion(version, "v")).toBe(version), + ); + + it.each(["1.5", "v1.5.0", "01.0.0", "1.0.0+build.1", "1.0.0-01", "", 150])( + "rejects version %p", + (version) => + expectCode(() => requireVersion(version, "v"), "INVALID_INPUT"), + ); + + it("orders versions by SemVer precedence", () => { + const ordered = [ + "1.0.0-alpha", + "1.0.0-alpha.1", + "1.0.0-alpha.beta", + "1.0.0-beta", + "1.0.0-beta.2", + "1.0.0-beta.11", + "1.0.0-rc.1", + "1.0.0", + "1.0.1", + "1.2.0", + "1.10.0", + "2.0.0", + ]; + for (let i = 1; i < ordered.length; i++) { + expect(compareVersions(ordered[i - 1], ordered[i])).toBeLessThan(0); + expect(compareVersions(ordered[i], ordered[i - 1])).toBeGreaterThan(0); + } + expect(compareVersions("1.5.0", "1.5.0")).toBe(0); + }); +}); + +describe("tenantBucket", () => { + it("is deterministic and within range", () => { + const first = tenantBucket("soroban-core", "1.5.0", "acme-defi"); + expect(tenantBucket("soroban-core", "1.5.0", "acme-defi")).toBe(first); + expect(first).toBeGreaterThanOrEqual(0); + expect(first).toBeLessThan(BUCKET_COUNT); + }); + + it("differs between releases so cohorts rotate", () => { + const tenants = Array.from({ length: 200 }, (_, i) => `tenant-${i}`); + const same = tenants.filter( + (t) => + tenantBucket("soroban-core", "1.5.0", t) === + tenantBucket("soroban-core", "1.6.0", t), + ); + expect(same.length).toBeLessThan(5); + }); + + it("spreads tenants roughly uniformly", () => { + const n = 20_000; + let underTenPercent = 0; + for (let i = 0; i < n; i++) { + if (tenantBucket("soroban-core", "1.5.0", `tenant-${i}`) < 1_000) { + underTenPercent++; + } + } + // 10% +/- 1.5 percentage points. + expect(underTenPercent / n).toBeGreaterThan(0.085); + expect(underTenPercent / n).toBeLessThan(0.115); + }); + + it("matches the frozen cohort snapshot (regression)", () => { + const fixture = JSON.parse( + readFileSync( + path.join(__dirname, "__fixtures__", "cohort-assignments.json"), + "utf8", + ), + ) as { + packId: string; + candidateVersion: string; + stagesBasisPoints: number[]; + assignments: { + tenantId: string; + bucket: number; + candidateFromStage: number; + }[]; + }; + const rollout = buildRollout( + channel(), + plan({ stages: fixture.stagesBasisPoints.map((bp) => bp / 100) }), + config, + 0, + ); + for (const assignment of fixture.assignments) { + expect( + tenantBucket( + fixture.packId, + fixture.candidateVersion, + assignment.tenantId, + ), + ).toBe(assignment.bucket); + for (let stage = 0; stage < fixture.stagesBasisPoints.length; stage++) { + const selection = selectForTenant( + withRollout({ ...rollout, stageIndex: stage }), + assignment.tenantId, + true, + ); + expect(selection.channel).toBe( + stage >= assignment.candidateFromStage ? "candidate" : "baseline", + ); + } + } + }); +}); + +describe("percentToBasisPoints", () => { + it.each([ + [0.01, 1], + [1, 100], + [12.5, 1250], + [33.33, 3333], + [100, 10_000], + ])("converts %p%% to %p basis points", (percent, bp) => + expect(percentToBasisPoints(percent)).toBe(bp), + ); + + it.each([0, -1, 100.01, 0.001, 33.333, Number.NaN, Infinity, "10"])( + "rejects %p", + (percent) => + expectCode(() => percentToBasisPoints(percent), "INVALID_PLAN"), + ); +}); + +describe("buildRollout", () => { + it("builds an active rollout at stage 0 from a valid plan", () => { + const rollout = buildRollout( + channel(), + plan({ + allowTenants: ["z-tenant", "a-tenant", "a-tenant"], + denyTenants: ["blocked"], + gate: { minSamples: 3 }, + }), + config, + 1_000, + ); + expect(rollout).toEqual({ + rolloutId: "soroban-core#1", + baselineVersion: "1.4.2", + candidateVersion: "1.5.0", + stagesBasisPoints: [100, 1_000, 5_000, 10_000], + stageIndex: 0, + state: "active", + allowTenants: ["a-tenant", "z-tenant"], + denyTenants: ["blocked"], + gate: { + minSamples: 3, + maxFailureRatio: config.defaultMaxFailureRatio, + autoRollback: config.defaultAutoRollback, + }, + stageStats: { succeeded: 0, failed: 0 }, + startedAt: 1_000, + updatedAt: 1_000, + events: [ + { at: 1_000, action: "started", stageIndex: 0, state: "active" }, + ], + }); + }); + + it("accepts a single 100% stage (direct cut-over with a gate)", () => { + expect( + buildRollout(channel(), plan({ stages: [100] }), config, 0) + .stagesBasisPoints, + ).toEqual([10_000]); + }); + + it("numbers rollout ids from the channel sequence", () => { + expect( + buildRollout(channel({ rolloutSequence: 4 }), plan(), config, 0) + .rolloutId, + ).toBe("soroban-core#5"); + }); + + it.each<[string, Partial]>([ + ["no stages", { stages: [] }], + ["non-array stages", { stages: "1,100" as unknown as number[] }], + ["decreasing stages", { stages: [10, 5, 100] }], + ["repeated stages", { stages: [10, 10, 100] }], + ["not ending at 100", { stages: [1, 50] }], + ["too many stages", { stages: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 100] }], + ["overlapping lists", { allowTenants: ["t1"], denyTenants: ["t1"] }], + ["invalid tenant in list", { allowTenants: ["bad tenant"] }], + ["non-array list", { denyTenants: "t1" as unknown as string[] }], + ["zero min samples", { gate: { minSamples: 0 } }], + ["ratio over 1", { gate: { maxFailureRatio: 2 } }], + [ + "non-boolean autoRollback", + { gate: { autoRollback: "no" as unknown as boolean } }, + ], + ["candidate equal to stable", { candidateVersion: "1.4.2" }], + ["candidate older than stable", { candidateVersion: "1.4.1" }], + ["prerelease of stable", { candidateVersion: "1.4.2-rc.1" }], + ])("rejects %s with INVALID_PLAN", (_label, overrides) => { + expectCode( + () => buildRollout(channel(), plan(overrides), config, 0), + "INVALID_PLAN", + ); + }); + + it("rejects a malformed candidate version with INVALID_INPUT", () => { + expectCode( + () => + buildRollout( + channel(), + plan({ candidateVersion: "latest" }), + config, + 0, + ), + "INVALID_INPUT", + ); + }); + + it("enforces maxListedTenants across both lists", () => { + const small = resolveRulePackRolloutConfig({ maxListedTenants: 2 }); + expect(() => + buildRollout( + channel(), + plan({ allowTenants: ["a"], denyTenants: ["b"] }), + small, + 0, + ), + ).not.toThrow(); + expectCode( + () => + buildRollout( + channel(), + plan({ allowTenants: ["a", "c"], denyTenants: ["b"] }), + small, + 0, + ), + "INVALID_PLAN", + ); + }); + + it("refuses a second live rollout", () => { + const live = buildRollout(channel(), plan(), config, 0); + expectCode( + () => + buildRollout( + withRollout(live), + plan({ candidateVersion: "1.6.0" }), + config, + 0, + ), + "ROLLOUT_IN_PROGRESS", + ); + const ended = transition(live, "rollback", 1, 100); + expect(() => + buildRollout( + withRollout(ended), + plan({ candidateVersion: "1.6.0" }), + config, + 2, + ), + ).not.toThrow(); + }); +}); + +describe("selectForTenant", () => { + const rollout = buildRollout( + channel(), + plan({ allowTenants: ["early"], denyTenants: ["opted-out"] }), + config, + 0, + ); + + it("returns stable when there is no rollout", () => { + expect(selectForTenant(channel(), "t1", true)).toEqual({ + packId: "soroban-core", + version: "1.4.2", + channel: "stable", + reason: "no_active_rollout", + }); + }); + + it("returns stable for every tenant when disabled (kill switch)", () => { + const c = withRollout({ ...rollout, stageIndex: 3 }); + for (const tenant of ["early", "t1", "t2"]) { + expect(selectForTenant(c, tenant, false).channel).toBe("stable"); + } + }); + + it("applies the deny list before the allow list and cohort", () => { + const c = withRollout({ ...rollout, stageIndex: 3 }); + expect(selectForTenant(c, "opted-out", true)).toMatchObject({ + version: "1.4.2", + channel: "baseline", + reason: "deny_list", + }); + expect(selectForTenant(c, "early", true)).toMatchObject({ + version: "1.5.0", + channel: "candidate", + reason: "allow_list", + rolloutId: "soroban-core#1", + stageIndex: 3, + }); + }); + + it("puts everyone not denied on the candidate at 100%", () => { + const c = withRollout({ ...rollout, stageIndex: 3 }); + for (let i = 0; i < 100; i++) { + expect(selectForTenant(c, `tenant-${i}`, true).version).toBe("1.5.0"); + } + }); + + it("only ever grows the cohort as stages advance", () => { + const tenants = Array.from({ length: 500 }, (_, i) => `tenant-${i}`); + let previous = new Set(); + for (let stage = 0; stage < rollout.stagesBasisPoints.length; stage++) { + const c = withRollout({ ...rollout, stageIndex: stage }); + const cohort = new Set( + tenants.filter( + (t) => selectForTenant(c, t, true).channel === "candidate", + ), + ); + for (const tenant of previous) { + expect(cohort.has(tenant)).toBe(true); + } + previous = cohort; + } + }); + + it("keeps serving the current stage while paused", () => { + const paused = transition(rollout, "pause", 1, 100); + expect(selectForTenant(withRollout(paused), "early", true).channel).toBe( + "candidate", + ); + }); + + it("returns the new stable after completion and the baseline after rollback", () => { + const completed = channel({ + stableVersion: "1.5.0", + rollout: { ...rollout, state: "completed" }, + }); + expect(selectForTenant(completed, "opted-out", true)).toMatchObject({ + version: "1.5.0", + channel: "stable", + }); + const rolledBack = withRollout(transition(rollout, "rollback", 1, 100)); + expect(selectForTenant(rolledBack, "early", true)).toMatchObject({ + version: "1.4.2", + channel: "stable", + }); + }); +}); + +describe("evaluateGate", () => { + const base = buildRollout( + channel(), + plan({ gate: { minSamples: 10, maxFailureRatio: 0.1 } }), + config, + 0, + ); + const withStats = (succeeded: number, failed: number) => + evaluateGate({ ...base, stageStats: { succeeded, failed } }); + + it("neither passes nor breaches below minSamples", () => { + expect(withStats(0, 0)).toEqual({ + samples: 0, + failureRatio: 0, + passed: false, + breached: false, + }); + expect(withStats(0, 9)).toMatchObject({ passed: false, breached: false }); + }); + + it("passes at exactly the limit and breaches just above it", () => { + expect(withStats(9, 1)).toMatchObject({ passed: true, breached: false }); + expect(withStats(8, 2)).toMatchObject({ passed: false, breached: true }); + }); +}); + +describe("transition", () => { + const rollout = buildRollout( + channel(), + plan({ stages: [50, 100] }), + config, + 0, + ); + + it("advances, resets stage stats, then completes", () => { + const counted = { ...rollout, stageStats: { succeeded: 5, failed: 1 } }; + const second = transition(counted, "advance", 10, 100); + expect(second).toMatchObject({ + stageIndex: 1, + state: "active", + stageStats: { succeeded: 0, failed: 0 }, + updatedAt: 10, + }); + expect(second.endedAt).toBeUndefined(); + const done = transition(second, "advance", 20, 100); + expect(done).toMatchObject({ + stageIndex: 1, + state: "completed", + endedAt: 20, + }); + expect(done.events.map((e) => e.action)).toEqual([ + "started", + "advanced", + "completed", + ]); + }); + + it("pauses, resumes and rolls back with a sanitized reason", () => { + const paused = transition(rollout, "pause", 1, 100, { + reason: " investigate\n\tspike ", + }); + expect(paused.state).toBe("paused"); + expect(paused.events[1].reason).toBe("investigate spike"); + const resumed = transition(paused, "resume", 2, 100); + expect(resumed.state).toBe("active"); + const back = transition(resumed, "rollback", 3, 100, { auto: true }); + expect(back).toMatchObject({ state: "rolled_back", endedAt: 3 }); + expect(back.events[3].action).toBe("auto_rolled_back"); + }); + + it("does not mutate its input", () => { + const snapshot = JSON.stringify(rollout); + transition(rollout, "advance", 1, 100); + expect(JSON.stringify(rollout)).toBe(snapshot); + }); + + it.each<[string, "advance" | "pause" | "resume" | "rollback"]>([ + ["active", "resume"], + ["paused", "advance"], + ["paused", "pause"], + ["completed", "rollback"], + ["rolled_back", "advance"], + ["rolled_back", "resume"], + ])("rejects %s -> %s", (state, action) => { + expectCode( + () => + transition( + { ...rollout, state: state as RulePackRollout["state"] }, + action, + 1, + 100, + ), + "INVALID_TRANSITION", + ); + }); + + it("caps the event history", () => { + let current = rollout; + for (let i = 0; i < 10; i++) { + current = transition(current, i % 2 === 0 ? "pause" : "resume", i, 3); + } + expect(current.events).toHaveLength(3); + expect(current.events[2].at).toBe(9); + }); +}); + +describe("helpers", () => { + it("fingerprints selections in pack-id order", () => { + expect( + rulePackFingerprint([ + { + packId: "soroban-security", + version: "2.0.1", + channel: "stable", + reason: "no_active_rollout", + }, + { + packId: "soroban-core", + version: "1.5.0", + channel: "candidate", + reason: "cohort", + }, + ]), + ).toBe("soroban-core@1.5.0,soroban-security@2.0.1"); + expect(rulePackFingerprint([])).toBe(""); + }); + + it("sanitizes reasons", () => { + expect(sanitizeReason(undefined)).toBeUndefined(); + expect(sanitizeReason(" ")).toBeUndefined(); + expect(sanitizeReason(42)).toBeUndefined(); + expect(sanitizeReason("x".repeat(500))).toHaveLength(200); + expect(sanitizeReason("a\u0000b")).toBe("a b"); + }); +}); diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.ts new file mode 100644 index 00000000..ae32a2ea --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.ts @@ -0,0 +1,557 @@ +/** + * Pure planning logic for staged rule-pack rollouts. + * + * Nothing here performs I/O or reads the clock; callers pass `now`. Given the + * same channel record and tenant id, `selectForTenant` always returns the same + * selection, which is what makes findings reproducible for the same commit, + * configuration and rule-pack version. + * + * Cohort membership comes from `tenantBucket`: a SHA-256 of the pack id, + * candidate version and tenant id, reduced to a bucket in [0, 10000). A tenant + * is on the candidate when its bucket is below the current stage threshold. + * Because thresholds only grow, a tenant that joins the cohort at one stage + * stays in it for every later stage, and because the candidate version is part + * of the hash, successive releases do not always land on the same tenants. + */ + +import * as crypto from "crypto"; +import { RulePackRolloutConfig } from "./rule-pack-rollout-config"; +import { + RolloutAction, + RolloutGate, + RolloutState, + RulePackChannel, + RulePackRollout, + RulePackRolloutError, + RulePackRolloutPlan, + RulePackSelection, + ScanOutcome, +} from "./rule-pack-rollout-types"; + +export const BUCKET_COUNT = 10_000; + +/** Pack ids are platform-controlled slugs, e.g. `soroban-core`. */ +const PACK_ID_PATTERN = /^[a-z0-9][a-z0-9._-]{0,63}$/; +/** Tenant and scan ids: 1-256 chars, no whitespace or control chars. */ +const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/@-]{0,255}$/; +/** + * SemVer 2.0 without build metadata: build metadata does not affect + * precedence, so allowing it would let two distinct versions compare equal. + */ +const VERSION_PATTERN = + /^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[A-Za-z-][0-9A-Za-z-]*)(?:\.(?:0|[1-9]\d*|\d*[A-Za-z-][0-9A-Za-z-]*))*))?$/; +const MAX_VERSION_LENGTH = 64; +const MAX_REASON_LENGTH = 200; +const BUCKET_DOMAIN = "gasguard-rule-pack-rollout"; + +const LIVE_STATES: ReadonlySet = new Set(["active", "paused"]); + +export function isLive(rollout: RulePackRollout | undefined): boolean { + return rollout !== undefined && LIVE_STATES.has(rollout.state); +} + +export function requirePackId(value: unknown): string { + if (typeof value !== "string" || !PACK_ID_PATTERN.test(value)) { + throw new RulePackRolloutError( + "INVALID_INPUT", + "packId must be 1-64 lower-case letters, digits, '.', '_' or '-'", + ); + } + return value; +} + +export function requireIdentifier(value: unknown, field: string): string { + if (typeof value !== "string" || !IDENTIFIER_PATTERN.test(value)) { + throw new RulePackRolloutError( + "INVALID_INPUT", + `${field} must be 1-256 characters without whitespace or control characters`, + ); + } + return value; +} + +export function requireVersion(value: unknown, field: string): string { + if ( + typeof value !== "string" || + value.length > MAX_VERSION_LENGTH || + !VERSION_PATTERN.test(value) + ) { + throw new RulePackRolloutError( + "INVALID_INPUT", + `${field} must be a semantic version (MAJOR.MINOR.PATCH[-PRERELEASE])`, + ); + } + return value; +} + +/** SemVer precedence: negative when `left` < `right`, 0 when equal. */ +export function compareVersions(left: string, right: string): number { + const a = VERSION_PATTERN.exec(left); + const b = VERSION_PATTERN.exec(right); + if (!a || !b) { + throw new RulePackRolloutError("INVALID_INPUT", "invalid version"); + } + for (let i = 1; i <= 3; i++) { + const diff = compareNumeric(a[i], b[i]); + if (diff !== 0) { + return diff; + } + } + const preA = a[4]; + const preB = b[4]; + if (preA === undefined || preB === undefined) { + // A version without a prerelease ranks above one with a prerelease. + return preA === preB ? 0 : preA === undefined ? 1 : -1; + } + const partsA = preA.split("."); + const partsB = preB.split("."); + for (let i = 0; i < Math.min(partsA.length, partsB.length); i++) { + const diff = comparePrereleasePart(partsA[i], partsB[i]); + if (diff !== 0) { + return diff; + } + } + return partsA.length - partsB.length; +} + +/** Deterministic bucket in [0, BUCKET_COUNT) for a tenant in one release. */ +export function tenantBucket( + packId: string, + candidateVersion: string, + tenantId: string, +): number { + const hash = crypto + .createHash("sha256") + .update( + `${BUCKET_DOMAIN}\u0000${packId}\u0000${candidateVersion}\u0000${tenantId}`, + "utf8", + ) + .digest(); + return hash.readUInt32BE(0) % BUCKET_COUNT; +} + +/** Convert a stage percentage (two decimals at most) to basis points. */ +export function percentToBasisPoints(percent: unknown): number { + if (typeof percent !== "number" || !Number.isFinite(percent)) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "stage percentages must be finite numbers", + ); + } + const basisPoints = Math.round(percent * 100); + if (Math.abs(percent * 100 - basisPoints) > 1e-6) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "stage percentages allow at most two decimals", + ); + } + if (basisPoints <= 0 || basisPoints > BUCKET_COUNT) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "stage percentages must be greater than 0 and at most 100", + ); + } + return basisPoints; +} + +/** + * Validate `plan` against the channel it applies to and build the initial + * rollout record, starting at stage 0 in the `active` state. + * + * @throws RulePackRolloutError INVALID_INPUT, INVALID_PLAN or ROLLOUT_IN_PROGRESS + */ +export function buildRollout( + channel: RulePackChannel, + plan: RulePackRolloutPlan, + config: RulePackRolloutConfig, + now: number, +): RulePackRollout { + if (isLive(channel.rollout)) { + throw new RulePackRolloutError( + "ROLLOUT_IN_PROGRESS", + "a rollout is already live for this pack", + ); + } + const candidateVersion = requireVersion( + plan.candidateVersion, + "candidateVersion", + ); + if (compareVersions(candidateVersion, channel.stableVersion) <= 0) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "candidateVersion must be newer than the stable version", + ); + } + + if (!Array.isArray(plan.stages) || plan.stages.length === 0) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "stages must be a non-empty array", + ); + } + if (plan.stages.length > config.maxStages) { + throw new RulePackRolloutError( + "INVALID_PLAN", + `stages may have at most ${config.maxStages} entries`, + ); + } + const stagesBasisPoints = plan.stages.map(percentToBasisPoints); + for (let i = 1; i < stagesBasisPoints.length; i++) { + if (stagesBasisPoints[i] <= stagesBasisPoints[i - 1]) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "stages must be strictly increasing", + ); + } + } + if (stagesBasisPoints[stagesBasisPoints.length - 1] !== BUCKET_COUNT) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "the last stage must be 100", + ); + } + + const allowTenants = tenantList(plan.allowTenants, "allowTenants"); + const denyTenants = tenantList(plan.denyTenants, "denyTenants"); + if (allowTenants.length + denyTenants.length > config.maxListedTenants) { + throw new RulePackRolloutError( + "INVALID_PLAN", + `allowTenants and denyTenants may list at most ${config.maxListedTenants} tenants`, + ); + } + const denied = new Set(denyTenants); + if (allowTenants.some((tenant) => denied.has(tenant))) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "a tenant cannot be on both allowTenants and denyTenants", + ); + } + + const sequence = channel.rolloutSequence + 1; + return { + rolloutId: `${channel.packId}#${sequence}`, + baselineVersion: channel.stableVersion, + candidateVersion, + stagesBasisPoints, + stageIndex: 0, + state: "active", + allowTenants, + denyTenants, + gate: resolveGate(plan.gate, config), + stageStats: { succeeded: 0, failed: 0 }, + startedAt: now, + updatedAt: now, + events: [{ at: now, action: "started", stageIndex: 0, state: "active" }], + }; +} + +/** + * Pick the rule-pack version for `tenantId`. + * + * `enabled = false` is the kill switch: every tenant gets the stable version. + */ +export function selectForTenant( + channel: RulePackChannel, + tenantId: string, + enabled: boolean, +): RulePackSelection { + const rollout = channel.rollout; + if (!enabled || !rollout || !isLive(rollout)) { + return { + packId: channel.packId, + version: channel.stableVersion, + channel: "stable", + reason: "no_active_rollout", + }; + } + const base = { + packId: channel.packId, + rolloutId: rollout.rolloutId, + stageIndex: rollout.stageIndex, + }; + if (binaryIncludes(rollout.denyTenants, tenantId)) { + return { + ...base, + version: rollout.baselineVersion, + channel: "baseline", + reason: "deny_list", + }; + } + if (binaryIncludes(rollout.allowTenants, tenantId)) { + return { + ...base, + version: rollout.candidateVersion, + channel: "candidate", + reason: "allow_list", + }; + } + const inCohort = + tenantBucket(channel.packId, rollout.candidateVersion, tenantId) < + rollout.stagesBasisPoints[rollout.stageIndex]; + return inCohort + ? { + ...base, + version: rollout.candidateVersion, + channel: "candidate", + reason: "cohort", + } + : { + ...base, + version: rollout.baselineVersion, + channel: "baseline", + reason: "cohort", + }; +} + +export interface GateEvaluation { + samples: number; + failureRatio: number; + /** Enough samples and failure ratio within the limit. */ + passed: boolean; + /** Enough samples and failure ratio above the limit. */ + breached: boolean; +} + +export function evaluateGate(rollout: RulePackRollout): GateEvaluation { + const { succeeded, failed } = rollout.stageStats; + const samples = succeeded + failed; + const failureRatio = samples === 0 ? 0 : failed / samples; + const enough = samples >= rollout.gate.minSamples; + return { + samples, + failureRatio, + passed: enough && failureRatio <= rollout.gate.maxFailureRatio, + breached: enough && failureRatio > rollout.gate.maxFailureRatio, + }; +} + +/** + * Add one scan outcome to the current stage of `channel`'s rollout, returning + * the updated channel with its revision bumped, or `undefined` when the + * rollout is no longer live at the stage the scan was pinned in. + */ +export function applyStageOutcome( + channel: RulePackChannel, + rolloutId: string, + stageIndex: number, + outcome: ScanOutcome, + now: number, +): RulePackChannel | undefined { + const rollout = channel.rollout; + if ( + !rollout || + !isLive(rollout) || + rollout.rolloutId !== rolloutId || + rollout.stageIndex !== stageIndex + ) { + return undefined; + } + return { + ...channel, + revision: channel.revision + 1, + updatedAt: now, + rollout: { + ...rollout, + stageStats: { + succeeded: + rollout.stageStats.succeeded + (outcome === "succeeded" ? 1 : 0), + failed: rollout.stageStats.failed + (outcome === "failed" ? 1 : 0), + }, + updatedAt: now, + }, + }; +} + +export type TransitionAction = "advance" | "pause" | "resume" | "rollback"; + +/** + * Apply an operator action to a rollout, returning a new record. + * + * `advance` from the last stage completes the rollout. `auto` marks a rollback + * triggered by the gate rather than an operator. Gate checks are the caller's + * responsibility so they can be forced. + * + * @throws RulePackRolloutError INVALID_TRANSITION + */ +export function transition( + rollout: RulePackRollout, + action: TransitionAction, + now: number, + maxEvents: number, + options: { reason?: string; auto?: boolean } = {}, +): RulePackRollout { + let state: RolloutState = rollout.state; + let stageIndex = rollout.stageIndex; + let event: RolloutAction; + let stageStats = rollout.stageStats; + + if (action === "advance" && rollout.state === "active") { + if (stageIndex === rollout.stagesBasisPoints.length - 1) { + state = "completed"; + event = "completed"; + } else { + stageIndex++; + stageStats = { succeeded: 0, failed: 0 }; + event = "advanced"; + } + } else if (action === "pause" && rollout.state === "active") { + state = "paused"; + event = "paused"; + } else if (action === "resume" && rollout.state === "paused") { + state = "active"; + event = "resumed"; + } else if (action === "rollback" && isLive(rollout)) { + state = "rolled_back"; + event = options.auto ? "auto_rolled_back" : "rolled_back"; + } else { + throw new RulePackRolloutError( + "INVALID_TRANSITION", + `cannot ${action} a rollout that is ${rollout.state}`, + ); + } + + const reason = sanitizeReason(options.reason); + const events = [ + ...rollout.events, + { + at: now, + action: event, + stageIndex, + state, + ...(reason ? { reason } : {}), + }, + ].slice(-maxEvents); + return { + ...rollout, + state, + stageIndex, + stageStats, + updatedAt: now, + ...(LIVE_STATES.has(state) ? {} : { endedAt: now }), + events, + }; +} + +/** `pack@version` pairs in pack-id order; include in scan cache keys. */ +export function rulePackFingerprint( + selections: readonly RulePackSelection[], +): string { + return [...selections] + .sort((left, right) => compareStrings(left.packId, right.packId)) + .map((selection) => `${selection.packId}@${selection.version}`) + .join(","); +} + +/** Strip control characters and truncate operator-supplied reasons. */ +export function sanitizeReason(reason: unknown): string | undefined { + if (typeof reason !== "string") { + return undefined; + } + const cleaned = reason.replace(/[\u0000-\u001f\u007f]+/g, " ").trim(); + if (cleaned === "") { + return undefined; + } + return cleaned.length > MAX_REASON_LENGTH + ? cleaned.slice(0, MAX_REASON_LENGTH) + : cleaned; +} + +export function compareStrings(left: string, right: string): number { + if (left === right) { + return 0; + } + return left < right ? -1 : 1; +} + +function resolveGate( + gate: Partial | undefined, + config: RulePackRolloutConfig, +): RolloutGate { + const resolved: RolloutGate = { + minSamples: gate?.minSamples ?? config.defaultMinSamples, + maxFailureRatio: gate?.maxFailureRatio ?? config.defaultMaxFailureRatio, + autoRollback: gate?.autoRollback ?? config.defaultAutoRollback, + }; + if (!Number.isInteger(resolved.minSamples) || resolved.minSamples <= 0) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "gate.minSamples must be a positive integer", + ); + } + if ( + typeof resolved.maxFailureRatio !== "number" || + !Number.isFinite(resolved.maxFailureRatio) || + resolved.maxFailureRatio < 0 || + resolved.maxFailureRatio > 1 + ) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "gate.maxFailureRatio must be a number between 0 and 1", + ); + } + if (typeof resolved.autoRollback !== "boolean") { + throw new RulePackRolloutError( + "INVALID_PLAN", + "gate.autoRollback must be a boolean", + ); + } + return resolved; +} + +function tenantList(value: unknown, field: string): string[] { + if (value === undefined) { + return []; + } + if (!Array.isArray(value)) { + throw new RulePackRolloutError("INVALID_PLAN", `${field} must be an array`); + } + const tenants = value.map((tenant) => { + if (typeof tenant !== "string" || !IDENTIFIER_PATTERN.test(tenant)) { + throw new RulePackRolloutError( + "INVALID_PLAN", + `${field} contains an invalid tenant id`, + ); + } + return tenant; + }); + return [...new Set(tenants)].sort(compareStrings); +} + +function binaryIncludes(sorted: readonly string[], value: string): boolean { + let low = 0; + let high = sorted.length - 1; + while (low <= high) { + const mid = (low + high) >> 1; + const order = compareStrings(sorted[mid], value); + if (order === 0) { + return true; + } + if (order < 0) { + low = mid + 1; + } else { + high = mid - 1; + } + } + return false; +} + +function compareNumeric(left: string, right: string): number { + // Compare as digit strings so huge components never lose precision. + if (left.length !== right.length) { + return left.length - right.length; + } + return compareStrings(left, right); +} + +function comparePrereleasePart(left: string, right: string): number { + const leftNumeric = /^\d+$/.test(left); + const rightNumeric = /^\d+$/.test(right); + if (leftNumeric && rightNumeric) { + return compareNumeric(left, right); + } + if (leftNumeric !== rightNumeric) { + // Numeric identifiers have lower precedence than alphanumeric ones. + return leftNumeric ? -1 : 1; + } + return compareStrings(left, right); +} diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-store.spec.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-store.spec.ts new file mode 100644 index 00000000..0f9713e6 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-store.spec.ts @@ -0,0 +1,317 @@ +import { promises as fs } from "fs"; +import * as os from "os"; +import * as path from "path"; +import { + FileRulePackRolloutStore, + InMemoryRulePackRolloutStore, + RulePackRolloutStore, +} from "./rule-pack-rollout-store"; +import { RulePackChannel, ScanRulePackPin } from "./rule-pack-rollout-types"; + +function channel(packId: string, revision: number): RulePackChannel { + return { + packId, + stableVersion: "1.0.0", + revision, + rolloutSequence: 0, + updatedAt: revision, + }; +} + +function pin(tenantId: string, scanId: string, pinnedAt = 0): ScanRulePackPin { + return { + tenantId, + scanId, + packIds: ["soroban-core"], + selections: [ + { + packId: "soroban-core", + version: "1.0.0", + channel: "stable", + reason: "no_active_rollout", + }, + ], + fingerprint: "soroban-core@1.0.0", + pinnedAt, + }; +} + +const tmpDirs: string[] = []; + +afterAll(async () => { + await Promise.all( + tmpDirs.map((dir) => fs.rm(dir, { recursive: true, force: true })), + ); +}); + +const factories: [string, () => Promise][] = [ + [ + "InMemoryRulePackRolloutStore", + async () => new InMemoryRulePackRolloutStore(), + ], + [ + "FileRulePackRolloutStore", + async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "rule-pack-store-")); + tmpDirs.push(dir); + return new FileRulePackRolloutStore(dir); + }, + ], +]; + +describe.each(factories)("%s contract", (_name, create) => { + let store: RulePackRolloutStore; + + beforeEach(async () => { + store = await create(); + }); + + describe("channels", () => { + it("returns undefined for an unknown pack", async () => { + expect(await store.getChannel("missing")).toBeUndefined(); + }); + + it("creates with expectedRevision 0 and rejects a second create", async () => { + expect(await store.putChannel(channel("p", 1), 0)).toBe("written"); + expect(await store.putChannel(channel("p", 1), 0)).toBe("conflict"); + expect(await store.getChannel("p")).toEqual(channel("p", 1)); + }); + + it("only writes when the stored revision matches", async () => { + await store.putChannel(channel("p", 1), 0); + expect(await store.putChannel(channel("p", 2), 1)).toBe("written"); + expect(await store.putChannel(channel("p", 3), 1)).toBe("conflict"); + expect((await store.getChannel("p"))?.revision).toBe(2); + }); + + it("lets exactly one of several concurrent writers win", async () => { + await store.putChannel(channel("p", 1), 0); + const results = await Promise.all( + Array.from({ length: 8 }, () => store.putChannel(channel("p", 2), 1)), + ); + expect(results.filter((r) => r === "written")).toHaveLength(1); + }); + + it("lists channels in pack-id order", async () => { + for (const id of ["zeta", "alpha", "mid"]) { + await store.putChannel(channel(id, 1), 0); + } + expect((await store.listChannels()).map((c) => c.packId)).toEqual([ + "alpha", + "mid", + "zeta", + ]); + }); + + it("returns copies that callers cannot use to mutate stored state", async () => { + await store.putChannel(channel("p", 1), 0); + const read = (await store.getChannel("p"))!; + read.stableVersion = "9.9.9"; + expect((await store.getChannel("p"))?.stableVersion).toBe("1.0.0"); + }); + }); + + describe("addStageOutcome", () => { + const live = (): RulePackChannel => ({ + ...channel("p", 1), + rolloutSequence: 1, + rollout: { + rolloutId: "p#1", + baselineVersion: "1.0.0", + candidateVersion: "1.1.0", + stagesBasisPoints: [5_000, 10_000], + stageIndex: 0, + state: "active", + allowTenants: [], + denyTenants: [], + gate: { minSamples: 1, maxFailureRatio: 0, autoRollback: false }, + stageStats: { succeeded: 0, failed: 0 }, + startedAt: 0, + updatedAt: 0, + events: [], + }, + }); + + it("counts toward the matching live stage and bumps the revision", async () => { + await store.putChannel(live(), 0); + const updated = await store.addStageOutcome("p", "p#1", 0, "failed", 7); + expect(updated).toMatchObject({ + revision: 2, + updatedAt: 7, + rollout: { stageStats: { succeeded: 0, failed: 1 } }, + }); + expect(await store.getChannel("p")).toEqual(updated); + }); + + it("ignores unknown packs, other rollouts, other stages and ended rollouts", async () => { + expect( + await store.addStageOutcome("p", "p#1", 0, "failed", 1), + ).toBeUndefined(); + await store.putChannel(live(), 0); + expect( + await store.addStageOutcome("p", "p#0", 0, "failed", 1), + ).toBeUndefined(); + expect( + await store.addStageOutcome("p", "p#1", 1, "failed", 1), + ).toBeUndefined(); + const ended = live(); + ended.revision = 2; + ended.rollout!.state = "rolled_back"; + await store.putChannel(ended, 1); + expect( + await store.addStageOutcome("p", "p#1", 0, "failed", 1), + ).toBeUndefined(); + expect((await store.getChannel("p"))!.revision).toBe(2); + }); + + it("never loses a concurrent increment", async () => { + await store.putChannel(live(), 0); + await Promise.all( + Array.from({ length: 30 }, (_, i) => + store.addStageOutcome( + "p", + "p#1", + 0, + i % 3 ? "succeeded" : "failed", + i, + ), + ), + ); + expect(await store.getChannel("p")).toMatchObject({ + revision: 31, + rollout: { stageStats: { succeeded: 20, failed: 10 } }, + }); + }); + }); + + describe("pins", () => { + it("creates once and reports later creates as existing", async () => { + expect(await store.createPin(pin("t1", "s1"))).toBe("written"); + const second = { ...pin("t1", "s1"), fingerprint: "other" }; + expect(await store.createPin(second)).toBe("exists"); + expect((await store.getPin("t1", "s1"))?.fingerprint).toBe( + "soroban-core@1.0.0", + ); + }); + + it("isolates tenants that share a scan id", async () => { + await store.createPin(pin("tenant-a", "scan-1")); + expect(await store.getPin("tenant-b", "scan-1")).toBeUndefined(); + expect(await store.createPin(pin("tenant-b", "scan-1"))).toBe("written"); + expect(await store.setPinOutcome("tenant-b", "scan-1", "failed", 5)).toBe( + "written", + ); + expect( + (await store.getPin("tenant-a", "scan-1"))?.outcome, + ).toBeUndefined(); + }); + + it("sets an outcome at most once", async () => { + expect(await store.setPinOutcome("t1", "s1", "failed", 1)).toBe( + "not_found", + ); + await store.createPin(pin("t1", "s1")); + expect(await store.setPinOutcome("t1", "s1", "succeeded", 2)).toBe( + "written", + ); + expect(await store.setPinOutcome("t1", "s1", "failed", 3)).toBe( + "already_recorded", + ); + expect(await store.getPin("t1", "s1")).toMatchObject({ + outcome: "succeeded", + outcomeAt: 2, + }); + }); + + it("records exactly one of several concurrent outcomes", async () => { + await store.createPin(pin("t1", "s1")); + const results = await Promise.all( + Array.from({ length: 6 }, (_, i) => + store.setPinOutcome("t1", "s1", i % 2 ? "failed" : "succeeded", i), + ), + ); + expect(results.filter((r) => r === "written")).toHaveLength(1); + }); + + it("deletes only pins created before the cutoff", async () => { + await store.createPin(pin("t1", "old", 10)); + await store.createPin(pin("t2", "edge", 100)); + await store.createPin(pin("t2", "new", 200)); + expect(await store.deletePinsBefore(100)).toBe(1); + expect(await store.getPin("t1", "old")).toBeUndefined(); + expect(await store.getPin("t2", "edge")).toBeDefined(); + expect(await store.deletePinsBefore(100)).toBe(0); + }); + }); +}); + +describe("FileRulePackRolloutStore", () => { + let dir: string; + + beforeEach(async () => { + dir = await fs.mkdtemp(path.join(os.tmpdir(), "rule-pack-file-")); + tmpDirs.push(dir); + }); + + it("rejects an empty root directory", () => { + expect(() => new FileRulePackRolloutStore(" ")).toThrow(TypeError); + }); + + it("persists across instances", async () => { + await new FileRulePackRolloutStore(dir).putChannel(channel("p", 1), 0); + await new FileRulePackRolloutStore(dir).createPin(pin("t1", "s1")); + const reopened = new FileRulePackRolloutStore(dir); + expect((await reopened.getChannel("p"))?.revision).toBe(1); + expect((await reopened.getPin("t1", "s1"))?.scanId).toBe("s1"); + }); + + it("keeps crafted identifiers inside the root directory", async () => { + const store = new FileRulePackRolloutStore(dir); + await store.createPin(pin("../../escape", "../../../etc/passwd")); + const walk = async (d: string): Promise => { + const entries = await fs.readdir(d, { withFileTypes: true }); + const nested = await Promise.all( + entries.map((e) => + e.isDirectory() ? walk(path.join(d, e.name)) : [path.join(d, e.name)], + ), + ); + return nested.flat(); + }; + const files = await walk(dir); + expect(files).toHaveLength(1); + expect(path.relative(dir, files[0])).toMatch( + /^pins\/[0-9a-f]{64}\/[0-9a-f]{64}\.json$/, + ); + }); + + it("writes files readable only by the owner", async () => { + const store = new FileRulePackRolloutStore(dir); + await store.putChannel(channel("p", 1), 0); + const [name] = await fs.readdir(path.join(dir, "channels")); + const stat = await fs.stat(path.join(dir, "channels", name)); + expect(stat.mode & 0o077).toBe(0); + }); + + it("throws on a corrupt record instead of treating it as missing", async () => { + const store = new FileRulePackRolloutStore(dir); + await store.putChannel(channel("p", 1), 0); + const [name] = await fs.readdir(path.join(dir, "channels")); + await fs.writeFile(path.join(dir, "channels", name), "{not json"); + await expect(store.getChannel("p")).rejects.toThrow(); + await expect(store.putChannel(channel("p", 1), 0)).rejects.toThrow(); + }); + + it("throws when a file holds a different key's record", async () => { + const store = new FileRulePackRolloutStore(dir); + await store.putChannel(channel("a", 1), 0); + await store.putChannel(channel("b", 1), 0); + const names = await fs.readdir(path.join(dir, "channels")); + const contents = await Promise.all( + names.map((n) => fs.readFile(path.join(dir, "channels", n), "utf8")), + ); + await fs.writeFile(path.join(dir, "channels", names[0]), contents[1]); + await expect(store.listChannels()).resolves.toHaveLength(2); + const victim = JSON.parse(contents[0]).packId as string; + await expect(store.getChannel(victim)).rejects.toThrow(/does not match/); + }); +}); diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-store.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-store.ts new file mode 100644 index 00000000..2411cd21 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-store.ts @@ -0,0 +1,433 @@ +/** + * Persistence boundary for rule-pack channels and scan pins. + * + * Channels are platform-scoped and keyed by pack id. Writes are optimistic: + * `putChannel` only succeeds when the stored revision still equals the one the + * caller read, so two operators (or an operator and the auto-rollback path) + * can never silently overwrite each other. + * + * Pins are tenant-scoped and keyed by `(tenantId, scanId)`. Every pin read and + * write takes the tenant id explicitly, and implementations must never resolve + * a pin across tenants, so one tenant cannot read or alter another's pin even + * when scan ids collide. Pins are write-once (`createPin`), and their outcome + * is set at most once, which makes retries and duplicate reports idempotent. + * + * Implementations throw plain errors on storage failure; the service maps them + * to the retryable STORE_UNAVAILABLE code. + */ + +import * as crypto from "crypto"; +import { promises as fs } from "fs"; +import * as path from "path"; +import { applyStageOutcome } from "./rule-pack-rollout-planner"; +import { + RulePackChannel, + ScanOutcome, + ScanRulePackPin, +} from "./rule-pack-rollout-types"; + +export type ChannelWriteResult = "written" | "conflict"; +export type PinCreateResult = "written" | "exists"; +export type PinOutcomeResult = "written" | "already_recorded" | "not_found"; + +export interface RulePackRolloutStore { + getChannel(packId: string): Promise; + /** + * Write `channel` when the stored revision equals `expectedRevision` + * (`0` means the channel must not exist yet). + */ + putChannel( + channel: RulePackChannel, + expectedRevision: number, + ): Promise; + /** + * Atomically count one outcome toward the live rollout's current stage when + * `rolloutId` and `stageIndex` still match (see `applyStageOutcome`), + * bumping the revision. Returns the updated channel, or `undefined` when it + * did not match. A database store implements this as one conditional + * `UPDATE`; it must never be a client-side read-modify-write, because + * outcome traffic is far heavier than operator traffic. + */ + addStageOutcome( + packId: string, + rolloutId: string, + stageIndex: number, + outcome: ScanOutcome, + at: number, + ): Promise; + /** All channels, sorted by pack id. */ + listChannels(): Promise; + getPin( + tenantId: string, + scanId: string, + ): Promise; + /** Store `pin` unless a pin already exists for its tenant and scan. */ + createPin(pin: ScanRulePackPin): Promise; + /** Set the outcome once; later calls leave the first outcome in place. */ + setPinOutcome( + tenantId: string, + scanId: string, + outcome: ScanOutcome, + at: number, + ): Promise; + /** Remove pins created before `cutoff`; returns how many were removed. */ + deletePinsBefore(cutoff: number): Promise; +} + +export class InMemoryRulePackRolloutStore implements RulePackRolloutStore { + // Stored serialized so callers can never mutate persisted state by reference. + private readonly channels: Map = new Map(); + private readonly pins: Map = new Map(); + + async getChannel(packId: string): Promise { + const raw = this.channels.get(packId); + return raw === undefined ? undefined : (JSON.parse(raw) as RulePackChannel); + } + + async putChannel( + channel: RulePackChannel, + expectedRevision: number, + ): Promise { + const raw = this.channels.get(channel.packId); + const current = + raw === undefined ? 0 : (JSON.parse(raw) as RulePackChannel).revision; + if (current !== expectedRevision) { + return "conflict"; + } + this.channels.set(channel.packId, JSON.stringify(channel)); + return "written"; + } + + async addStageOutcome( + packId: string, + rolloutId: string, + stageIndex: number, + outcome: ScanOutcome, + at: number, + ): Promise { + const raw = this.channels.get(packId); + if (raw === undefined) { + return undefined; + } + const next = applyStageOutcome( + JSON.parse(raw) as RulePackChannel, + rolloutId, + stageIndex, + outcome, + at, + ); + if (next) { + this.channels.set(packId, JSON.stringify(next)); + } + return next; + } + + async listChannels(): Promise { + return [...this.channels.keys()] + .sort() + .map((key) => JSON.parse(this.channels.get(key)!) as RulePackChannel); + } + + async getPin( + tenantId: string, + scanId: string, + ): Promise { + const raw = this.pins.get(pinKey(tenantId, scanId)); + return raw === undefined ? undefined : (JSON.parse(raw) as ScanRulePackPin); + } + + async createPin(pin: ScanRulePackPin): Promise { + const key = pinKey(pin.tenantId, pin.scanId); + if (this.pins.has(key)) { + return "exists"; + } + this.pins.set(key, JSON.stringify(pin)); + return "written"; + } + + async setPinOutcome( + tenantId: string, + scanId: string, + outcome: ScanOutcome, + at: number, + ): Promise { + const key = pinKey(tenantId, scanId); + const raw = this.pins.get(key); + if (raw === undefined) { + return "not_found"; + } + const pin = JSON.parse(raw) as ScanRulePackPin; + if (pin.outcome !== undefined) { + return "already_recorded"; + } + this.pins.set(key, JSON.stringify({ ...pin, outcome, outcomeAt: at })); + return "written"; + } + + async deletePinsBefore(cutoff: number): Promise { + let removed = 0; + for (const [key, raw] of this.pins) { + if ((JSON.parse(raw) as ScanRulePackPin).pinnedAt < cutoff) { + this.pins.delete(key); + removed++; + } + } + return removed; + } + + pinCount(): number { + return this.pins.size; + } +} + +/** + * Durable store writing one JSON file per channel and per scan pin under + * `rootDir`. + * + * Paths are derived from SHA-256 digests of the identifiers, never from the + * identifiers themselves, so crafted ids cannot traverse outside `rootDir`, and + * each tenant's pins live in their own directory. Files are written to a + * temporary name and renamed into place, so a crash mid-write leaves either the + * old record or the new one, never a torn file. + * + * Conditional writes are serialized per key within this process. Deployments + * that run several API processes against shared storage should use a database + * store that performs the revision check as a single conditional update. + */ +export class FileRulePackRolloutStore implements RulePackRolloutStore { + private readonly rootDir: string; + private readonly locks: Map> = new Map(); + + constructor(rootDir: string) { + if (typeof rootDir !== "string" || rootDir.trim() === "") { + throw new TypeError("rootDir must be a non-empty string"); + } + this.rootDir = path.resolve(rootDir); + } + + getChannel(packId: string): Promise { + return this.readChannel(this.channelPath(packId), packId); + } + + putChannel( + channel: RulePackChannel, + expectedRevision: number, + ): Promise { + const file = this.channelPath(channel.packId); + return this.withLock(file, async () => { + const existing = await this.readChannel(file, channel.packId); + if ((existing?.revision ?? 0) !== expectedRevision) { + return "conflict"; + } + await this.writeAtomic(file, channel); + return "written"; + }); + } + + addStageOutcome( + packId: string, + rolloutId: string, + stageIndex: number, + outcome: ScanOutcome, + at: number, + ): Promise { + const file = this.channelPath(packId); + return this.withLock(file, async () => { + const existing = await this.readChannel(file, packId); + const next = + existing && + applyStageOutcome(existing, rolloutId, stageIndex, outcome, at); + if (next) { + await this.writeAtomic(file, next); + } + return next; + }); + } + + async listChannels(): Promise { + const dir = path.join(this.rootDir, "channels"); + const names = await readDirOrEmpty(dir); + const channels: RulePackChannel[] = []; + for (const name of names.filter((n) => n.endsWith(".json")).sort()) { + channels.push( + JSON.parse(await fs.readFile(path.join(dir, name), "utf8")), + ); + } + return channels.sort((a, b) => + a.packId === b.packId ? 0 : a.packId < b.packId ? -1 : 1, + ); + } + + getPin( + tenantId: string, + scanId: string, + ): Promise { + return this.readPin(this.pinPath(tenantId, scanId), tenantId, scanId); + } + + createPin(pin: ScanRulePackPin): Promise { + const file = this.pinPath(pin.tenantId, pin.scanId); + return this.withLock(file, async () => { + if (await this.readPin(file, pin.tenantId, pin.scanId)) { + return "exists"; + } + await this.writeAtomic(file, pin); + return "written"; + }); + } + + setPinOutcome( + tenantId: string, + scanId: string, + outcome: ScanOutcome, + at: number, + ): Promise { + const file = this.pinPath(tenantId, scanId); + return this.withLock(file, async () => { + const pin = await this.readPin(file, tenantId, scanId); + if (!pin) { + return "not_found"; + } + if (pin.outcome !== undefined) { + return "already_recorded"; + } + await this.writeAtomic(file, { ...pin, outcome, outcomeAt: at }); + return "written"; + }); + } + + async deletePinsBefore(cutoff: number): Promise { + const pinsDir = path.join(this.rootDir, "pins"); + let removed = 0; + for (const tenantDir of (await readDirOrEmpty(pinsDir)).sort()) { + const dir = path.join(pinsDir, tenantDir); + for (const name of (await readDirOrEmpty(dir)).sort()) { + if (!name.endsWith(".json")) { + continue; + } + const file = path.join(dir, name); + const deleted = await this.withLock(file, async () => { + let pin: ScanRulePackPin; + try { + pin = JSON.parse(await fs.readFile(file, "utf8")); + } catch (error) { + if (isNotFound(error)) { + return false; + } + throw error; + } + if (pin.pinnedAt >= cutoff) { + return false; + } + await fs.rm(file, { force: true }); + return true; + }); + if (deleted) { + removed++; + } + } + } + return removed; + } + + private async readChannel( + file: string, + packId: string, + ): Promise { + const channel = await readJson(file); + // Defence in depth: the digest path must hold the record it was asked for. + if (channel && channel.packId !== packId) { + throw new Error("Stored rule-pack channel does not match its key"); + } + return channel; + } + + private async readPin( + file: string, + tenantId: string, + scanId: string, + ): Promise { + const pin = await readJson(file); + if (pin && (pin.tenantId !== tenantId || pin.scanId !== scanId)) { + throw new Error("Stored scan pin does not match its key"); + } + return pin; + } + + private async writeAtomic(file: string, value: unknown): Promise { + await fs.mkdir(path.dirname(file), { recursive: true }); + const tmp = `${file}.${process.pid}.${crypto.randomBytes(6).toString("hex")}.tmp`; + try { + await fs.writeFile(tmp, JSON.stringify(value), { + encoding: "utf8", + mode: 0o600, + }); + await fs.rename(tmp, file); + } catch (error) { + await fs.rm(tmp, { force: true }).catch(() => undefined); + throw error; + } + } + + private channelPath(packId: string): string { + return path.join(this.rootDir, "channels", `${digest(packId)}.json`); + } + + private pinPath(tenantId: string, scanId: string): string { + return path.join( + this.rootDir, + "pins", + digest(tenantId), + `${digest(scanId)}.json`, + ); + } + + private withLock(key: string, task: () => Promise): Promise { + const previous = this.locks.get(key) ?? Promise.resolve(); + const run = previous.then(task, task); + const settled = run.catch(() => undefined); + this.locks.set(key, settled); + void settled.then(() => { + if (this.locks.get(key) === settled) { + this.locks.delete(key); + } + }); + return run; + } +} + +function pinKey(tenantId: string, scanId: string): string { + return `${tenantId}\u0000${scanId}`; +} + +function digest(value: string): string { + return crypto.createHash("sha256").update(value, "utf8").digest("hex"); +} + +async function readJson(file: string): Promise { + let raw: string; + try { + raw = await fs.readFile(file, "utf8"); + } catch (error) { + if (isNotFound(error)) { + return undefined; + } + throw error; + } + return JSON.parse(raw) as T; +} + +async function readDirOrEmpty(dir: string): Promise { + try { + return await fs.readdir(dir); + } catch (error) { + if (isNotFound(error)) { + return []; + } + throw error; + } +} + +function isNotFound(error: unknown): boolean { + return (error as NodeJS.ErrnoException)?.code === "ENOENT"; +} diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-types.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-types.ts new file mode 100644 index 00000000..0656e98b --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-types.ts @@ -0,0 +1,170 @@ +/** + * Shared types for staged rule-pack rollouts. + * + * A rule pack (a versioned bundle of analyzer rules, e.g. `soroban-core`) has + * one stable version. A rollout introduces a candidate version to a growing, + * deterministic cohort of tenants in stages, and ends either `completed` (the + * candidate becomes stable) or `rolled_back` (everyone stays on the baseline). + * + * Rule packs are platform-scoped: the channel record is operator data. The + * tenant-facing surface is `RulePackSelection`, which never names other tenants + * or the allow/deny lists. + */ + +export const ROLLOUT_STATES = [ + "active", + "paused", + "completed", + "rolled_back", +] as const; +export type RolloutState = (typeof ROLLOUT_STATES)[number]; + +export type RolloutAction = + | "started" + | "advanced" + | "paused" + | "resumed" + | "completed" + | "rolled_back" + | "auto_rolled_back"; + +export interface RolloutGate { + /** Candidate scan outcomes needed in a stage before it can pass or fail. */ + minSamples: number; + /** Highest tolerated failed / (failed + succeeded) ratio in a stage (0-1). */ + maxFailureRatio: number; + /** Roll back automatically as soon as a stage breaches the gate. */ + autoRollback: boolean; +} + +/** Operator input to `startRollout`. */ +export interface RulePackRolloutPlan { + packId: string; + candidateVersion: string; + /** + * Cumulative percentage of tenants on the candidate at each stage, strictly + * increasing and ending at 100. Up to two decimals, e.g. `[1, 10, 50, 100]`. + */ + stages: number[]; + /** Tenants that always get the candidate while the rollout is live. */ + allowTenants?: string[]; + /** Tenants that always stay on the baseline while the rollout is live. */ + denyTenants?: string[]; + gate?: Partial; +} + +export interface RolloutStageStats { + succeeded: number; + failed: number; +} + +export interface RolloutEvent { + at: number; + action: RolloutAction; + stageIndex: number; + state: RolloutState; + /** Operator-supplied reason, truncated; never contains scan content. */ + reason?: string; +} + +export interface RulePackRollout { + /** `${packId}#${sequence}`; unique per pack. */ + rolloutId: string; + baselineVersion: string; + candidateVersion: string; + /** Stage thresholds in basis points (1/100 of a percent), last is 10000. */ + stagesBasisPoints: number[]; + stageIndex: number; + state: RolloutState; + /** Sorted and unique. */ + allowTenants: string[]; + /** Sorted and unique. */ + denyTenants: string[]; + gate: RolloutGate; + /** Candidate outcomes for scans pinned during the current stage. */ + stageStats: RolloutStageStats; + startedAt: number; + updatedAt: number; + endedAt?: number; + /** Most recent events, oldest first, capped by `maxEventsPerRollout`. */ + events: RolloutEvent[]; +} + +export interface RulePackChannel { + packId: string; + stableVersion: string; + /** Optimistic-concurrency revision, incremented on every write. */ + revision: number; + /** Number of rollouts ever started for this pack. */ + rolloutSequence: number; + updatedAt: number; + /** Current rollout, or the most recent finished one. */ + rollout?: RulePackRollout; +} + +export type SelectionChannel = "stable" | "baseline" | "candidate"; + +export type SelectionReason = + /** No live rollout (none started, completed, rolled back, or disabled). */ + "no_active_rollout" | "allow_list" | "deny_list" | "cohort"; + +/** Rule-pack version chosen for one tenant. Safe to return to that tenant. */ +export interface RulePackSelection { + packId: string; + version: string; + channel: SelectionChannel; + reason: SelectionReason; + rolloutId?: string; + stageIndex?: number; +} + +export type ScanOutcome = "succeeded" | "failed"; + +/** + * Versions fixed for one scan the first time it resolves rule packs, so + * retries of that scan run the same rules even if the rollout moves on. + */ +export interface ScanRulePackPin { + tenantId: string; + scanId: string; + /** Every pack the scan asked for, sorted and unique. */ + packIds: string[]; + /** Managed packs only, sorted by `packId`. */ + selections: RulePackSelection[]; + /** `pack@version` pairs joined by `,`, in `packId` order. */ + fingerprint: string; + pinnedAt: number; + outcome?: ScanOutcome; + outcomeAt?: number; +} + +export type RulePackRolloutErrorCode = + | "ROLLOUT_DISABLED" + | "INVALID_INPUT" + | "INVALID_PLAN" + | "INVALID_TRANSITION" + | "PACK_NOT_FOUND" + | "PACK_ALREADY_REGISTERED" + | "ROLLOUT_IN_PROGRESS" + | "GATE_NOT_MET" + | "PIN_MISMATCH" + | "CONFLICT" + | "STORE_UNAVAILABLE"; + +const RETRYABLE_CODES: ReadonlySet = new Set([ + "CONFLICT", + "STORE_UNAVAILABLE", +]); + +export class RulePackRolloutError extends Error { + readonly code: RulePackRolloutErrorCode; + /** True when the same call may succeed if retried unchanged. */ + readonly retryable: boolean; + + constructor(code: RulePackRolloutErrorCode, message: string) { + super(message); + this.name = "RulePackRolloutError"; + this.code = code; + this.retryable = RETRYABLE_CODES.has(code); + } +} diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout-worker.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout-worker.ts new file mode 100644 index 00000000..717973c5 --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout-worker.ts @@ -0,0 +1,67 @@ +/** + * Worker-side adapter for staged rule-pack rollouts. + * + * `runScanWithRulePacks` wraps one scan attempt: it resolves (or re-reads) + * the scan's pinned rule-pack versions, runs the scan with them, and reports + * the outcome. A failed attempt is only reported when the queue will not retry + * it (`isFinalAttempt`), so a transient failure that a retry recovers from + * never counts against a candidate. + * + * Reporting is best-effort: a failure to record an outcome is logged by the + * service but never changes the scan's own result or error. + */ + +import { + RulePackRolloutService, + ScanRulePackResolution, +} from "./rule-pack-rollout.service"; + +export interface ScanAttempt { + tenantId: string; + scanId: string; + packIds: string[]; + /** True when this attempt's failure will not be retried. */ + isFinalAttempt: boolean; +} + +export async function runScanWithRulePacks( + service: RulePackRolloutService, + attempt: ScanAttempt, + scan: (resolution: ScanRulePackResolution) => Promise, +): Promise { + // Resolution errors propagate unchanged: STORE_UNAVAILABLE is retryable, + // so the queue's normal retry policy applies before any analysis runs. + const resolution = await service.resolveForScan({ + tenantId: attempt.tenantId, + scanId: attempt.scanId, + packIds: attempt.packIds, + }); + + let result: T; + try { + result = await scan(resolution); + } catch (error) { + if (attempt.isFinalAttempt) { + await report(service, attempt, "failed"); + } + throw error; + } + await report(service, attempt, "succeeded"); + return result; +} + +async function report( + service: RulePackRolloutService, + attempt: ScanAttempt, + outcome: "succeeded" | "failed", +): Promise { + try { + await service.recordScanOutcome({ + tenantId: attempt.tenantId, + scanId: attempt.scanId, + outcome, + }); + } catch { + // Already counted and logged by the service; the scan result stands. + } +} diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout.integration.spec.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout.integration.spec.ts new file mode 100644 index 00000000..318c925f --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout.integration.spec.ts @@ -0,0 +1,405 @@ +/** + * Integration tests across the rule-pack rollout boundaries: operator-facing + * service, worker adapter, and the durable file store, including service + * restarts over the same storage. + */ + +import { createHash } from "crypto"; +import { promises as fs, readFileSync } from "fs"; +import * as os from "os"; +import * as path from "path"; +import { + RulePackRolloutLogFields, + RulePackRolloutLogLevel, + RulePackRolloutLogger, +} from "./rule-pack-rollout-metrics"; +import { FileRulePackRolloutStore } from "./rule-pack-rollout-store"; +import { + RulePackRolloutService, + ScanRulePackResolution, +} from "./rule-pack-rollout.service"; +import { RulePackRolloutPlan } from "./rule-pack-rollout-types"; +import { runScanWithRulePacks } from "./rule-pack-rollout-worker"; + +interface RolloutFixture { + stableVersion: string; + plan: RulePackRolloutPlan; + otherManagedPacks: { packId: string; stableVersion: string }[]; + scan: { + tenantId: string; + repository: string; + commit: string; + scanId: string; + packIds: string[]; + }; +} + +const fixture = JSON.parse( + readFileSync( + path.join(__dirname, "__fixtures__", "soroban-core-rollout.json"), + "utf8", + ), +) as RolloutFixture; + +class CapturingLogger implements RulePackRolloutLogger { + readonly lines: string[] = []; + log( + level: RulePackRolloutLogLevel, + event: string, + fields: RulePackRolloutLogFields, + ): void { + this.lines.push(JSON.stringify({ level, event, ...fields })); + } +} + +/** + * Stand-in analyzer: findings are a pure function of the commit and the + * resolved rule-pack fingerprint, as real analysis must be. + */ +function analyze(commit: string, resolution: ScanRulePackResolution) { + const digest = createHash("sha256") + .update(`${commit}\u0000${resolution.fingerprint}`) + .digest("hex"); + return { + rulePacks: resolution.fingerprint, + unmanaged: resolution.unmanagedPackIds, + findingsHash: digest.slice(0, 16), + }; +} + +let rootDir: string; +let logger: CapturingLogger; +let now: number; + +/** A fresh service over the same directory, i.e. a restarted process. */ +function boot(config = {}) { + return new RulePackRolloutService({ + store: new FileRulePackRolloutStore(rootDir), + logger, + clock: () => now, + config, + }); +} + +async function operatorSetup(): Promise { + const operator = boot(); + await operator.registerPack(fixture.plan.packId, fixture.stableVersion); + for (const pack of fixture.otherManagedPacks) { + await operator.registerPack(pack.packId, pack.stableVersion); + } + await operator.startRollout(fixture.plan, "storage-rent rules"); + return operator; +} + +/** Run successful dogfood scans until the current stage gate passes. */ +async function satisfyGate(service: RulePackRolloutService, label: string) { + for (let i = 0; i < fixture.plan.gate!.minSamples!; i++) { + await runScanWithRulePacks( + service, + { + tenantId: "gasguard-dogfood", + scanId: `dogfood-${label}-${i}`, + packIds: [fixture.plan.packId], + isFinalAttempt: true, + }, + async () => "ok", + ); + } +} + +beforeEach(async () => { + rootDir = await fs.mkdtemp(path.join(os.tmpdir(), "rule-pack-rollout-it-")); + logger = new CapturingLogger(); + now = 1_700_000_000_000; +}); + +afterEach(async () => { + await fs.rm(rootDir, { recursive: true, force: true }); +}); + +describe("realistic repository: soroban AMM scanned during a soroban-core rollout", () => { + it("moves the tenant to the candidate at its stage, deterministically and durably", async () => { + const operator = await operatorSetup(); + const { scan } = fixture; + const scanAt = async (scanId: string) => + runScanWithRulePacks( + boot(), + { + tenantId: scan.tenantId, + scanId, + packIds: scan.packIds, + isFinalAttempt: true, + }, + async (resolution) => analyze(scan.commit, resolution), + ); + + // Stage 0 (1%): acme-defi's bucket (3188) is outside the cohort. + const baseline = await scanAt(scan.scanId); + expect(baseline).toEqual({ + rulePacks: "soroban-core@1.4.2,soroban-security@2.0.1", + unmanaged: ["evm-gas"], + findingsHash: expect.any(String), + }); + // The same commit, configuration and versions give the same findings. + expect(await scanAt(`${scan.scanId}:rerun`)).toEqual(baseline); + + // Allow and deny lists win over the cohort. + expect( + (await operator.resolve("gasguard-dogfood", "soroban-core"))!.version, + ).toBe("1.5.0"); + expect( + (await operator.resolve("regulated-bank-eu", "soroban-core"))!.reason, + ).toBe("deny_list"); + + await satisfyGate(operator, "s0"); + await boot().advance("soroban-core"); // 10%: still outside + expect((await scanAt(`${scan.scanId}:s1`)).rulePacks).toBe( + baseline.rulePacks, + ); + + await satisfyGate(operator, "s1"); + await boot().advance("soroban-core"); // 50%: inside + const candidate = await scanAt(`${scan.scanId}:s2`); + expect(candidate.rulePacks).toBe( + "soroban-core@1.5.0,soroban-security@2.0.1", + ); + expect(candidate.findingsHash).not.toBe(baseline.findingsHash); + expect(await scanAt(`${scan.scanId}:s2-rerun`)).toEqual(candidate); + + // The original scan id is still pinned to its first resolution. + expect(await scanAt(scan.scanId)).toEqual(baseline); + + await satisfyGate(operator, "s2"); + await boot().advance("soroban-core"); // 100% + await satisfyGate(operator, "s3"); + const done = await boot().advance("soroban-core"); + expect(done).toMatchObject({ + stableVersion: "1.5.0", + rollout: { state: "completed" }, + }); + // After completion even the denied tenant is on the new stable. + expect( + (await boot().resolve("regulated-bank-eu", "soroban-core"))!.version, + ).toBe("1.5.0"); + + const history = (await boot().getChannel("soroban-core"))!.rollout!.events; + expect(history.map((e) => e.action)).toEqual([ + "started", + "advanced", + "advanced", + "advanced", + "completed", + ]); + }); +}); + +describe("worker retries", () => { + it("counts a transient failure that a retry recovers from as one success", async () => { + await operatorSetup(); + const attempt = (isFinalAttempt: boolean) => ({ + tenantId: "gasguard-dogfood", + scanId: "flaky-scan", + packIds: ["soroban-core"], + isFinalAttempt, + }); + + await expect( + runScanWithRulePacks(boot(), attempt(false), async () => { + throw new Error("RPC timeout"); + }), + ).rejects.toThrow("RPC timeout"); + expect( + (await boot().getChannel("soroban-core"))!.rollout!.stageStats, + ).toEqual({ + succeeded: 0, + failed: 0, + }); + + await runScanWithRulePacks(boot(), attempt(true), async () => "ok"); + expect( + (await boot().getChannel("soroban-core"))!.rollout!.stageStats, + ).toEqual({ + succeeded: 1, + failed: 0, + }); + }); + + it("counts a failure on the final attempt and rethrows the scan's error", async () => { + await operatorSetup(); + const error = new Error("analyzer crashed"); + await expect( + runScanWithRulePacks( + boot(), + { + tenantId: "gasguard-dogfood", + scanId: "s1", + packIds: ["soroban-core"], + isFinalAttempt: true, + }, + async () => { + throw error; + }, + ), + ).rejects.toBe(error); + expect( + (await boot().getChannel("soroban-core"))!.rollout!.stageStats.failed, + ).toBe(1); + }); + + it("keeps the scan result when recording the outcome fails", async () => { + await operatorSetup(); + const service = boot(); + jest + .spyOn(service, "recordScanOutcome") + .mockRejectedValueOnce(new Error("store down")); + await expect( + runScanWithRulePacks( + service, + { + tenantId: "t1", + scanId: "s1", + packIds: ["soroban-core"], + isFinalAttempt: true, + }, + async () => "findings", + ), + ).resolves.toBe("findings"); + }); + + it("does not run the scan when resolution fails, so the queue can retry", async () => { + await operatorSetup(); + const scan = jest.fn(); + await expect( + runScanWithRulePacks( + boot(), + { + tenantId: "bad tenant", + scanId: "s1", + packIds: ["soroban-core"], + isFinalAttempt: true, + }, + scan, + ), + ).rejects.toMatchObject({ code: "INVALID_INPUT" }); + expect(scan).not.toHaveBeenCalled(); + }); +}); + +describe("concurrent workers sharing one store", () => { + it("counts every outcome exactly once", async () => { + await operatorSetup(); + const store = new FileRulePackRolloutStore(rootDir); + const workers = Array.from( + { length: 4 }, + () => new RulePackRolloutService({ store, logger, clock: () => now }), + ); + const scans = Array.from({ length: 24 }, (_, i) => i); + await Promise.all( + scans.map((i) => + runScanWithRulePacks( + workers[i % workers.length], + { + tenantId: "gasguard-dogfood", + scanId: `parallel-${i}`, + packIds: ["soroban-core"], + isFinalAttempt: true, + }, + async () => "ok", + ), + ), + ); + // Duplicate reports from a second worker are ignored. + await Promise.all( + scans.map((i) => + workers[(i + 1) % workers.length].recordScanOutcome({ + tenantId: "gasguard-dogfood", + scanId: `parallel-${i}`, + outcome: "failed", + }), + ), + ); + expect( + (await boot().getChannel("soroban-core"))!.rollout!.stageStats, + ).toEqual({ + succeeded: 24, + failed: 0, + }); + }); +}); + +describe("regression: auto-rollback survives a restart", () => { + it("returns new scans to the baseline after a bad candidate, even in a new process", async () => { + await operatorSetup(); + const minSamples = fixture.plan.gate!.minSamples!; + let rolledBack: string[] = []; + for (let i = 0; i < minSamples; i++) { + const worker = boot(); + await expect( + runScanWithRulePacks( + worker, + { + tenantId: "gasguard-dogfood", + scanId: `bad-${i}`, + packIds: ["soroban-core"], + isFinalAttempt: true, + }, + async () => { + throw new Error("rule crashed on storage-rent check"); + }, + ), + ).rejects.toThrow(); + const channel = await worker.getChannel("soroban-core"); + if (channel!.rollout!.state === "rolled_back") { + rolledBack = [`bad-${i}`]; + break; + } + } + expect(rolledBack).toEqual([`bad-${minSamples - 1}`]); + + const restarted = boot(); + expect(await restarted.getChannel("soroban-core")).toMatchObject({ + stableVersion: "1.4.2", + rollout: { state: "rolled_back", endedAt: now }, + }); + const fresh = await restarted.resolveForScan({ + tenantId: "gasguard-dogfood", + scanId: "after-rollback", + packIds: fixture.scan.packIds, + }); + expect(fresh.fingerprint).toBe("soroban-core@1.4.2,soroban-security@2.0.1"); + expect( + logger.lines.some((l) => + l.includes('"event":"rollout_auto_rolled_back"'), + ), + ).toBe(true); + }); +}); + +describe("logs", () => { + it("never contain scan content, store errors or credentials", async () => { + const operator = await operatorSetup(); + await operator.advance("soroban-core", { + force: true, + reason: "token=ghp_supersecret".replace( + "token=", + "approved by release mgr ", + ), + }); + await runScanWithRulePacks( + boot(), + { + tenantId: "acme-defi", + scanId: "s1", + packIds: fixture.scan.packIds, + isFinalAttempt: true, + }, + async () => "contract Amm { fn swap() {} }", + ); + const all = logger.lines.join("\n"); + expect(all).not.toContain("contract Amm"); + expect(all).not.toContain(rootDir); + for (const line of logger.lines) { + expect(() => JSON.parse(line)).not.toThrow(); + } + }); +}); diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout.service.spec.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout.service.spec.ts new file mode 100644 index 00000000..790c341a --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout.service.spec.ts @@ -0,0 +1,785 @@ +import { + RulePackRolloutLogFields, + RulePackRolloutLogLevel, + RulePackRolloutLogger, +} from "./rule-pack-rollout-metrics"; +import { selectForTenant, tenantBucket } from "./rule-pack-rollout-planner"; +import { + ChannelWriteResult, + InMemoryRulePackRolloutStore, + RulePackRolloutStore, +} from "./rule-pack-rollout-store"; +import { + RulePackRolloutService, + RulePackRolloutServiceOptions, +} from "./rule-pack-rollout.service"; +import { + RulePackChannel, + RulePackRolloutError, + RulePackRolloutPlan, +} from "./rule-pack-rollout-types"; + +class CapturingLogger implements RulePackRolloutLogger { + readonly entries: { + level: RulePackRolloutLogLevel; + event: string; + fields: RulePackRolloutLogFields; + }[] = []; + log( + level: RulePackRolloutLogLevel, + event: string, + fields: RulePackRolloutLogFields, + ): void { + this.entries.push({ level, event, fields }); + } + events(): string[] { + return this.entries.map((e) => e.event); + } +} + +/** Store whose methods can be made to throw or to report conflicts. */ +class FaultyStore extends InMemoryRulePackRolloutStore { + failAll = false; + conflictsRemaining = 0; + + async getChannel(packId: string) { + this.maybeFail(); + return super.getChannel(packId); + } + async putChannel( + channel: RulePackChannel, + expectedRevision: number, + ): Promise { + this.maybeFail(); + if (this.conflictsRemaining > 0) { + this.conflictsRemaining--; + return "conflict"; + } + return super.putChannel(channel, expectedRevision); + } + async getPin(tenantId: string, scanId: string) { + this.maybeFail(); + return super.getPin(tenantId, scanId); + } + private maybeFail(): void { + if (this.failAll) { + throw new Error("connect ECONNREFUSED postgres://admin:hunter2@db:5432"); + } + } +} + +const PLAN: RulePackRolloutPlan = { + packId: "soroban-core", + candidateVersion: "1.5.0", + stages: [10, 50, 100], + gate: { minSamples: 3, maxFailureRatio: 0.34, autoRollback: true }, +}; + +/** First tenant (in index order) whose bucket satisfies `predicate`. */ +function tenantWhere( + predicate: (bucket: number) => boolean, + version = "1.5.0", +): string { + for (let i = 0; ; i++) { + const id = `tenant-${i}`; + if (predicate(tenantBucket("soroban-core", version, id))) { + return id; + } + } +} + +const IN_FIRST_STAGE = tenantWhere((b) => b < 1_000); +const IN_SECOND_STAGE_ONLY = tenantWhere((b) => b >= 1_000 && b < 5_000); + +function setup(overrides: Partial = {}) { + let now = 1_000; + const store = new FaultyStore(); + const logger = new CapturingLogger(); + const service = new RulePackRolloutService({ + store, + logger, + clock: () => now, + ...overrides, + }); + return { + store, + logger, + service, + tick: (ms = 1) => (now += ms), + }; +} + +async function expectCode(promise: Promise, code: string) { + await expect(promise).rejects.toMatchObject({ + name: "RulePackRolloutError", + code, + }); +} + +/** Record `n` candidate outcomes for fresh scans of first-stage tenants. */ +async function feed( + service: RulePackRolloutService, + outcomes: ("succeeded" | "failed")[], + tenant = IN_FIRST_STAGE, +) { + const results = []; + for (const outcome of outcomes) { + const scanId = `scan-${Math.random().toString(36).slice(2)}`; + await service.resolveForScan({ + tenantId: tenant, + scanId, + packIds: ["soroban-core"], + }); + results.push( + await service.recordScanOutcome({ tenantId: tenant, scanId, outcome }), + ); + } + return results; +} + +describe("registerPack", () => { + it("registers a pack and is idempotent for the same version", async () => { + const { service } = setup(); + const first = await service.registerPack("soroban-core", "1.4.2"); + expect(first).toMatchObject({ + stableVersion: "1.4.2", + revision: 1, + rolloutSequence: 0, + }); + expect(await service.registerPack("soroban-core", "1.4.2")).toEqual(first); + }); + + it("refuses to change the stable version outside a rollout", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await expectCode( + service.registerPack("soroban-core", "2.0.0"), + "PACK_ALREADY_REGISTERED", + ); + }); + + it.each([ + ["Bad Pack", "1.0.0"], + ["soroban-core", "1.0"], + ["soroban-core", ""], + ])("rejects invalid input (%p, %p)", async (packId, version) => { + const { service } = setup(); + await expectCode(service.registerPack(packId, version), "INVALID_INPUT"); + }); + + it("settles on the winner when two registrations race", async () => { + // A competing process registers between our read and our write. + class RacingStore extends InMemoryRulePackRolloutStore { + competitorVersion = "1.4.2"; + async putChannel(channel: RulePackChannel, expected: number) { + if (expected === 0 && !(await this.getChannel(channel.packId))) { + await super.putChannel( + { + ...channel, + stableVersion: this.competitorVersion, + updatedAt: -1, + }, + 0, + ); + } + return super.putChannel(channel, expected); + } + } + const same = new RacingStore(); + const service = new RulePackRolloutService({ + store: same, + logger: new CapturingLogger(), + }); + await expect( + service.registerPack("soroban-core", "1.4.2"), + ).resolves.toMatchObject({ + stableVersion: "1.4.2", + updatedAt: -1, + }); + + const different = new RacingStore(); + different.competitorVersion = "1.4.3"; + const loser = new RulePackRolloutService({ + store: different, + logger: new CapturingLogger(), + }); + await expectCode( + loser.registerPack("soroban-core", "1.4.2"), + "PACK_ALREADY_REGISTERED", + ); + }); +}); + +describe("rollout lifecycle", () => { + it("starts, advances through gated stages and completes", async () => { + const { service, logger } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + const started = await service.startRollout(PLAN, "storage-rent rules"); + expect(started.rollout).toMatchObject({ + rolloutId: "soroban-core#1", + state: "active", + stageIndex: 0, + }); + expect(started.rollout!.events[0].reason).toBe("storage-rent rules"); + + await expectCode(service.advance("soroban-core"), "GATE_NOT_MET"); + await feed(service, ["succeeded", "succeeded", "succeeded"]); + expect((await service.advance("soroban-core")).rollout!.stageIndex).toBe(1); + + await feed( + service, + ["succeeded", "succeeded", "succeeded"], + IN_SECOND_STAGE_ONLY, + ); + await service.advance("soroban-core"); + await feed(service, ["succeeded", "succeeded", "failed"]); + const done = await service.advance("soroban-core"); + expect(done).toMatchObject({ + stableVersion: "1.5.0", + rollout: { state: "completed" }, + }); + expect(await service.resolve("anyone", "soroban-core")).toMatchObject({ + version: "1.5.0", + channel: "stable", + }); + expect(logger.events()).toEqual( + expect.arrayContaining([ + "rollout_started", + "rollout_advanced", + "rollout_completed", + ]), + ); + expect(service.metrics.snapshot().transitions).toEqual({ + started: 1, + advanced: 2, + completed: 1, + }); + }); + + it("allows a new rollout once the previous one has ended", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.startRollout(PLAN); + await expectCode( + service.startRollout({ ...PLAN, candidateVersion: "1.6.0" }), + "ROLLOUT_IN_PROGRESS", + ); + await service.rollback("soroban-core", "bad findings"); + const next = await service.startRollout({ + ...PLAN, + candidateVersion: "1.5.1", + }); + expect(next.rollout).toMatchObject({ + rolloutId: "soroban-core#2", + baselineVersion: "1.4.2", + }); + }); + + it("forces an advance past the gate only with a reason, logging a warning", async () => { + const { service, logger } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.startRollout(PLAN); + await expectCode( + service.advance("soroban-core", { force: true }), + "INVALID_INPUT", + ); + await expectCode( + service.advance("soroban-core", { force: true, reason: " " }), + "INVALID_INPUT", + ); + const forced = await service.advance("soroban-core", { + force: true, + reason: "hotfix SEC-12", + }); + expect(forced.rollout!.stageIndex).toBe(1); + expect(logger.entries.at(-1)).toMatchObject({ + level: "warn", + event: "rollout_advanced", + fields: { forced: true, reason: "hotfix SEC-12" }, + }); + }); + + it("pauses and resumes, refusing advance while paused", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.startRollout(PLAN); + await service.pause("soroban-core"); + await expectCode( + service.advance("soroban-core", { force: true, reason: "x" }), + "INVALID_TRANSITION", + ); + await expectCode(service.pause("soroban-core"), "INVALID_TRANSITION"); + expect((await service.resume("soroban-core")).rollout!.state).toBe( + "active", + ); + }); + + it("rolls back to the baseline for every tenant", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.startRollout({ ...PLAN, allowTenants: ["early"] }); + expect((await service.resolve("early", "soroban-core"))!.channel).toBe( + "candidate", + ); + await service.rollback("soroban-core"); + expect(await service.resolve("early", "soroban-core")).toMatchObject({ + version: "1.4.2", + channel: "stable", + }); + await expectCode(service.rollback("soroban-core"), "INVALID_TRANSITION"); + }); + + it("reports PACK_NOT_FOUND and INVALID_TRANSITION for missing packs and rollouts", async () => { + const { service } = setup(); + await expectCode(service.startRollout(PLAN), "PACK_NOT_FOUND"); + await expectCode(service.advance("soroban-core"), "PACK_NOT_FOUND"); + await service.registerPack("soroban-core", "1.4.2"); + await expectCode(service.pause("soroban-core"), "INVALID_TRANSITION"); + }); + + it.each([null, "plan", { ...PLAN, packId: "BAD" }])( + "rejects malformed plan %p", + async (plan) => { + const { service } = setup(); + await expect( + service.startRollout(plan as unknown as RulePackRolloutPlan), + ).rejects.toBeInstanceOf(RulePackRolloutError); + }, + ); +}); + +describe("resolveForScan", () => { + it("pins versions so retries are unaffected by later stage changes", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.startRollout(PLAN); + const request = { + tenantId: IN_SECOND_STAGE_ONLY, + scanId: "scan-1", + packIds: ["soroban-core"], + }; + const first = await service.resolveForScan(request); + expect(first).toMatchObject({ + pin: "created", + fingerprint: "soroban-core@1.4.2", + }); + + await service.advance("soroban-core", { force: true, reason: "test" }); + expect( + (await service.resolve(IN_SECOND_STAGE_ONLY, "soroban-core"))!.version, + ).toBe("1.5.0"); + const retry = await service.resolveForScan(request); + expect(retry).toEqual({ ...first, pin: "reused" }); + expect(service.metrics.snapshot().pinsReused).toBe(1); + }); + + it("normalizes pack order and lists unmanaged packs for backward compatibility", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.registerPack("soroban-security", "2.0.1"); + const result = await service.resolveForScan({ + tenantId: "t1", + scanId: "s1", + packIds: ["soroban-security", "evm-gas", "soroban-core", "soroban-core"], + }); + expect(result.selections.map((s) => s.packId)).toEqual([ + "soroban-core", + "soroban-security", + ]); + expect(result.unmanagedPackIds).toEqual(["evm-gas"]); + expect(result.fingerprint).toBe( + "soroban-core@1.4.2,soroban-security@2.0.1", + ); + // Same set in a different order is the same pin. + await expect( + service.resolveForScan({ + tenantId: "t1", + scanId: "s1", + packIds: ["evm-gas", "soroban-core", "soroban-security"], + }), + ).resolves.toMatchObject({ pin: "reused" }); + }); + + it("treats every pack as unmanaged when none are registered", async () => { + const { service } = setup(); + expect(await service.resolve("t1", "soroban-core")).toBeUndefined(); + expect( + await service.resolveForScan({ + tenantId: "t1", + scanId: "s1", + packIds: ["soroban-core"], + }), + ).toEqual({ + selections: [], + unmanagedPackIds: ["soroban-core"], + fingerprint: "", + pin: "created", + }); + }); + + it("refuses to re-pin a scan with a different pack set", async () => { + const { service } = setup(); + await service.resolveForScan({ + tenantId: "t1", + scanId: "s1", + packIds: ["soroban-core"], + }); + await expectCode( + service.resolveForScan({ + tenantId: "t1", + scanId: "s1", + packIds: ["soroban-core", "evm-gas"], + }), + "PIN_MISMATCH", + ); + }); + + it("keeps pins tenant-scoped when scan ids collide", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.startRollout({ + ...PLAN, + allowTenants: ["tenant-a"], + denyTenants: ["tenant-b"], + }); + const a = await service.resolveForScan({ + tenantId: "tenant-a", + scanId: "shared", + packIds: ["soroban-core"], + }); + const b = await service.resolveForScan({ + tenantId: "tenant-b", + scanId: "shared", + packIds: ["soroban-core"], + }); + expect(a.selections[0].version).toBe("1.5.0"); + expect(b).toMatchObject({ + pin: "created", + selections: [{ version: "1.4.2", reason: "deny_list" }], + }); + }); + + it.each([ + [{ tenantId: "", scanId: "s", packIds: ["p"] }], + [{ tenantId: "t", scanId: "has space", packIds: ["p"] }], + [{ tenantId: "t", scanId: "s", packIds: [] }], + [{ tenantId: "t", scanId: "s", packIds: "p" }], + [{ tenantId: "t", scanId: "s", packIds: ["Upper"] }], + [ + { + tenantId: "t", + scanId: "s", + packIds: Array.from({ length: 65 }, (_, i) => `p${i}`), + }, + ], + [undefined], + ])("rejects invalid request %#", async (request) => { + const { service } = setup(); + await expectCode(service.resolveForScan(request as never), "INVALID_INPUT"); + }); +}); + +describe("recordScanOutcome", () => { + async function started(gate = PLAN.gate) { + const ctx = setup(); + await ctx.service.registerPack("soroban-core", "1.4.2"); + await ctx.service.startRollout({ ...PLAN, gate }); + return ctx; + } + + it("counts candidate outcomes once per scan", async () => { + const { service } = await started(); + await service.resolveForScan({ + tenantId: IN_FIRST_STAGE, + scanId: "s1", + packIds: ["soroban-core"], + }); + expect( + await service.recordScanOutcome({ + tenantId: IN_FIRST_STAGE, + scanId: "s1", + outcome: "succeeded", + }), + ).toEqual({ + status: "recorded", + countedPackIds: ["soroban-core"], + autoRolledBackPackIds: [], + }); + expect( + await service.recordScanOutcome({ + tenantId: IN_FIRST_STAGE, + scanId: "s1", + outcome: "failed", + }), + ).toEqual({ status: "duplicate" }); + const channel = await service.getChannel("soroban-core"); + expect(channel!.rollout!.stageStats).toEqual({ succeeded: 1, failed: 0 }); + }); + + it("ignores unpinned scans, baseline scans and other tenants' scans", async () => { + const { service } = await started(); + expect( + await service.recordScanOutcome({ + tenantId: "t1", + scanId: "never", + outcome: "failed", + }), + ).toEqual({ status: "not_pinned" }); + await service.resolveForScan({ + tenantId: IN_SECOND_STAGE_ONLY, + scanId: "s1", + packIds: ["soroban-core"], + }); + expect( + await service.recordScanOutcome({ + tenantId: IN_SECOND_STAGE_ONLY, + scanId: "s1", + outcome: "failed", + }), + ).toEqual({ + status: "recorded", + countedPackIds: [], + autoRolledBackPackIds: [], + }); + expect( + await service.recordScanOutcome({ + tenantId: "intruder", + scanId: "s1", + outcome: "failed", + }), + ).toEqual({ status: "not_pinned" }); + expect(service.metrics.snapshot().outcomesIgnored).toEqual({ + not_pinned: 2, + not_candidate: 1, + }); + }); + + it("ignores outcomes from scans pinned in an earlier stage", async () => { + const { service } = await started(); + await service.resolveForScan({ + tenantId: IN_FIRST_STAGE, + scanId: "old", + packIds: ["soroban-core"], + }); + await service.advance("soroban-core", { force: true, reason: "test" }); + expect( + await service.recordScanOutcome({ + tenantId: IN_FIRST_STAGE, + scanId: "old", + outcome: "failed", + }), + ).toMatchObject({ countedPackIds: [] }); + expect(service.metrics.snapshot().outcomesIgnored).toEqual({ + stale_stage: 1, + }); + }); + + it("rolls back automatically when the gate is breached", async () => { + const { service, logger } = await started(); + const results = await feed(service, ["succeeded", "failed", "failed"]); + expect(results[2]).toEqual({ + status: "recorded", + countedPackIds: ["soroban-core"], + autoRolledBackPackIds: ["soroban-core"], + }); + const channel = await service.getChannel("soroban-core"); + expect(channel).toMatchObject({ + stableVersion: "1.4.2", + rollout: { state: "rolled_back" }, + }); + expect(channel!.rollout!.events.at(-1)).toMatchObject({ + action: "auto_rolled_back", + }); + expect( + logger.entries.find((e) => e.event === "rollout_auto_rolled_back"), + ).toMatchObject({ + level: "warn", + fields: { packId: "soroban-core", samples: 3 }, + }); + expect( + (await service.resolve(IN_FIRST_STAGE, "soroban-core"))!.version, + ).toBe("1.4.2"); + }); + + it("holds the stage instead of rolling back when autoRollback is off", async () => { + const { service } = await started({ + minSamples: 2, + maxFailureRatio: 0, + autoRollback: false, + }); + await feed(service, ["failed", "failed"]); + const channel = await service.getChannel("soroban-core"); + expect(channel!.rollout!.state).toBe("active"); + await expectCode(service.advance("soroban-core"), "GATE_NOT_MET"); + }); + + it("rejects an invalid outcome", async () => { + const { service } = await started(); + await expectCode( + service.recordScanOutcome({ + tenantId: "t", + scanId: "s", + outcome: "maybe" as never, + }), + "INVALID_INPUT", + ); + }); +}); + +describe("kill switch", () => { + it("serves stable to everyone, refuses mutations, and still honours pins", async () => { + const store = new InMemoryRulePackRolloutStore(); + const live = new RulePackRolloutService({ + store, + logger: new CapturingLogger(), + }); + await live.registerPack("soroban-core", "1.4.2"); + await live.startRollout({ ...PLAN, allowTenants: ["early"] }); + const pinned = await live.resolveForScan({ + tenantId: "early", + scanId: "s1", + packIds: ["soroban-core"], + }); + expect(pinned.selections[0].version).toBe("1.5.0"); + + const disabled = new RulePackRolloutService({ + store, + logger: new CapturingLogger(), + config: { enabled: false }, + }); + expect(await disabled.resolve("early", "soroban-core")).toMatchObject({ + version: "1.4.2", + channel: "stable", + }); + expect( + ( + await disabled.resolveForScan({ + tenantId: "early", + scanId: "s2", + packIds: ["soroban-core"], + }) + ).selections[0].version, + ).toBe("1.4.2"); + expect( + await disabled.resolveForScan({ + tenantId: "early", + scanId: "s1", + packIds: ["soroban-core"], + }), + ).toEqual({ ...pinned, pin: "reused" }); + await expectCode(disabled.advance("soroban-core"), "ROLLOUT_DISABLED"); + await expectCode(disabled.rollback("soroban-core"), "ROLLOUT_DISABLED"); + await expectCode( + disabled.registerPack("other", "1.0.0"), + "ROLLOUT_DISABLED", + ); + }); +}); + +describe("failure handling", () => { + it("maps store errors to retryable STORE_UNAVAILABLE without logging their message", async () => { + const { service, store, logger } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + store.failAll = true; + const error = await service + .resolveForScan({ + tenantId: "t1", + scanId: "s1", + packIds: ["soroban-core"], + }) + .catch((e) => e); + expect(error).toMatchObject({ code: "STORE_UNAVAILABLE", retryable: true }); + expect(error.message).not.toContain("hunter2"); + expect(JSON.stringify(logger.entries)).not.toMatch( + /hunter2|postgres|ECONNREFUSED/, + ); + expect(logger.entries.at(-1)).toMatchObject({ + level: "error", + event: "operation_failed", + fields: { + operation: "resolve_for_scan", + code: "STORE_UNAVAILABLE", + retryable: true, + }, + }); + expect(service.metrics.snapshot().errors).toEqual({ STORE_UNAVAILABLE: 1 }); + + store.failAll = false; + await expect( + service.resolveForScan({ + tenantId: "t1", + scanId: "s1", + packIds: ["soroban-core"], + }), + ).resolves.toMatchObject({ pin: "created" }); + }); + + it("retries channel writes on conflict and gives up with CONFLICT", async () => { + const { service, store } = setup({ config: { maxConflictRetries: 2 } }); + await service.registerPack("soroban-core", "1.4.2"); + store.conflictsRemaining = 2; + await expect(service.startRollout(PLAN)).resolves.toBeDefined(); + expect(service.metrics.snapshot().conflicts).toBe(2); + + store.conflictsRemaining = 3; + const error = await service.pause("soroban-core").catch((e) => e); + expect(error).toMatchObject({ code: "CONFLICT", retryable: true }); + expect((await service.getChannel("soroban-core"))!.rollout!.state).toBe( + "active", + ); + }); + + it("does not leak another tenant's identity through selections", async () => { + const { service } = setup(); + await service.registerPack("soroban-core", "1.4.2"); + await service.startRollout({ + ...PLAN, + allowTenants: ["confidential-customer"], + }); + const selection = await service.resolve("someone-else", "soroban-core"); + expect(JSON.stringify(selection)).not.toContain("confidential-customer"); + }); +}); + +describe("purgeExpiredPins", () => { + it("removes pins older than the retention window", async () => { + const { service, store, tick } = setup({ config: { pinRetentionMs: 100 } }); + await service.resolveForScan({ + tenantId: "t1", + scanId: "old", + packIds: ["p"], + }); + tick(60); + await service.resolveForScan({ + tenantId: "t1", + scanId: "new", + packIds: ["p"], + }); + tick(60); + expect(await service.purgeExpiredPins()).toBe(1); + expect(store.pinCount()).toBe(1); + }); +}); + +describe("determinism", () => { + it("resolves identically across service instances over the same state", async () => { + const store = new InMemoryRulePackRolloutStore(); + const a = new RulePackRolloutService({ + store, + logger: new CapturingLogger(), + }); + await a.registerPack("soroban-core", "1.4.2"); + await a.startRollout(PLAN); + const b = new RulePackRolloutService({ + store, + logger: new CapturingLogger(), + }); + const channel = (await store.getChannel("soroban-core"))!; + for (let i = 0; i < 50; i++) { + const tenant = `tenant-${i}`; + const expected = selectForTenant(channel, tenant, true); + expect(await a.resolve(tenant, "soroban-core")).toEqual(expected); + expect(await b.resolve(tenant, "soroban-core")).toEqual(expected); + } + }); +}); diff --git a/apps/api/src/rule-pack-rollout/rule-pack-rollout.service.ts b/apps/api/src/rule-pack-rollout/rule-pack-rollout.service.ts new file mode 100644 index 00000000..4eece6fe --- /dev/null +++ b/apps/api/src/rule-pack-rollout/rule-pack-rollout.service.ts @@ -0,0 +1,734 @@ +/** + * Staged rollout of rule-pack updates. + * + * Operators register a pack's stable version, start a rollout for a candidate + * version, and move it through its stages with `advance`, `pause`, `resume` + * and `rollback`. Workers call `resolveForScan` before analysing a repository + * and `recordScanOutcome` once the scan has finished; outcomes from candidate + * scans feed the stage gate, which can roll a bad candidate back on its own. + * + * Determinism: a scan's versions are pinned the first time it resolves, so a + * retry of the same scan always runs the same rule-pack versions, and the + * pin's `fingerprint` belongs in any cache key for scan results. + * + * Backward compatibility: packs that were never registered are "unmanaged". + * `resolve` returns `undefined` and `resolveForScan` lists them in + * `unmanagedPackIds`, so callers keep using whatever version they used before. + */ + +import { + DEFAULT_RULE_PACK_ROLLOUT_CONFIG, + RulePackRolloutConfig, + resolveRulePackRolloutConfig, +} from "./rule-pack-rollout-config"; +import { + RulePackRolloutLogger, + RulePackRolloutMetrics, + consoleRulePackRolloutLogger, + sanitizeLogFields, +} from "./rule-pack-rollout-metrics"; +import { + TransitionAction, + buildRollout, + compareStrings, + evaluateGate, + isLive, + requireIdentifier, + requirePackId, + requireVersion, + rulePackFingerprint, + sanitizeReason, + selectForTenant, + transition, +} from "./rule-pack-rollout-planner"; +import { RulePackRolloutStore } from "./rule-pack-rollout-store"; +import { + RulePackChannel, + RulePackRolloutError, + RulePackRolloutPlan, + RulePackSelection, + ScanOutcome, + ScanRulePackPin, +} from "./rule-pack-rollout-types"; + +/** Largest number of packs one scan may resolve. */ +export const MAX_PACKS_PER_SCAN = 64; + +export interface RulePackRolloutServiceOptions { + store: RulePackRolloutStore; + config?: Partial; + logger?: RulePackRolloutLogger; + metrics?: RulePackRolloutMetrics; + clock?: () => number; +} + +export interface ScanRulePackRequest { + tenantId: string; + scanId: string; + packIds: string[]; +} + +export interface ScanRulePackResolution { + /** Managed packs, sorted by `packId`. */ + selections: RulePackSelection[]; + /** Requested packs with no channel; callers keep their existing default. */ + unmanagedPackIds: string[]; + fingerprint: string; + /** `reused` when the scan was already pinned (a retry). */ + pin: "created" | "reused"; +} + +export interface ScanOutcomeRequest { + tenantId: string; + scanId: string; + outcome: ScanOutcome; +} + +export type ScanOutcomeResult = + | { status: "not_pinned" } + | { status: "duplicate" } + | { + status: "recorded"; + /** Packs whose current stage gate counted this outcome. */ + countedPackIds: string[]; + /** Packs whose rollout this outcome rolled back. */ + autoRolledBackPackIds: string[]; + }; + +export interface AdvanceOptions { + /** Advance even though the gate has not passed. Requires `reason`. */ + force?: boolean; + reason?: string; +} + +export class RulePackRolloutService { + readonly config: RulePackRolloutConfig; + readonly metrics: RulePackRolloutMetrics; + private readonly store: RulePackRolloutStore; + private readonly logger: RulePackRolloutLogger; + private readonly clock: () => number; + + constructor(options: RulePackRolloutServiceOptions) { + this.config = resolveRulePackRolloutConfig( + options.config ?? DEFAULT_RULE_PACK_ROLLOUT_CONFIG, + ); + this.store = options.store; + // Sanitize here as well as in the default logger, so a custom logger can + // never receive more than identifiers, versions, counts, and codes. + const logger = options.logger ?? consoleRulePackRolloutLogger; + this.logger = { + log: (level, event, fields) => + logger.log(level, event, sanitizeLogFields(fields)), + }; + this.metrics = options.metrics ?? new RulePackRolloutMetrics(); + this.clock = options.clock ?? Date.now; + } + + /** + * Put a pack under rollout management with its current stable version. + * Idempotent for the same version. + * + * @throws RulePackRolloutError ROLLOUT_DISABLED, INVALID_INPUT, + * PACK_ALREADY_REGISTERED, CONFLICT or STORE_UNAVAILABLE + */ + registerPack( + packId: string, + stableVersion: string, + ): Promise { + return this.guard("register_pack", async () => { + this.requireEnabled(); + requirePackId(packId); + requireVersion(stableVersion, "stableVersion"); + // A create only conflicts when someone else created the channel, so a + // second pass always finds it and settles on whatever won. + for (let attempt = 0; attempt < 2; attempt++) { + const existing = await this.store.getChannel(packId); + if (existing) { + if (existing.stableVersion === stableVersion) { + return existing; + } + throw new RulePackRolloutError( + "PACK_ALREADY_REGISTERED", + "pack is registered with a different stable version; start a rollout to change it", + ); + } + const channel: RulePackChannel = { + packId, + stableVersion, + revision: 1, + rolloutSequence: 0, + updatedAt: this.clock(), + }; + if ((await this.store.putChannel(channel, 0)) === "written") { + this.logger.log("info", "pack_registered", { packId, stableVersion }); + return channel; + } + this.metrics.recordConflict(); + } + throw new RulePackRolloutError( + "CONFLICT", + "channel was modified concurrently; retry the operation", + ); + }); + } + + /** + * Start a rollout at its first stage. + * + * @throws RulePackRolloutError ROLLOUT_DISABLED, INVALID_INPUT, INVALID_PLAN, + * PACK_NOT_FOUND, ROLLOUT_IN_PROGRESS, CONFLICT or STORE_UNAVAILABLE + */ + startRollout( + plan: RulePackRolloutPlan, + reason?: string, + ): Promise { + return this.guard("start_rollout", async () => { + this.requireEnabled(); + if (typeof plan !== "object" || plan === null) { + throw new RulePackRolloutError( + "INVALID_PLAN", + "plan must be an object", + ); + } + const packId = requirePackId(plan.packId); + const channel = await this.mutateChannel(packId, (current, now) => { + const rollout = buildRollout(current, plan, this.config, now); + const sanitized = sanitizeReason(reason); + if (sanitized) { + rollout.events[0].reason = sanitized; + } + return { + ...current, + rolloutSequence: current.rolloutSequence + 1, + rollout, + }; + }); + const rollout = channel.rollout!; + this.metrics.recordTransition("started"); + this.logger.log("info", "rollout_started", { + packId, + rolloutId: rollout.rolloutId, + baselineVersion: rollout.baselineVersion, + candidateVersion: rollout.candidateVersion, + stages: rollout.stagesBasisPoints.length, + stagePercent: rollout.stagesBasisPoints[0] / 100, + reason: sanitizeReason(reason), + }); + return channel; + }); + } + + /** + * Move to the next stage, or complete the rollout from the last stage, which + * makes the candidate the new stable version. + * + * @throws RulePackRolloutError GATE_NOT_MET unless `force`, plus the codes + * of `startRollout` and INVALID_TRANSITION + */ + advance( + packId: string, + options: AdvanceOptions = {}, + ): Promise { + return this.guard("advance", async () => { + this.requireEnabled(); + requirePackId(packId); + const reason = sanitizeReason(options.reason); + if (options.force && !reason) { + throw new RulePackRolloutError( + "INVALID_INPUT", + "a reason is required to force an advance", + ); + } + let forcedPastGate = false; + const channel = await this.mutateChannel(packId, (current, now) => { + const rollout = this.requireRollout(current); + if (rollout.state === "active") { + const gate = evaluateGate(rollout); + if (!gate.passed && !options.force) { + throw new RulePackRolloutError( + "GATE_NOT_MET", + `stage gate not met: ${gate.samples}/${rollout.gate.minSamples} samples, ` + + `failure ratio ${gate.failureRatio.toFixed(4)} (max ${rollout.gate.maxFailureRatio})`, + ); + } + forcedPastGate = !gate.passed; + } + const next = transition( + rollout, + "advance", + now, + this.config.maxEventsPerRollout, + { reason }, + ); + return { + ...current, + stableVersion: + next.state === "completed" + ? next.candidateVersion + : current.stableVersion, + rollout: next, + }; + }); + const rollout = channel.rollout!; + const completed = rollout.state === "completed"; + this.metrics.recordTransition(completed ? "completed" : "advanced"); + this.logger.log( + forcedPastGate ? "warn" : "info", + completed ? "rollout_completed" : "rollout_advanced", + { + packId, + rolloutId: rollout.rolloutId, + stageIndex: rollout.stageIndex, + stagePercent: rollout.stagesBasisPoints[rollout.stageIndex] / 100, + stableVersion: channel.stableVersion, + forced: forcedPastGate, + reason, + }, + ); + return channel; + }); + } + + /** Hold the current stage. Tenants already on the candidate stay on it. */ + pause(packId: string, reason?: string): Promise { + return this.operatorTransition("pause", packId, reason); + } + + resume(packId: string, reason?: string): Promise { + return this.operatorTransition("resume", packId, reason); + } + + /** Return every tenant to the baseline version and end the rollout. */ + rollback(packId: string, reason?: string): Promise { + return this.operatorTransition("rollback", packId, reason); + } + + /** Operator view of a channel, including allow and deny lists. */ + getChannel(packId: string): Promise { + return this.guard("get_channel", async () => + this.store.getChannel(requirePackId(packId)), + ); + } + + listChannels(): Promise { + return this.guard("list_channels", () => this.store.listChannels()); + } + + /** + * The version `tenantId` would get for `packId` right now, without pinning. + * `undefined` for unmanaged packs. + */ + resolve( + tenantId: string, + packId: string, + ): Promise { + return this.guard("resolve", async () => { + requireIdentifier(tenantId, "tenantId"); + requirePackId(packId); + const channel = await this.store.getChannel(packId); + return channel + ? selectForTenant(channel, tenantId, this.config.enabled) + : undefined; + }); + } + + /** + * Resolve and pin rule-pack versions for one scan. Calling again with the + * same tenant, scan and packs returns the pinned versions unchanged. + * + * @throws RulePackRolloutError INVALID_INPUT, PIN_MISMATCH (the scan was + * pinned with a different pack set) or STORE_UNAVAILABLE + */ + resolveForScan( + request: ScanRulePackRequest, + ): Promise { + return this.guard("resolve_for_scan", async () => { + const tenantId = requireIdentifier(request?.tenantId, "tenantId"); + const scanId = requireIdentifier(request.scanId, "scanId"); + const packIds = requirePackIds(request.packIds); + + const existing = await this.store.getPin(tenantId, scanId); + if (existing) { + return this.reusePin(existing, packIds); + } + + const selections: RulePackSelection[] = []; + for (const packId of packIds) { + const channel = await this.store.getChannel(packId); + if (channel) { + selections.push( + selectForTenant(channel, tenantId, this.config.enabled), + ); + } + } + const pin: ScanRulePackPin = { + tenantId, + scanId, + packIds, + selections, + fingerprint: rulePackFingerprint(selections), + pinnedAt: this.clock(), + }; + if ((await this.store.createPin(pin)) === "exists") { + // A concurrent attempt of the same scan pinned first; use its versions. + const winner = await this.store.getPin(tenantId, scanId); + if (winner) { + return this.reusePin(winner, packIds); + } + } + for (const selection of selections) { + this.metrics.recordResolution(selection.channel, selection.reason); + } + this.logger.log("info", "scan_pinned", { + tenantId, + scanId, + fingerprint: pin.fingerprint, + candidatePacks: selections.filter((s) => s.channel === "candidate") + .length, + unmanagedPacks: packIds.length - selections.length, + }); + return toResolution(pin, "created"); + }); + } + + /** + * Record a scan's final outcome. Only the first report per scan counts, and + * only candidate selections pinned during the rollout's current stage feed + * its gate. Report failure only once the scan will not be retried. + * + * Counting is at most once: if the store fails after the pin is marked but + * before the gate is updated, that sample is dropped rather than counted + * twice on retry. + */ + recordScanOutcome(request: ScanOutcomeRequest): Promise { + return this.guard("record_outcome", async () => { + const tenantId = requireIdentifier(request?.tenantId, "tenantId"); + const scanId = requireIdentifier(request.scanId, "scanId"); + const outcome = request.outcome; + if (outcome !== "succeeded" && outcome !== "failed") { + throw new RulePackRolloutError( + "INVALID_INPUT", + "outcome must be 'succeeded' or 'failed'", + ); + } + + const pin = await this.store.getPin(tenantId, scanId); + if (!pin) { + this.metrics.recordOutcomeIgnored("not_pinned"); + return { status: "not_pinned" }; + } + const marked = await this.store.setPinOutcome( + tenantId, + scanId, + outcome, + this.clock(), + ); + if (marked !== "written") { + this.metrics.recordOutcomeIgnored( + marked === "already_recorded" ? "duplicate" : "not_pinned", + ); + return marked === "already_recorded" + ? { status: "duplicate" } + : { status: "not_pinned" }; + } + + const countedPackIds: string[] = []; + const autoRolledBackPackIds: string[] = []; + for (const selection of pin.selections) { + if (selection.channel !== "candidate") { + this.metrics.recordOutcomeIgnored("not_candidate"); + continue; + } + const result = await this.countCandidateOutcome(selection, outcome); + if (result === "ignored") { + this.metrics.recordOutcomeIgnored("stale_stage"); + continue; + } + countedPackIds.push(selection.packId); + this.metrics.recordCandidateOutcome(outcome); + if (result === "rolled_back") { + autoRolledBackPackIds.push(selection.packId); + } + } + return { status: "recorded", countedPackIds, autoRolledBackPackIds }; + }); + } + + /** Remove pins older than `pinRetentionMs`. */ + purgeExpiredPins(): Promise { + return this.guard("purge_pins", async () => { + const removed = await this.store.deletePinsBefore( + this.clock() - this.config.pinRetentionMs, + ); + if (removed > 0) { + this.logger.log("info", "pins_purged", { removed }); + } + return removed; + }); + } + + private async countCandidateOutcome( + selection: RulePackSelection, + outcome: ScanOutcome, + ): Promise<"counted" | "rolled_back" | "ignored"> { + const updated = await this.store.addStageOutcome( + selection.packId, + selection.rolloutId!, + selection.stageIndex!, + outcome, + this.clock(), + ); + if (!updated) { + return "ignored"; + } + const rollout = updated.rollout!; + if (!rollout.gate.autoRollback || !evaluateGate(rollout).breached) { + return "counted"; + } + return (await this.autoRollback(selection)) ? "rolled_back" : "counted"; + } + + /** + * Roll back a rollout whose gate is breached. Returns false when another + * worker got there first or the stage moved on. A CONFLICT here is not an + * error for the caller: the outcome is already counted, and the next + * outcome for this stage re-checks the gate and retries the rollback. + */ + private async autoRollback(selection: RulePackSelection): Promise { + const state = { rolledBack: false, samples: 0, failureRatio: 0 }; + try { + await this.mutateChannel(selection.packId, (current, now) => { + const rollout = current.rollout; + state.rolledBack = false; + if ( + !rollout || + !isLive(rollout) || + rollout.rolloutId !== selection.rolloutId || + rollout.stageIndex !== selection.stageIndex + ) { + return undefined; + } + const gate = evaluateGate(rollout); + if (!gate.breached) { + return undefined; + } + state.rolledBack = true; + state.samples = gate.samples; + state.failureRatio = gate.failureRatio; + return { + ...current, + rollout: transition( + rollout, + "rollback", + now, + this.config.maxEventsPerRollout, + { + auto: true, + reason: `gate breached: failure ratio ${gate.failureRatio.toFixed(4)} over ${gate.samples} samples`, + }, + ), + }; + }); + } catch (error) { + if (error instanceof RulePackRolloutError && error.code === "CONFLICT") { + this.logger.log("warn", "auto_rollback_deferred", { + packId: selection.packId, + rolloutId: selection.rolloutId, + }); + return false; + } + throw error; + } + if (state.rolledBack) { + this.metrics.recordTransition("auto_rolled_back"); + this.logger.log("warn", "rollout_auto_rolled_back", { + packId: selection.packId, + rolloutId: selection.rolloutId, + stageIndex: selection.stageIndex, + samples: state.samples, + failureRatio: state.failureRatio, + }); + } + return state.rolledBack; + } + + private operatorTransition( + action: Exclude, + packId: string, + reason?: string, + ): Promise { + return this.guard(action, async () => { + this.requireEnabled(); + requirePackId(packId); + const sanitized = sanitizeReason(reason); + const channel = await this.mutateChannel(packId, (current, now) => ({ + ...current, + rollout: transition( + this.requireRollout(current), + action, + now, + this.config.maxEventsPerRollout, + { reason: sanitized }, + ), + })); + const event = + action === "pause" + ? "paused" + : action === "resume" + ? "resumed" + : "rolled_back"; + this.metrics.recordTransition(event); + this.logger.log( + action === "rollback" ? "warn" : "info", + `rollout_${event}`, + { + packId, + rolloutId: channel.rollout!.rolloutId, + stageIndex: channel.rollout!.stageIndex, + reason: sanitized, + }, + ); + return channel; + }); + } + + /** + * Read-modify-write a channel with optimistic concurrency, retrying on + * conflict. `mutate` returns `undefined` to leave the channel unchanged. + */ + private async mutateChannel( + packId: string, + mutate: ( + channel: RulePackChannel, + now: number, + ) => RulePackChannel | undefined, + ): Promise { + for ( + let attempt = 0; + attempt <= this.config.maxConflictRetries; + attempt++ + ) { + const current = await this.store.getChannel(packId); + if (!current) { + throw new RulePackRolloutError( + "PACK_NOT_FOUND", + "pack is not registered", + ); + } + const now = this.clock(); + const next = mutate(current, now); + if (!next) { + return current; + } + const written: RulePackChannel = { + ...next, + revision: current.revision + 1, + updatedAt: now, + }; + if ( + (await this.store.putChannel(written, current.revision)) === "written" + ) { + return written; + } + this.metrics.recordConflict(); + } + throw new RulePackRolloutError( + "CONFLICT", + "channel was modified concurrently; retry the operation", + ); + } + + private reusePin( + pin: ScanRulePackPin, + packIds: string[], + ): ScanRulePackResolution { + if ( + pin.packIds.length !== packIds.length || + pin.packIds.some((packId, i) => packId !== packIds[i]) + ) { + throw new RulePackRolloutError( + "PIN_MISMATCH", + "scan is already pinned with a different set of rule packs", + ); + } + this.metrics.recordPinReused(); + return toResolution(pin, "reused"); + } + + private requireRollout(channel: RulePackChannel) { + if (!channel.rollout) { + throw new RulePackRolloutError( + "INVALID_TRANSITION", + "pack has no rollout", + ); + } + return channel.rollout; + } + + private requireEnabled(): void { + if (!this.config.enabled) { + throw new RulePackRolloutError( + "ROLLOUT_DISABLED", + "rule-pack rollouts are disabled", + ); + } + } + + /** + * Run `task`, mapping unexpected (store) errors to STORE_UNAVAILABLE and + * counting every failure by code. Store error messages are never logged: + * they can contain file paths or connection strings. + */ + private async guard( + operation: string, + task: () => Promise, + ): Promise { + try { + return await task(); + } catch (error) { + const mapped = + error instanceof RulePackRolloutError + ? error + : new RulePackRolloutError( + "STORE_UNAVAILABLE", + "rule-pack rollout storage is unavailable", + ); + this.metrics.recordError(mapped.code); + this.logger.log(mapped.retryable ? "error" : "warn", "operation_failed", { + operation, + code: mapped.code, + retryable: mapped.retryable, + }); + throw mapped; + } + } +} + +function requirePackIds(value: unknown): string[] { + if (!Array.isArray(value) || value.length === 0) { + throw new RulePackRolloutError( + "INVALID_INPUT", + "packIds must be a non-empty array", + ); + } + if (value.length > MAX_PACKS_PER_SCAN) { + throw new RulePackRolloutError( + "INVALID_INPUT", + `a scan may resolve at most ${MAX_PACKS_PER_SCAN} packs`, + ); + } + return [...new Set(value.map(requirePackId))].sort(compareStrings); +} + +function toResolution( + pin: ScanRulePackPin, + status: "created" | "reused", +): ScanRulePackResolution { + const managed = new Set(pin.selections.map((s) => s.packId)); + return { + selections: pin.selections, + unmanagedPackIds: pin.packIds.filter((packId) => !managed.has(packId)), + fingerprint: pin.fingerprint, + pin: status, + }; +} diff --git a/docs/RULE_PACK_STAGED_ROLLOUT.md b/docs/RULE_PACK_STAGED_ROLLOUT.md new file mode 100644 index 00000000..ad349832 --- /dev/null +++ b/docs/RULE_PACK_STAGED_ROLLOUT.md @@ -0,0 +1,293 @@ +# Staged Rule-Pack Rollout + +Part of the **Release and Governance** workstream. + +A new rule-pack version can change findings for every repository GasGuard scans. Shipping it to all +tenants at once means a bad rule (a crash, a false-positive flood) hits everyone before anyone +notices. Staged rollout ships a candidate version to a growing, deterministic cohort of tenants, +gates each stage on observed scan outcomes, and rolls back automatically when the gate is breached. + +| Piece | Location | +| --- | --- | +| Types and error codes | `apps/api/src/rule-pack-rollout/rule-pack-rollout-types.ts` | +| Configuration (`RULE_PACK_ROLLOUT_*`) | `apps/api/src/rule-pack-rollout/rule-pack-rollout-config.ts` | +| Pure planning: validation, cohorts, gate, state machine | `apps/api/src/rule-pack-rollout/rule-pack-rollout-planner.ts` | +| Persistence boundary (in-memory and file stores) | `apps/api/src/rule-pack-rollout/rule-pack-rollout-store.ts` | +| Operator and worker API | `apps/api/src/rule-pack-rollout/rule-pack-rollout.service.ts` | +| Worker adapter | `apps/api/src/rule-pack-rollout/rule-pack-rollout-worker.ts` | +| Metrics and structured logs | `apps/api/src/rule-pack-rollout/rule-pack-rollout-metrics.ts` | +| Fixtures (realistic repository, cohort regression snapshot) | `apps/api/src/rule-pack-rollout/__fixtures__/` | + +The module is self-contained: it depends on no other GasGuard module, and existing scan paths are +unchanged until a worker opts in through `runScanWithRulePacks`. + +## Concepts + +- **Pack**: a versioned bundle of rules, identified by a slug such as `soroban-core` + (`^[a-z0-9][a-z0-9._-]{0,63}$`). Versions are SemVer `MAJOR.MINOR.PATCH[-PRERELEASE]`; build + metadata (`+build`) is rejected because it does not affect precedence, so two distinct versions + could otherwise compare equal. +- **Channel**: the platform-scoped record for one pack: its stable version plus the current or most + recent rollout. +- **Rollout**: moves the pack from a *baseline* (the stable version when it started) to a + *candidate*, through stages that give the candidate to a cumulative percentage of tenants. +- **Pin**: the versions fixed for one scan the first time it resolves rule packs. Retries of that scan + always use its pin. + +### Lifecycle + +``` + advance (gate passed, or force + reason) + ┌───────────────┐ + ▼ │ + start ─► active ─────────┴─► (last stage) advance ─► completed stable := candidate + │ ▲ + pause │ │ resume + ▼ │ + paused + │ + rollback │ (operator, or automatic on a gate breach from active/paused) + ▼ + rolled_back stable unchanged +``` + +Only one live (`active` or `paused`) rollout per pack. `completed` and `rolled_back` are terminal; start +a new rollout to try again (it gets a new `rolloutId`, e.g. `soroban-core#2`). + +### Who gets the candidate + +For a live rollout, in order: + +1. Tenant on `denyTenants`: baseline (`reason: "deny_list"`). +2. Tenant on `allowTenants`: candidate (`reason: "allow_list"`). +3. Otherwise `tenantBucket(packId, candidateVersion, tenantId) < stage threshold`: candidate, else + baseline (`reason: "cohort"`). + +`tenantBucket` is the first 32 bits of +`SHA-256("gasguard-rule-pack-rollout\0" + packId + "\0" + candidateVersion + "\0" + tenantId)` +modulo 10,000. Stage thresholds are basis points (a stage of `12.5` is 1,250). Consequences: + +- **Monotonic**: thresholds only grow, so a tenant on the candidate stays on it for every later stage. +- **Rotating**: the candidate version is in the hash, so successive releases sample different tenants. +- **Stable across processes and machines**: no randomness, no clock, no locale. + +With no live rollout (none started, completed, rolled back, or the kill switch on), every tenant gets +the stable version (`channel: "stable"`, `reason: "no_active_rollout"`). + +## Determinism + +Findings are reproducible for the same commit, configuration and rule-pack versions because: + +1. Selection is a pure function of the channel record and the tenant id. +2. `resolveForScan` pins the selection per `(tenantId, scanId)` on first call. Later calls for the same + scan, including retries after the rollout advanced, paused, completed, rolled back or was disabled, + return the pinned versions (`pin: "reused"`). +3. The resolution's `fingerprint` (`pack@version` pairs in pack-id order, e.g. + `soroban-core@1.5.0,soroban-security@2.0.1`) identifies the exact rule set. **Include it in any + cache key for scan results**: a cache keyed only on repository and commit would serve findings + computed with a different rule-pack version. + +The `cohort-assignments.json` fixture freezes bucket values and stage membership for a set of tenant +ids. If that test fails, cohorts would reshuffle mid-rollout; change the snapshot only together with a +documented migration (finish or roll back every live rollout first). + +## Interface + +### Operator + +```ts +import { + FileRulePackRolloutStore, + RulePackRolloutService, + loadRulePackRolloutConfigFromEnv, +} from './rule-pack-rollout'; + +const rollouts = new RulePackRolloutService({ + store: new FileRulePackRolloutStore('/var/lib/gasguard/rule-pack-rollout'), + config: loadRulePackRolloutConfigFromEnv(), +}); + +await rollouts.registerPack('soroban-core', '1.4.2'); // idempotent for the same version +await rollouts.startRollout({ + packId: 'soroban-core', + candidateVersion: '1.5.0', // must be newer than stable + stages: [1, 10, 50, 100], // strictly increasing, ends at 100 + allowTenants: ['gasguard-dogfood'], + denyTenants: ['regulated-bank-eu'], + gate: { minSamples: 50, maxFailureRatio: 0.02, autoRollback: true }, +}, 'storage-rent rules, RFC-31'); + +await rollouts.advance('soroban-core'); // GATE_NOT_MET until the gate passes +await rollouts.advance('soroban-core', { force: true, reason: 'SEC-12 hotfix' }); // audited override +await rollouts.pause('soroban-core', 'investigating spike'); +await rollouts.resume('soroban-core'); +await rollouts.rollback('soroban-core', 'false positives on AMM repos'); + +await rollouts.getChannel('soroban-core'); // full record incl. events history +await rollouts.listChannels(); +``` + +`getChannel` and `listChannels` return operator data (including allow and deny lists, which name +tenants) and must only be exposed to platform operators. + +### Worker + +```ts +import { runScanWithRulePacks } from './rule-pack-rollout'; + +const findings = await runScanWithRulePacks( + rollouts, + { + tenantId: job.data.tenantId, + scanId: job.data.scanId, // stable across retries of the same scan + packIds: ['soroban-core', 'soroban-security', 'evm-gas'], + isFinalAttempt: job.attemptsMade + 1 >= job.opts.attempts, + }, + async ({ selections, unmanagedPackIds, fingerprint }) => + analyze(repository, commit, selections, unmanagedPackIds, fingerprint), +); +``` + +The adapter resolves (or re-reads) the pin, runs the scan, and reports the outcome. Lower-level calls +are available as `resolveForScan` and `recordScanOutcome`; `resolve(tenantId, packId)` previews a +selection without pinning. + +### Housekeeping + +`purgeExpiredPins()` deletes pins older than `pinRetentionMs`. Run it periodically, for example from +the existing scheduled-maintenance job. Keep the retention longer than the longest a scan can be +retried. + +## Configuration and secure defaults + +| Option | Env variable | Default | Notes | +| --- | --- | --- | --- | +| `enabled` | `RULE_PACK_ROLLOUT_ENABLED` | `true` | Kill switch. See below. | +| `defaultMinSamples` | `RULE_PACK_ROLLOUT_MIN_SAMPLES` | `20` | Gate default when a plan omits it. | +| `defaultMaxFailureRatio` | `RULE_PACK_ROLLOUT_MAX_FAILURE_RATIO` | `0.05` | 0-1. | +| `defaultAutoRollback` | `RULE_PACK_ROLLOUT_AUTO_ROLLBACK` | `true` | | +| `maxStages` | `RULE_PACK_ROLLOUT_MAX_STAGES` | `10` | | +| `maxListedTenants` | `RULE_PACK_ROLLOUT_MAX_LISTED_TENANTS` | `1000` | Allow plus deny list size. | +| `maxConflictRetries` | `RULE_PACK_ROLLOUT_MAX_CONFLICT_RETRIES` | `5` | Re-reads after an optimistic write conflict. | +| `pinRetentionMs` | `RULE_PACK_ROLLOUT_PIN_RETENTION_MS` | 7 days | | +| `maxEventsPerRollout` | `RULE_PACK_ROLLOUT_MAX_EVENTS` | `100` | History cap, oldest dropped first. | + +Unset or empty variables keep their defaults. A set but unparseable or out-of-range value throws a +`RangeError` naming the option at startup; nothing silently falls back. + +**Kill switch** (`enabled: false`): new resolutions return the stable version for every tenant, so the +candidate is withdrawn instantly without touching stored state, and every mutation fails with +`ROLLOUT_DISABLED`. Existing pins are still honoured so in-flight retries stay deterministic. +Re-enabling restores the rollout exactly where it was. + +### Gate + +A stage gate counts the final outcomes of candidate scans **pinned during that stage**. Scans pinned +in an earlier stage, baseline scans and duplicate reports are ignored (and counted in +`gasguard_rule_pack_outcomes_ignored_total`). + +- **Passed**: at least `minSamples` outcomes and `failed / total <= maxFailureRatio`. Required for + `advance` unless forced. +- **Breached**: at least `minSamples` outcomes and `failed / total > maxFailureRatio`. With + `autoRollback`, the outcome that breaches it rolls the rollout back. +- Below `minSamples`, the gate is neither, so a single early failure cannot roll back a stage. + +Advancing resets the stage counters. + +## Failure semantics + +All service errors are `RulePackRolloutError` with a stable `code` and a `retryable` flag. + +| Code | Retryable | When | +| --- | --- | --- | +| `ROLLOUT_DISABLED` | no | Mutation while the kill switch is on. | +| `INVALID_INPUT` | no | Malformed pack id, tenant or scan id, version, outcome, or a forced advance without a reason. | +| `INVALID_PLAN` | no | Stages not strictly increasing, not ending at 100, too many, more than two decimals; overlapping or oversized tenant lists; bad gate; candidate not newer than stable. | +| `INVALID_TRANSITION` | no | Action not allowed from the current state, or no rollout. | +| `PACK_NOT_FOUND` | no | Rollout operation on an unregistered pack. | +| `PACK_ALREADY_REGISTERED` | no | `registerPack` with a different stable version. Use a rollout to change versions. | +| `ROLLOUT_IN_PROGRESS` | no | `startRollout` while a rollout is live. | +| `GATE_NOT_MET` | no | `advance` before the gate passes. The message gives samples and failure ratio. | +| `PIN_MISMATCH` | no | The scan was already pinned with a different set of packs. Use a new scan id. | +| `CONFLICT` | yes | Optimistic write conflicts exceeded `maxConflictRetries`. Nothing was written. | +| `STORE_UNAVAILABLE` | yes | The store threw. The store's message is deliberately discarded (it may contain paths or connection strings). | + +### Unsupported input + +Packs that were never registered are **unmanaged**: `resolve` returns `undefined` and +`resolveForScan` lists them in `unmanagedPackIds` with no selection, so callers keep using whatever +version they used before. This is the backward-compatibility path: adopting the module changes nothing +until a pack is registered. Malformed identifiers fail fast with `INVALID_INPUT` before any I/O. + +### Retries and partial failure + +- **Resolution failure**: `runScanWithRulePacks` throws before running the scan, so the queue's normal + retry policy applies (`STORE_UNAVAILABLE` is retryable). No analysis runs on an unknown rule set. +- **Concurrent attempts of the same scan**: pins are write-once; the loser of a race reads and uses + the winner's pin. +- **Failed attempts**: the adapter reports `failed` only when `isFinalAttempt` is true, so a transient + failure that a retry recovers from counts as one success, not a failure plus a success. +- **Duplicate reports**: a pin's outcome is set once; later reports return `{ status: "duplicate" }`. +- **Outcome recording is best-effort and at most once**: the pin is marked before the stage counter is + incremented. If the store fails in between, that sample is dropped rather than double-counted on + retry; this can only delay a gate decision. A failure to record never changes the scan's own result. +- **Concurrent outcomes**: counters are incremented with the store's atomic `addStageOutcome`, never a + client-side read-modify-write, so heavy outcome traffic neither loses increments nor starves + operator writes. If an automatic rollback loses its optimistic write race repeatedly, it logs + `auto_rollback_deferred` and the next outcome for that stage retries it. +- **Operator races**: channel writes are conditional on the revision read, retried up to + `maxConflictRetries`, then fail with `CONFLICT` having written nothing. + +## Tenant isolation + +- Channels are platform data. The tenant-facing result (`RulePackSelection`) contains only the pack, + version, channel, reason, rollout id and stage, never other tenants or the allow/deny lists. +- Pins are keyed by `(tenantId, scanId)`. Every pin read and write takes the tenant id, so a tenant + cannot read, reuse or report an outcome for another tenant's scan even when scan ids collide. +- `FileRulePackRolloutStore` derives every path from SHA-256 digests, so crafted ids cannot traverse + outside the root, and each tenant's pins live in their own directory. Files are written `0600` via + temp-file-and-rename and verified against their key on read. + +## Metrics and logs + +`service.metrics.toPrometheus(await service.listChannels())`: + +| Metric | Type | Labels | Use | +| --- | --- | --- | --- | +| `gasguard_rule_pack_resolutions_total` | counter | `channel`, `reason` | Candidate share per reason. | +| `gasguard_rule_pack_pins_reused_total` | counter | | Retry volume. | +| `gasguard_rule_pack_rollout_transitions_total` | counter | `action` | Alert on `auto_rolled_back`. | +| `gasguard_rule_pack_candidate_outcomes_total` | counter | `outcome` | Candidate failure rate. | +| `gasguard_rule_pack_outcomes_ignored_total` | counter | `reason` | `not_pinned`, `duplicate`, `not_candidate`, `stale_stage`. | +| `gasguard_rule_pack_rollout_conflicts_total` | counter | | Write contention. | +| `gasguard_rule_pack_rollout_errors_total` | counter | `code` | Alert on `STORE_UNAVAILABLE`. | +| `gasguard_rule_pack_rollout_stage_percent` | gauge | `pack` | Current exposure, 0 when not live. | +| `gasguard_rule_pack_rollout_state` | gauge | `pack`, `state` | | + +Labels come from closed sets plus the validated pack slug; tenant, repository and scan ids never +become labels. + +Logs are one JSON object per line with `component: "rule-pack-rollout"`. Events: `pack_registered`, +`rollout_started`, `rollout_advanced` (`warn` when forced past the gate), `rollout_completed`, +`rollout_paused`, `rollout_resumed`, `rollout_rolled_back`, `rollout_auto_rolled_back` (`warn`, with +samples and failure ratio), `auto_rollback_deferred`, `scan_pinned`, `pins_purged`, and +`operation_failed` (`error` when retryable, with `operation` and `code`). Fields are identifiers, +versions, counts, codes and sanitized operator reasons only. Every logger, including a custom one, is +wrapped in `sanitizeLogFields`, which drops keys that look like payloads, source or credentials and +truncates strings to 200 characters. Store error messages and scan content are never logged. + +## Limitations + +- `FileRulePackRolloutStore` serializes writes per key **within one process**. Several API or worker + processes sharing storage need a database store that implements `putChannel` and `addStageOutcome` + as single conditional updates and `createPin` as an insert-if-absent. +- The gate measures scan failures, not finding quality. Use small early stages and the allow list + (dogfood tenants) to review finding diffs before widening. + +## Verification + +```bash +npx jest apps/api/src/rule-pack-rollout # unit + integration +npx jest apps/api/src/rule-pack-rollout/rule-pack-rollout.integration.spec.ts +npx eslint apps/api/src/rule-pack-rollout +```