From 3b6e156897c7fbe49fca71a288e3d473be0dd83e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMainnet-ops=E2=80=9D?= <“footballwatch68@gmail.com”> Date: Sun, 18 Oct 2026 16:27:12 +0100 Subject: [PATCH] feat: add supported-language lifecycle policy Give every analyzable language a documented lifecycle stage (preview, supported, deprecated, retired) with scheduled deprecation and sunset dates, so languages can be trialled, announced for removal, and retired without retired files silently reading as clean results. - Versioned, validated policy with a canonical digest; scheduled dates move languages forward on their own (inclusive UTC-day boundaries). - Governance checks between policy versions: no removals, no retiring a supported language without deprecation, minimum notice from the publish day, no pulling a sunset forward. Overrides need a reason and are audited. - Per-scan write-once pins (policy digest, asOf day, tenant opt-ins, config) keep retries deterministic across restarts, publishes and sunsets; the pin fingerprint belongs in scan-result cache keys. - planScan filters the engine's path-to-language map for AnalyzerRegistry.analyzeMultiple and reports skipped files and deprecation warnings; complete / partial / unsupported / empty statuses. - Tenant-scoped preview opt-ins and pins; digest-derived file-store paths. - Backward compatible: built-in default marks every engine language supported, unlisted languages are analyzed, and LANGUAGE_LIFECYCLE_ENABLED =false restores pre-policy behaviour while honouring existing pins. - Retryable CONFLICT / STORE_UNAVAILABLE vs non-retryable INVALID_INPUT, INVALID_POLICY, TRANSITION_REJECTED, UNSUPPORTED_REPOSITORY. - Prometheus metrics with bounded labels; sanitized logs that never include file paths, content, or store error messages. - Fixtures: realistic Soroban DEX repository with EVM/Starknet bridges, and a frozen sunset-boundary regression snapshot. - Docs: docs/LANGUAGE_LIFECYCLE_POLICY.md Co-Authored-By: Claude Opus 5.5 --- .../__fixtures__/governed-policy.json | 30 + .../__fixtures__/soroban-dex-expected.json | 38 + .../__fixtures__/soroban-dex-repository.json | 26 + .../__fixtures__/sunset-boundary-policy.json | 15 + .../sunset-boundary-regression.json | 143 ++++ apps/api/src/language-lifecycle/index.ts | 7 + .../language-lifecycle-config.spec.ts | 88 ++ .../language-lifecycle-config.ts | 151 ++++ .../language-lifecycle-metrics.spec.ts | 123 +++ .../language-lifecycle-metrics.ts | 261 ++++++ .../language-lifecycle-policy.spec.ts | 788 ++++++++++++++++++ .../language-lifecycle-policy.ts | 722 ++++++++++++++++ .../language-lifecycle-store.spec.ts | 223 +++++ .../language-lifecycle-store.ts | 364 ++++++++ .../language-lifecycle-types.ts | 212 +++++ .../language-lifecycle-worker.ts | 68 ++ .../language-lifecycle.integration.spec.ts | 438 ++++++++++ .../language-lifecycle.service.spec.ts | 459 ++++++++++ .../language-lifecycle.service.ts | 526 ++++++++++++ docs/LANGUAGE_LIFECYCLE_POLICY.md | 306 +++++++ 20 files changed, 4988 insertions(+) create mode 100644 apps/api/src/language-lifecycle/__fixtures__/governed-policy.json create mode 100644 apps/api/src/language-lifecycle/__fixtures__/soroban-dex-expected.json create mode 100644 apps/api/src/language-lifecycle/__fixtures__/soroban-dex-repository.json create mode 100644 apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-policy.json create mode 100644 apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-regression.json create mode 100644 apps/api/src/language-lifecycle/index.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-config.spec.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-config.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-metrics.spec.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-metrics.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-policy.spec.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-policy.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-store.spec.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-store.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-types.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle-worker.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle.integration.spec.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle.service.spec.ts create mode 100644 apps/api/src/language-lifecycle/language-lifecycle.service.ts create mode 100644 docs/LANGUAGE_LIFECYCLE_POLICY.md diff --git a/apps/api/src/language-lifecycle/__fixtures__/governed-policy.json b/apps/api/src/language-lifecycle/__fixtures__/governed-policy.json new file mode 100644 index 00000000..680bc164 --- /dev/null +++ b/apps/api/src/language-lifecycle/__fixtures__/governed-policy.json @@ -0,0 +1,30 @@ +{ + "version": "1.2.0", + "languages": [ + { "language": "cairo", "stage": "preview", "extensions": [".cairo"] }, + { + "language": "javascript", + "stage": "retired", + "extensions": [".cjs", ".js", ".mjs"], + "sunsetAt": "2026-06-30", + "replacement": "typescript" + }, + { "language": "move", "stage": "preview", "extensions": [".move"] }, + { "language": "rust", "stage": "supported", "extensions": [".rs"] }, + { "language": "solidity", "stage": "supported", "extensions": [".sol"] }, + { "language": "soroban", "stage": "supported", "extensions": [] }, + { + "language": "typescript", + "stage": "supported", + "extensions": [".cts", ".mts", ".ts"] + }, + { + "language": "vyper", + "stage": "deprecated", + "extensions": [".vy"], + "deprecatedAt": "2026-07-01", + "sunsetAt": "2027-01-31", + "replacement": "solidity" + } + ] +} diff --git a/apps/api/src/language-lifecycle/__fixtures__/soroban-dex-expected.json b/apps/api/src/language-lifecycle/__fixtures__/soroban-dex-expected.json new file mode 100644 index 00000000..f9978d9a --- /dev/null +++ b/apps/api/src/language-lifecycle/__fixtures__/soroban-dex-expected.json @@ -0,0 +1,38 @@ +{ + "policyDigest": "87f2c0b5b54cba2ad6ab9318f3232aeafc31d1c0605901c08000f59ba5e165eb", + "fingerprint": "c34cb45f82b5c476c87ce338db4e07cf11d5beded43b494af21948240a6b018c", + "status": "partial", + "analyze": { + "bridge/evm/contracts/Bridge.sol": "solidity", + "bridge/evm/contracts/interfaces/IBridge.sol": "solidity", + "bridge/evm/legacy/Vault.vy": "vyper", + "contracts/amm/src/lib.rs": "soroban", + "contracts/amm/src/pool.rs": "soroban", + "contracts/amm/src/test.rs": "soroban", + "contracts/token/src/lib.rs": "soroban", + "frontend/src/index.test.ts": "typescript", + "scripts/deploy.ts": "typescript", + "tools/indexer/src/main.rs": "rust" + }, + "skipped": [ + { + "path": "bridge/starknet/src/bridge.cairo", + "language": "cairo", + "reason": "preview_not_enabled" + }, + { + "path": "scripts/legacy-deploy.js", + "language": "javascript", + "reason": "retired" + } + ], + "warnings": [ + { + "language": "vyper", + "files": 1, + "message": "Language \"vyper\" is deprecated and will no longer be analyzed from 2027-01-31. Migrate to \"solidity\".", + "sunsetAt": "2027-01-31", + "replacement": "solidity" + } + ] +} diff --git a/apps/api/src/language-lifecycle/__fixtures__/soroban-dex-repository.json b/apps/api/src/language-lifecycle/__fixtures__/soroban-dex-repository.json new file mode 100644 index 00000000..a5c06289 --- /dev/null +++ b/apps/api/src/language-lifecycle/__fixtures__/soroban-dex-repository.json @@ -0,0 +1,26 @@ +{ + "tenantId": "tenant-stellar-dex", + "repository": "stellar-dex/amm-monorepo", + "commit": "9f3c2a71d4e85b0c6a1f27e3d98b54c0a2e1f6d7", + "commitDate": "2026-10-15", + "scanId": "scan-amm-9f3c2a71", + "files": [ + "Cargo.toml", + "README.md", + "bridge/evm/contracts/Bridge.sol", + "bridge/evm/contracts/interfaces/IBridge.sol", + "bridge/evm/legacy/Vault.vy", + "bridge/evm/package.json", + "bridge/starknet/src/bridge.cairo", + "contracts/amm/Cargo.toml", + "contracts/amm/src/lib.rs", + "contracts/amm/src/pool.rs", + "contracts/amm/src/test.rs", + "contracts/token/src/lib.rs", + "frontend/src/index.test.ts", + "scripts/deploy.ts", + "scripts/legacy-deploy.js", + "tools/indexer/src/main.rs" + ], + "sorobanPrefixes": ["contracts/"] +} diff --git a/apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-policy.json b/apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-policy.json new file mode 100644 index 00000000..d5a774c8 --- /dev/null +++ b/apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-policy.json @@ -0,0 +1,15 @@ +{ + "version": "2.0.0", + "languages": [ + { "language": "cairo", "stage": "preview", "extensions": [".cairo"] }, + { "language": "solidity", "stage": "supported", "extensions": [".sol"] }, + { + "language": "vyper", + "stage": "supported", + "extensions": [".vy"], + "deprecatedAt": "2026-11-01", + "sunsetAt": "2027-02-01", + "replacement": "solidity" + } + ] +} diff --git a/apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-regression.json b/apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-regression.json new file mode 100644 index 00000000..e12dff44 --- /dev/null +++ b/apps/api/src/language-lifecycle/__fixtures__/sunset-boundary-regression.json @@ -0,0 +1,143 @@ +{ + "policyDigest": "bbd4dbafd876b555d505cb437a6253b8460e26d6c83fd078df85477d09dd1e87", + "cases": [ + { + "asOf": "2026-10-31", + "previewOptIns": [], + "decisions": [ + { + "language": "cairo", + "stage": "preview", + "action": "skip", + "reason": "preview_not_enabled" + }, + { + "language": "solidity", + "stage": "supported", + "action": "analyze", + "reason": "supported" + }, + { + "language": "vyper", + "stage": "supported", + "action": "analyze", + "reason": "supported" + } + ], + "fingerprint": "e31754a2ddb4ea5d610a2b4001e51bdbc684e5ec853ab4d0acb11dfcb3b205ed" + }, + { + "asOf": "2026-11-01", + "previewOptIns": [], + "decisions": [ + { + "language": "cairo", + "stage": "preview", + "action": "skip", + "reason": "preview_not_enabled" + }, + { + "language": "solidity", + "stage": "supported", + "action": "analyze", + "reason": "supported" + }, + { + "language": "vyper", + "stage": "deprecated", + "action": "analyze", + "reason": "deprecated", + "sunsetAt": "2027-02-01", + "replacement": "solidity" + } + ], + "fingerprint": "5371bcc36a0dc6e831caf92476b94bc4ec04f437a020ca46bdfe06c85622314e" + }, + { + "asOf": "2027-01-31", + "previewOptIns": [], + "decisions": [ + { + "language": "cairo", + "stage": "preview", + "action": "skip", + "reason": "preview_not_enabled" + }, + { + "language": "solidity", + "stage": "supported", + "action": "analyze", + "reason": "supported" + }, + { + "language": "vyper", + "stage": "deprecated", + "action": "analyze", + "reason": "deprecated", + "sunsetAt": "2027-02-01", + "replacement": "solidity" + } + ], + "fingerprint": "ababddf2161b46f4de9b078edadbc398261085340181ee00ab160eec5a1bbbb5" + }, + { + "asOf": "2027-02-01", + "previewOptIns": [], + "decisions": [ + { + "language": "cairo", + "stage": "preview", + "action": "skip", + "reason": "preview_not_enabled" + }, + { + "language": "solidity", + "stage": "supported", + "action": "analyze", + "reason": "supported" + }, + { + "language": "vyper", + "stage": "retired", + "action": "skip", + "reason": "retired", + "sunsetAt": "2027-02-01", + "replacement": "solidity" + } + ], + "fingerprint": "843db2e646eadb6dad760d68001c03bbfbaa39343d70c97e6d55c4146bb3803c" + }, + { + "asOf": "2027-02-01", + "previewOptIns": ["cairo"], + "decisions": [ + { + "language": "cairo", + "stage": "preview", + "action": "analyze", + "reason": "preview_opt_in" + }, + { + "language": "solidity", + "stage": "supported", + "action": "analyze", + "reason": "supported" + }, + { + "language": "vyper", + "stage": "retired", + "action": "skip", + "reason": "retired", + "sunsetAt": "2027-02-01", + "replacement": "solidity" + } + ], + "fingerprint": "1a300dbfa24b7ba7a1953b6a65d58a8d4e97e70220cca41a1eb9de6d26a43574" + } + ], + "detection": { + "Contracts/Token.SOL": "solidity", + "a/b.vy": "vyper", + "lib/x.cairo": "cairo" + } +} diff --git a/apps/api/src/language-lifecycle/index.ts b/apps/api/src/language-lifecycle/index.ts new file mode 100644 index 00000000..ceb5ff01 --- /dev/null +++ b/apps/api/src/language-lifecycle/index.ts @@ -0,0 +1,7 @@ +export * from "./language-lifecycle-types"; +export * from "./language-lifecycle-config"; +export * from "./language-lifecycle-policy"; +export * from "./language-lifecycle-store"; +export * from "./language-lifecycle-metrics"; +export * from "./language-lifecycle.service"; +export * from "./language-lifecycle-worker"; diff --git a/apps/api/src/language-lifecycle/language-lifecycle-config.spec.ts b/apps/api/src/language-lifecycle/language-lifecycle-config.spec.ts new file mode 100644 index 00000000..29f5a580 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-config.spec.ts @@ -0,0 +1,88 @@ +import { + DEFAULT_LANGUAGE_LIFECYCLE_CONFIG, + loadLanguageLifecycleConfigFromEnv, + resolveLanguageLifecycleConfig, +} from "./language-lifecycle-config"; + +describe("resolveLanguageLifecycleConfig", () => { + it("returns the defaults when nothing is overridden", () => { + expect(resolveLanguageLifecycleConfig()).toEqual( + DEFAULT_LANGUAGE_LIFECYCLE_CONFIG, + ); + }); + + it("defaults to the pre-policy behaviour for unlisted languages", () => { + expect(DEFAULT_LANGUAGE_LIFECYCLE_CONFIG.unlistedLanguageAction).toBe( + "analyze", + ); + }); + + it("ignores undefined overrides", () => { + expect( + resolveLanguageLifecycleConfig({ maxFilesPerScan: undefined }), + ).toEqual(DEFAULT_LANGUAGE_LIFECYCLE_CONFIG); + }); + + it("accepts a zero notice period", () => { + expect( + resolveLanguageLifecycleConfig({ minDeprecationNoticeDays: 0 }) + .minDeprecationNoticeDays, + ).toBe(0); + }); + + it.each([ + [{ enabled: "yes" as unknown as boolean }, /enabled/], + [{ minDeprecationNoticeDays: -1 }, /minDeprecationNoticeDays/], + [{ minDeprecationNoticeDays: 1.5 }, /minDeprecationNoticeDays/], + [{ maxFilesPerScan: 0 }, /maxFilesPerScan/], + [{ maxLanguages: Number.NaN }, /maxLanguages/], + [{ pinRetentionMs: -5 }, /pinRetentionMs/], + [ + { unlistedLanguageAction: "drop" as unknown as "skip" }, + /unlistedLanguageAction/, + ], + ])("rejects %j", (overrides, message) => { + expect(() => resolveLanguageLifecycleConfig(overrides)).toThrow(message); + }); +}); + +describe("loadLanguageLifecycleConfigFromEnv", () => { + it("reads every variable", () => { + expect( + loadLanguageLifecycleConfigFromEnv({ + LANGUAGE_LIFECYCLE_ENABLED: "off", + LANGUAGE_LIFECYCLE_MIN_NOTICE_DAYS: "30", + LANGUAGE_LIFECYCLE_UNLISTED_ACTION: " SKIP ", + LANGUAGE_LIFECYCLE_MAX_FILES: "500", + LANGUAGE_LIFECYCLE_MAX_LANGUAGES: "8", + LANGUAGE_LIFECYCLE_PIN_RETENTION_MS: "1000", + }), + ).toEqual({ + enabled: false, + minDeprecationNoticeDays: 30, + unlistedLanguageAction: "skip", + maxFilesPerScan: 500, + maxLanguages: 8, + pinRetentionMs: 1000, + }); + }); + + it("keeps defaults for unset and blank variables", () => { + expect( + loadLanguageLifecycleConfigFromEnv({ + LANGUAGE_LIFECYCLE_MAX_FILES: " ", + }), + ).toEqual(DEFAULT_LANGUAGE_LIFECYCLE_CONFIG); + }); + + it.each([ + ["LANGUAGE_LIFECYCLE_ENABLED", "maybe"], + ["LANGUAGE_LIFECYCLE_MAX_FILES", "lots"], + ["LANGUAGE_LIFECYCLE_UNLISTED_ACTION", "ignore"], + ["LANGUAGE_LIFECYCLE_MIN_NOTICE_DAYS", "-3"], + ])("fails fast on %s=%s", (key, value) => { + expect(() => loadLanguageLifecycleConfigFromEnv({ [key]: value })).toThrow( + RangeError, + ); + }); +}); diff --git a/apps/api/src/language-lifecycle/language-lifecycle-config.ts b/apps/api/src/language-lifecycle/language-lifecycle-config.ts new file mode 100644 index 00000000..0d015e42 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-config.ts @@ -0,0 +1,151 @@ +/** + * Configuration surface for the supported-language lifecycle policy. + * + * Defaults reproduce today's behaviour (every language with an analyzer runs) + * until an operator publishes a policy. Values are validated once at + * construction; an invalid value fails fast with a `RangeError` naming the + * offending option rather than silently falling back. + */ + +import { UnlistedLanguageAction } from "./language-lifecycle-types"; + +export interface LanguageLifecycleConfig { + /** + * Master switch and kill switch. When false, new scans analyze every + * language (reason `policy_disabled`). Existing scan pins are still honoured + * so in-flight retries stay deterministic. + */ + enabled: boolean; + /** Shortest allowed gap between `deprecatedAt` and `sunsetAt`, in days. */ + minDeprecationNoticeDays: number; + /** + * What happens to files in a language the policy does not list. `analyze` + * keeps the pre-policy behaviour; `skip` reports them as skipped. + */ + unlistedLanguageAction: UnlistedLanguageAction; + /** Largest number of files one scan plan may classify. */ + maxFilesPerScan: number; + /** Largest number of languages a policy may list. */ + maxLanguages: number; + /** How long scan pins are kept before `purgeExpiredPins` removes them. */ + pinRetentionMs: number; +} + +export const DEFAULT_LANGUAGE_LIFECYCLE_CONFIG: Readonly = + Object.freeze({ + enabled: true, + minDeprecationNoticeDays: 90, + unlistedLanguageAction: "analyze", + maxFilesPerScan: 100_000, + maxLanguages: 64, + pinRetentionMs: 7 * 24 * 60 * 60 * 1000, + }); + +const ENV_KEYS: Record = { + enabled: "LANGUAGE_LIFECYCLE_ENABLED", + minDeprecationNoticeDays: "LANGUAGE_LIFECYCLE_MIN_NOTICE_DAYS", + unlistedLanguageAction: "LANGUAGE_LIFECYCLE_UNLISTED_ACTION", + maxFilesPerScan: "LANGUAGE_LIFECYCLE_MAX_FILES", + maxLanguages: "LANGUAGE_LIFECYCLE_MAX_LANGUAGES", + pinRetentionMs: "LANGUAGE_LIFECYCLE_PIN_RETENTION_MS", +}; + +const UNLISTED_ACTIONS: readonly UnlistedLanguageAction[] = ["analyze", "skip"]; + +/** Merge `overrides` over the defaults and validate the result. */ +export function resolveLanguageLifecycleConfig( + overrides: Partial = {}, +): LanguageLifecycleConfig { + const config: LanguageLifecycleConfig = { + ...DEFAULT_LANGUAGE_LIFECYCLE_CONFIG, + ...stripUndefined(overrides), + }; + + if (typeof config.enabled !== "boolean") { + throw new RangeError("enabled must be a boolean"); + } + requireNonNegativeInteger( + config.minDeprecationNoticeDays, + "minDeprecationNoticeDays", + ); + if (!UNLISTED_ACTIONS.includes(config.unlistedLanguageAction)) { + throw new RangeError( + `unlistedLanguageAction must be one of ${UNLISTED_ACTIONS.join(", ")}`, + ); + } + requirePositiveInteger(config.maxFilesPerScan, "maxFilesPerScan"); + requirePositiveInteger(config.maxLanguages, "maxLanguages"); + requirePositiveInteger(config.pinRetentionMs, "pinRetentionMs"); + return config; +} + +/** + * Build a config from `LANGUAGE_LIFECYCLE_*` environment variables. + * + * Unset or empty variables keep their defaults; set-but-unparseable values + * throw so a typo in deployment config is caught at startup. + */ +export function loadLanguageLifecycleConfigFromEnv( + env: Record = process.env, +): LanguageLifecycleConfig { + const overrides: Record = {}; + for (const [field, envKey] of Object.entries(ENV_KEYS) as [ + keyof LanguageLifecycleConfig, + string, + ][]) { + const raw = env[envKey]; + if (raw === undefined || raw.trim() === "") { + continue; + } + const value = raw.trim(); + if (field === "enabled") { + overrides[field] = parseBoolean(value, envKey); + } else if (field === "unlistedLanguageAction") { + const lowered = value.toLowerCase(); + if (!UNLISTED_ACTIONS.includes(lowered as UnlistedLanguageAction)) { + throw new RangeError( + `${envKey} must be one of ${UNLISTED_ACTIONS.join(", ")}`, + ); + } + overrides[field] = lowered; + } else { + const parsed = Number(value); + if (!Number.isFinite(parsed)) { + throw new RangeError(`${envKey} must be a number`); + } + overrides[field] = parsed; + } + } + return resolveLanguageLifecycleConfig( + overrides as Partial, + ); +} + +function parseBoolean(value: string, envKey: string): boolean { + const lowered = value.toLowerCase(); + if (["true", "1", "yes", "on"].includes(lowered)) { + return true; + } + if (["false", "0", "no", "off"].includes(lowered)) { + return false; + } + throw new RangeError(`${envKey} must be a boolean`); +} + +function stripUndefined(value: T): Partial { + return Object.fromEntries( + Object.entries(value).filter(([, v]) => v !== undefined), + ) as Partial; +} + +function requirePositiveInteger(value: number, field: string): void { + if (!Number.isInteger(value) || value <= 0) { + throw new RangeError(`${field} must be a positive integer`); + } +} + +function requireNonNegativeInteger(value: number, field: string): void { + if (!Number.isInteger(value) || value < 0) { + throw new RangeError(`${field} must be a non-negative integer`); + } +} diff --git a/apps/api/src/language-lifecycle/language-lifecycle-metrics.spec.ts b/apps/api/src/language-lifecycle/language-lifecycle-metrics.spec.ts new file mode 100644 index 00000000..656df073 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-metrics.spec.ts @@ -0,0 +1,123 @@ +import { + LanguageLifecycleMetrics, + consoleLanguageLifecycleLogger, + sanitizeLogFields, +} from "./language-lifecycle-metrics"; +import { LanguageScanPlan } from "./language-lifecycle-types"; + +function plan(overrides: Partial = {}): LanguageScanPlan { + return { + status: "partial", + analyze: new Map([["a.sol", "solidity"]]), + skipped: [ + { path: "secret/tenant/x.js", language: "javascript", reason: "retired" }, + { path: "y.js", language: "javascript", reason: "retired" }, + ], + warnings: [{ language: "vyper", files: 3, message: "m" }], + fingerprint: "f", + ...overrides, + }; +} + +describe("LanguageLifecycleMetrics", () => { + it("counts plans, skipped files, and deprecated files", () => { + const metrics = new LanguageLifecycleMetrics(); + metrics.recordPlan(plan()); + metrics.recordPlan(plan({ status: "complete", skipped: [], warnings: [] })); + metrics.recordPin("created"); + metrics.recordPin("reused"); + metrics.recordPin("reused"); + metrics.recordPublish("forced"); + metrics.recordViolation("SKIPPED_DEPRECATION"); + metrics.recordError("STORE_UNAVAILABLE"); + expect(metrics.snapshot()).toEqual({ + pinsCreated: 1, + pinsReused: 2, + plans: { partial: 1, complete: 1 }, + filesSkipped: { "javascript:retired": 2 }, + deprecatedFiles: { vyper: 3 }, + policyPublishes: { forced: 1 }, + violations: { SKIPPED_DEPRECATION: 1 }, + errors: { STORE_UNAVAILABLE: 1 }, + }); + }); + + it("exports Prometheus text without paths or tenant ids", () => { + const metrics = new LanguageLifecycleMetrics(); + metrics.recordPlan(plan()); + const text = metrics.toPrometheus( + { + version: "1.0.0", + digest: "d", + publishedAt: 0, + languages: [ + { + language: "vyper", + stage: "supported", + extensions: [], + deprecatedAt: "2026-07-01", + sunsetAt: "2027-01-31", + }, + ], + }, + "2026-10-18", + ); + expect(text).toContain( + 'gasguard_language_files_skipped_total{language="javascript",reason="retired"} 2', + ); + expect(text).toContain('gasguard_language_plans_total{status="partial"} 1'); + expect(text).toContain( + 'gasguard_language_stage{language="vyper",stage="deprecated"} 1', + ); + expect(text).not.toMatch(/secret|tenant|\.js"/); + expect(text.endsWith("\n")).toBe(true); + }); +}); + +describe("sanitizeLogFields", () => { + it("drops payload, path and credential keys and non-primitive values", () => { + expect( + sanitizeLogFields({ + tenantId: "t", + count: 2, + ok: true, + filePath: "x.sol", + paths: "a,b", + sourceCode: "contract X {}", + apiKey: "k", + authorization: "Bearer x", + nested: { a: 1 }, + list: [1], + missing: undefined, + }), + ).toEqual({ tenantId: "t", count: 2, ok: true }); + }); + + it("truncates long strings", () => { + const { reason } = sanitizeLogFields({ reason: "x".repeat(500) }); + expect(String(reason)).toHaveLength(201); + }); +}); + +describe("consoleLanguageLifecycleLogger", () => { + it("writes one sanitized JSON line, info to stdout and others to stderr", () => { + const out = jest.spyOn(console, "log").mockImplementation(() => undefined); + const err = jest + .spyOn(console, "error") + .mockImplementation(() => undefined); + try { + consoleLanguageLifecycleLogger.log("info", "e", { a: 1, token: "t" }); + consoleLanguageLifecycleLogger.log("warn", "w", {}); + expect(JSON.parse(out.mock.calls[0][0])).toEqual({ + level: "info", + component: "language-lifecycle", + event: "e", + a: 1, + }); + expect(err).toHaveBeenCalledTimes(1); + } finally { + out.mockRestore(); + err.mockRestore(); + } + }); +}); diff --git a/apps/api/src/language-lifecycle/language-lifecycle-metrics.ts b/apps/api/src/language-lifecycle/language-lifecycle-metrics.ts new file mode 100644 index 00000000..0a362f94 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-metrics.ts @@ -0,0 +1,261 @@ +/** + * Metrics and structured logging for the language lifecycle policy. + * + * Labels are drawn from closed sets (stages, actions, reasons, plan statuses, + * violation and error codes) plus the language, which is a validated slug from + * the operator-controlled policy. Tenant, repository, scan ids and file paths + * never become labels, so cardinality stays bounded and no customer + * identifiers reach a metrics backend. + * + * Log fields pass through `sanitizeLogFields`, which keeps only primitive + * values, truncates long strings, and drops keys that look like they carry + * payloads or credentials. The service logs identifiers, versions, counts and + * codes only; it never logs file paths, source, or store error messages. + */ + +import { + LanguagePlanStatus, + LanguageScanPlan, + SkipReason, + StoredLanguagePolicy, +} from "./language-lifecycle-types"; +import { effectiveStage } from "./language-lifecycle-policy"; + +export interface LanguageLifecycleMetricsSnapshot { + pinsCreated: number; + pinsReused: number; + /** Keyed by plan status. */ + plans: Record; + /** Keyed by `${language}:${reason}`. */ + filesSkipped: Record; + /** Files analyzed in a deprecated language, keyed by language. */ + deprecatedFiles: Record; + /** Keyed by `published`, `unchanged`, or `forced`. */ + policyPublishes: Record; + /** Transition violations seen at publish time, keyed by code. */ + violations: Record; + errors: Record; +} + +export class LanguageLifecycleMetrics { + private readonly plans: Map = new Map(); + private readonly filesSkipped: Map = new Map(); + private readonly deprecatedFiles: Map = new Map(); + private readonly policyPublishes: Map = new Map(); + private readonly violations: Map = new Map(); + private readonly errors: Map = new Map(); + private pinsCreated = 0; + private pinsReused = 0; + + recordPin(result: "created" | "reused"): void { + if (result === "created") { + this.pinsCreated++; + } else { + this.pinsReused++; + } + } + + recordPlan(plan: LanguageScanPlan): void { + increment(this.plans, plan.status); + for (const file of plan.skipped) { + increment(this.filesSkipped, `${file.language}:${file.reason}`); + } + for (const warning of plan.warnings) { + increment(this.deprecatedFiles, warning.language, warning.files); + } + } + + recordPublish(kind: "published" | "unchanged" | "forced"): void { + increment(this.policyPublishes, kind); + } + + recordViolation(code: string): void { + increment(this.violations, code); + } + + recordError(code: string): void { + increment(this.errors, code); + } + + snapshot(): LanguageLifecycleMetricsSnapshot { + return { + pinsCreated: this.pinsCreated, + pinsReused: this.pinsReused, + plans: Object.fromEntries(this.plans), + filesSkipped: Object.fromEntries(this.filesSkipped), + deprecatedFiles: Object.fromEntries(this.deprecatedFiles), + policyPublishes: Object.fromEntries(this.policyPublishes), + violations: Object.fromEntries(this.violations), + errors: Object.fromEntries(this.errors), + }; + } + + /** + * Prometheus text exposition of the counters, plus a per-language stage + * gauge when `policy` and `asOf` are given. + */ + toPrometheus(policy?: StoredLanguagePolicy, asOf?: string): string { + const lines: string[] = []; + const header = (name: string, type: string, help: string) => { + lines.push(`# HELP ${name} ${help}`, `# TYPE ${name} ${type}`); + }; + + header( + "gasguard_language_pins_total", + "counter", + "Scan language pins by whether they were created or reused (retries).", + ); + lines.push( + `gasguard_language_pins_total{result="created"} ${this.pinsCreated}`, + ); + lines.push( + `gasguard_language_pins_total{result="reused"} ${this.pinsReused}`, + ); + header( + "gasguard_language_plans_total", + "counter", + "Scan plans by status (complete, partial, unsupported, empty).", + ); + for (const [status, value] of this.plans) { + lines.push(`gasguard_language_plans_total{status="${status}"} ${value}`); + } + header( + "gasguard_language_files_skipped_total", + "counter", + "Files not analyzed because of the language lifecycle, by language and reason.", + ); + for (const [key, value] of this.filesSkipped) { + const [language, reason] = key.split(":"); + lines.push( + `gasguard_language_files_skipped_total{language="${language}",reason="${reason}"} ${value}`, + ); + } + header( + "gasguard_language_deprecated_files_total", + "counter", + "Files analyzed in a deprecated language, by language.", + ); + for (const [language, value] of this.deprecatedFiles) { + lines.push( + `gasguard_language_deprecated_files_total{language="${language}"} ${value}`, + ); + } + header( + "gasguard_language_policy_publishes_total", + "counter", + "Policy publish calls by result.", + ); + for (const [kind, value] of this.policyPublishes) { + lines.push( + `gasguard_language_policy_publishes_total{result="${kind}"} ${value}`, + ); + } + header( + "gasguard_language_policy_violations_total", + "counter", + "Transition violations found when publishing a policy, by code.", + ); + for (const [code, value] of this.violations) { + lines.push( + `gasguard_language_policy_violations_total{code="${code}"} ${value}`, + ); + } + header( + "gasguard_language_lifecycle_errors_total", + "counter", + "Language lifecycle operations that failed, by error code.", + ); + for (const [code, value] of this.errors) { + lines.push( + `gasguard_language_lifecycle_errors_total{code="${code}"} ${value}`, + ); + } + + if (policy && asOf) { + header( + "gasguard_language_stage", + "gauge", + "1 for each language's effective lifecycle stage in the current policy.", + ); + for (const entry of policy.languages) { + lines.push( + `gasguard_language_stage{language="${entry.language}",stage="${effectiveStage(entry, asOf)}"} 1`, + ); + } + } + return `${lines.join("\n")}\n`; + } +} + +/** Closed label sets, exported for tests and dashboards. */ +export const PLAN_STATUSES: readonly LanguagePlanStatus[] = [ + "complete", + "partial", + "unsupported", + "empty", +]; +export const SKIP_REASONS: readonly SkipReason[] = [ + "retired", + "preview_not_enabled", + "unlisted_language", +]; + +export type LanguageLifecycleLogLevel = "info" | "warn" | "error"; +export type LanguageLifecycleLogFields = Record< + string, + string | number | boolean | undefined +>; + +export interface LanguageLifecycleLogger { + log( + level: LanguageLifecycleLogLevel, + event: string, + fields: LanguageLifecycleLogFields, + ): void; +} + +/** Emits one JSON object per line on stdout/stderr. */ +export const consoleLanguageLifecycleLogger: LanguageLifecycleLogger = { + log(level, event, fields) { + const line = JSON.stringify({ + level, + component: "language-lifecycle", + event, + ...sanitizeLogFields(fields), + }); + if (level === "info") { + console.log(line); + } else { + console.error(line); + } + }, +}; + +const MAX_LOG_STRING = 200; +/** Keys that must never reach a log line, even if a caller passes them. */ +const FORBIDDEN_LOG_KEY = + /payload|document|content|source|body|path|file|secret|token|password|key|credential|authorization/i; + +export function sanitizeLogFields( + fields: Record, +): LanguageLifecycleLogFields { + const safe: LanguageLifecycleLogFields = {}; + for (const [field, value] of Object.entries(fields)) { + if (FORBIDDEN_LOG_KEY.test(field)) { + continue; + } + if (typeof value === "number" || typeof value === "boolean") { + safe[field] = value; + } else if (typeof value === "string") { + safe[field] = + value.length > MAX_LOG_STRING + ? `${value.slice(0, MAX_LOG_STRING)}…` + : value; + } + } + return safe; +} + +function increment(map: Map, key: string, by = 1): void { + map.set(key, (map.get(key) ?? 0) + by); +} diff --git a/apps/api/src/language-lifecycle/language-lifecycle-policy.spec.ts b/apps/api/src/language-lifecycle/language-lifecycle-policy.spec.ts new file mode 100644 index 00000000..61cc43e2 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-policy.spec.ts @@ -0,0 +1,788 @@ +import { readFileSync } from "fs"; +import * as path from "path"; +import { Language } from "@engine/core/analyzer-interface"; +import { + DEFAULT_LANGUAGE_POLICY, + buildPin, + checkPolicyCoverage, + checkPolicyTransition, + compareVersions, + detectLanguages, + effectiveStage, + evaluateDecisions, + isValidDay, + normalizePolicy, + planScan, + policyDigest, + utcDay, +} from "./language-lifecycle-policy"; +import { + LanguageLifecycleEntry, + LanguageLifecycleError, + LanguageLifecyclePolicy, + LanguageScanPin, +} from "./language-lifecycle-types"; + +const LIMITS = { maxLanguages: 64, minDeprecationNoticeDays: 90 }; + +function fixture(name: string): T { + return JSON.parse( + readFileSync(path.join(__dirname, "__fixtures__", name), "utf8"), + ) as T; +} + +function policy( + languages: Partial[], + version = "1.0.0", +): LanguageLifecyclePolicy { + return normalizePolicy( + { + version, + languages: languages.map((l) => ({ extensions: [], ...l })), + }, + LIMITS, + ); +} + +function invalidPolicyErrors(input: unknown, limits = LIMITS): string[] { + try { + normalizePolicy(input, limits); + } catch (error) { + expect(error).toBeInstanceOf(LanguageLifecycleError); + expect((error as LanguageLifecycleError).code).toBe("INVALID_POLICY"); + expect((error as LanguageLifecycleError).retryable).toBe(false); + return (error as LanguageLifecycleError).violations as string[]; + } + throw new Error("expected INVALID_POLICY"); +} + +function pinFor( + p: LanguageLifecyclePolicy, + overrides: Partial[0]> = {}, +): LanguageScanPin { + return buildPin({ + tenantId: "tenant-a", + scanId: "scan-1", + policy: p, + policyDigest: policyDigest(p), + asOf: "2026-10-18", + previewOptIns: [], + enabled: true, + unlistedAction: "analyze", + pinnedAt: 0, + ...overrides, + }); +} + +describe("normalizePolicy", () => { + it("accepts the built-in default, which covers every engine language", () => { + const p = normalizePolicy(DEFAULT_LANGUAGE_POLICY, LIMITS); + expect(p.languages.map((l) => l.language)).toEqual( + Object.values(Language).sort(), + ); + expect(p.languages.every((l) => l.stage === "supported")).toBe(true); + }); + + it("produces the same canonical form and digest regardless of order or extra keys", () => { + const a = normalizePolicy( + { + version: "1.0.0", + languages: [ + { language: "vyper", stage: "supported", extensions: [".vy"] }, + { language: "solidity", stage: "supported", extensions: [".sol"] }, + ], + }, + LIMITS, + ); + const b = normalizePolicy( + { + version: "1.0.0", + note: "ignored", + languages: [ + { + language: "solidity", + stage: "supported", + extensions: [".sol"], + owner: "x", + }, + { language: "vyper", stage: "supported", extensions: [".vy"] }, + ], + }, + LIMITS, + ); + expect(b).toEqual(a); + expect(policyDigest(b)).toBe(policyDigest(a)); + expect(a.languages.map((l) => l.language)).toEqual(["solidity", "vyper"]); + }); + + it("accepts a deprecation exactly at the minimum notice period", () => { + expect(() => + policy([ + { language: "solidity", stage: "supported" }, + { + language: "vyper", + stage: "deprecated", + deprecatedAt: "2026-01-01", + sunsetAt: "2026-04-01", // 90 days + replacement: "solidity", + }, + ]), + ).not.toThrow(); + }); + + it.each<[string, unknown, RegExp]>([ + ["a non-object", "policy", /must be an object/], + ["a bad version", { version: "1.0", languages: [{}] }, /version/], + ["no languages", { version: "1.0.0", languages: [] }, /non-empty/], + [ + "an upper-case language", + { + version: "1.0.0", + languages: [ + { language: "Solidity", stage: "supported", extensions: [] }, + ], + }, + /lower-case slug/, + ], + [ + "a duplicate language", + { + version: "1.0.0", + languages: [ + { language: "rust", stage: "supported", extensions: [] }, + { language: "rust", stage: "preview", extensions: [] }, + ], + }, + /more than once/, + ], + [ + "an unknown stage", + { + version: "1.0.0", + languages: [{ language: "rust", stage: "beta", extensions: [] }], + }, + /stage must be one of/, + ], + [ + "a shared extension", + { + version: "1.0.0", + languages: [ + { language: "rust", stage: "supported", extensions: [".rs"] }, + { language: "soroban", stage: "supported", extensions: [".rs"] }, + ], + }, + /already claimed by "rust"/, + ], + [ + "an extension without a dot", + { + version: "1.0.0", + languages: [ + { language: "rust", stage: "supported", extensions: ["rs"] }, + ], + }, + /extension "rs" is invalid/, + ], + [ + "an impossible date", + { + version: "1.0.0", + languages: [ + { + language: "vyper", + stage: "deprecated", + extensions: [], + deprecatedAt: "2026-02-30", + sunsetAt: "2027-01-01", + }, + ], + }, + /deprecatedAt must be a YYYY-MM-DD date/, + ], + [ + "a deprecation without a sunset", + { + version: "1.0.0", + languages: [ + { + language: "vyper", + stage: "deprecated", + extensions: [], + deprecatedAt: "2026-01-01", + }, + ], + }, + /need deprecatedAt and sunsetAt/, + ], + [ + "a half-scheduled deprecation", + { + version: "1.0.0", + languages: [ + { + language: "vyper", + stage: "supported", + extensions: [], + sunsetAt: "2027-01-01", + }, + ], + }, + /needs both deprecatedAt and sunsetAt/, + ], + [ + "too short a notice", + { + version: "1.0.0", + languages: [ + { + language: "vyper", + stage: "deprecated", + extensions: [], + deprecatedAt: "2026-01-01", + sunsetAt: "2026-03-31", + }, + ], + }, + /at least 90 days/, + ], + [ + "dates on a preview language", + { + version: "1.0.0", + languages: [ + { + language: "move", + stage: "preview", + extensions: [], + sunsetAt: "2027-01-01", + }, + ], + }, + /retire them directly/, + ], + [ + "an unlisted replacement", + { + version: "1.0.0", + languages: [ + { + language: "javascript", + stage: "retired", + extensions: [], + replacement: "typescript", + }, + ], + }, + /"typescript" is not listed/, + ], + [ + "a retired replacement", + { + version: "1.0.0", + languages: [ + { language: "javascript", stage: "retired", extensions: [] }, + { + language: "coffee", + stage: "retired", + extensions: [], + replacement: "javascript", + }, + ], + }, + /"javascript" is retired/, + ], + [ + "a self replacement", + { + version: "1.0.0", + languages: [ + { + language: "rust", + stage: "supported", + extensions: [], + replacement: "rust", + }, + ], + }, + /cannot be the language itself/, + ], + ])("rejects %s", (_name, input, message) => { + expect(invalidPolicyErrors(input).join("\n")).toMatch(message); + }); + + it("rejects more languages than the limit", () => { + const languages = ["a", "b", "c"].map((language) => ({ + language, + stage: "supported", + extensions: [], + })); + expect( + invalidPolicyErrors( + { version: "1.0.0", languages }, + { + ...LIMITS, + maxLanguages: 2, + }, + ).join(), + ).toMatch(/at most 2/); + }); + + it("reports every problem at once", () => { + expect( + invalidPolicyErrors({ + version: "x", + languages: [ + { language: "rust", stage: "nope", extensions: [] }, + { language: "BAD", stage: "supported", extensions: [] }, + ], + }), + ).toHaveLength(3); + }); +}); + +describe("effectiveStage", () => { + const scheduled: LanguageLifecycleEntry = { + language: "vyper", + stage: "supported", + extensions: [], + deprecatedAt: "2026-11-01", + sunsetAt: "2027-02-01", + }; + + it.each([ + ["2026-10-31", "supported"], + ["2026-11-01", "deprecated"], + ["2027-01-31", "deprecated"], + ["2027-02-01", "retired"], + ["2030-01-01", "retired"], + ])("on %s is %s", (asOf, stage) => { + expect(effectiveStage(scheduled, asOf)).toBe(stage); + }); + + it("never moves preview or retired entries", () => { + expect( + effectiveStage( + { language: "move", stage: "preview", extensions: [] }, + "2099-01-01", + ), + ).toBe("preview"); + expect( + effectiveStage( + { language: "js", stage: "retired", extensions: [] }, + "2000-01-01", + ), + ).toBe("retired"); + }); +}); + +describe("evaluateDecisions", () => { + const p = policy([ + { language: "cairo", stage: "preview" }, + { language: "javascript", stage: "retired", replacement: "typescript" }, + { language: "typescript", stage: "supported" }, + ]); + + it("skips preview languages unless the tenant opted in", () => { + expect(evaluateDecisions(p, "2026-10-18", [], true)[0]).toMatchObject({ + language: "cairo", + action: "skip", + reason: "preview_not_enabled", + }); + expect( + evaluateDecisions(p, "2026-10-18", ["cairo"], true)[0], + ).toMatchObject({ + action: "analyze", + reason: "preview_opt_in", + }); + }); + + it("skips retired languages and names the replacement", () => { + expect(evaluateDecisions(p, "2026-10-18", [], true)[1]).toEqual({ + language: "javascript", + stage: "retired", + action: "skip", + reason: "retired", + replacement: "typescript", + }); + }); + + it("analyzes everything when the policy is disabled", () => { + const decisions = evaluateDecisions(p, "2026-10-18", [], false); + expect(decisions.every((d) => d.action === "analyze")).toBe(true); + expect(decisions.every((d) => d.reason === "policy_disabled")).toBe(true); + }); +}); + +describe("checkPolicyTransition", () => { + const base = policy( + [ + { language: "cairo", stage: "preview" }, + { language: "solidity", stage: "supported" }, + { language: "rust", stage: "supported" }, + { + language: "vyper", + stage: "deprecated", + deprecatedAt: "2026-07-01", + sunsetAt: "2027-01-31", + }, + ], + "1.0.0", + ); + const asOf = "2026-10-18"; + const codes = (next: LanguageLifecyclePolicy) => + checkPolicyTransition(base, next, asOf, 90).map((v) => v.code); + const withEntries = ( + replace: Record | null>, + version = "1.1.0", + ) => + policy( + base.languages + .filter((e) => replace[e.language] !== null) + .map((e) => ({ ...e, ...(replace[e.language] ?? {}) })), + version, + ); + + it("accepts a clean minor update", () => { + expect(codes(withEntries({ cairo: { stage: "supported" } }))).toEqual([]); + }); + + it("requires a strictly greater version", () => { + expect(codes(withEntries({}, "1.0.0"))).toEqual(["VERSION_NOT_INCREASED"]); + expect(codes(withEntries({}, "0.9.9"))).toEqual(["VERSION_NOT_INCREASED"]); + }); + + it("forbids dropping a language instead of retiring it", () => { + expect(codes(withEntries({ rust: null }))).toEqual(["LANGUAGE_REMOVED"]); + }); + + it("forbids retiring a supported language without deprecation", () => { + expect(codes(withEntries({ solidity: { stage: "retired" } }))).toEqual([ + "SKIPPED_DEPRECATION", + ]); + }); + + it("allows retiring a preview language directly", () => { + expect(codes(withEntries({ cairo: { stage: "retired" } }))).toEqual([]); + }); + + it("requires the notice period to run from the publish day", () => { + const backdated = { + stage: "deprecated" as const, + deprecatedAt: "2026-07-01", + sunsetAt: "2026-12-01", + }; + expect(codes(withEntries({ rust: backdated }))).toEqual([ + "INSUFFICIENT_NOTICE", + ]); + expect( + codes( + withEntries({ + rust: { ...backdated, sunsetAt: "2027-01-16" }, // exactly 90 days + }), + ), + ).toEqual([]); + }); + + it("forbids pulling a sunset forward but allows pushing it back", () => { + expect(codes(withEntries({ vyper: { sunsetAt: "2027-01-30" } }))).toEqual([ + "SUNSET_PULLED_FORWARD", + ]); + expect(codes(withEntries({ vyper: { sunsetAt: "2027-06-30" } }))).toEqual( + [], + ); + }); + + it("allows reinstating a deprecated language", () => { + expect( + codes( + withEntries({ + vyper: { + stage: "supported", + deprecatedAt: undefined, + sunsetAt: undefined, + }, + }), + ), + ).toEqual([]); + }); +}); + +describe("checkPolicyCoverage", () => { + it("flags analyzable languages with no analyzer and unmanaged analyzer languages", () => { + const p = policy([ + { language: "javascript", stage: "retired" }, + { language: "move", stage: "preview" }, + { language: "solidity", stage: "supported" }, + ]); + expect(checkPolicyCoverage(p, ["solidity", "vyper"], "2026-10-18")).toEqual( + [ + expect.objectContaining({ kind: "no_analyzer", language: "move" }), + expect.objectContaining({ + kind: "unlisted_analyzer_language", + language: "vyper", + }), + ], + ); + }); +}); + +describe("detectLanguages", () => { + const p = normalizePolicy(DEFAULT_LANGUAGE_POLICY, LIMITS); + + it("detects by extension, case-insensitively, and ignores non-source files", () => { + expect( + Object.fromEntries( + detectLanguages( + [ + "src/Token.SOL", + "README.md", + "Makefile", + "lib.rs", + ".vy", + "a.sol/b", + ], + p, + ), + ), + ).toEqual({ "lib.rs": "rust", "src/Token.SOL": "solidity" }); + }); + + it("prefers the longest claimed extension", () => { + const custom = policy([ + { language: "typescript", stage: "supported", extensions: [".ts"] }, + { language: "tsdecl", stage: "supported", extensions: [".d.ts"] }, + ]); + expect( + Object.fromEntries(detectLanguages(["x.d.ts", "y.ts"], custom)), + ).toEqual({ "x.d.ts": "tsdecl", "y.ts": "typescript" }); + }); + + it("rejects malformed paths", () => { + expect(() => detectLanguages(["ok.sol", ""], p)).toThrow( + LanguageLifecycleError, + ); + }); +}); + +describe("buildPin", () => { + const p = policy([{ language: "solidity", stage: "supported" }]); + + it("is deterministic and independent of ids and pin time", () => { + const a = pinFor(p); + const b = pinFor(p, { tenantId: "other", scanId: "s2", pinnedAt: 99 }); + expect(b.fingerprint).toBe(a.fingerprint); + }); + + it("changes the fingerprint when anything that affects analysis changes", () => { + const base = pinFor(p).fingerprint; + expect(pinFor(p, { asOf: "2026-10-19" }).fingerprint).not.toBe(base); + expect(pinFor(p, { unlistedAction: "skip" }).fingerprint).not.toBe(base); + expect(pinFor(p, { enabled: false }).fingerprint).not.toBe(base); + expect(pinFor(p, { policyDigest: "other" }).fingerprint).not.toBe(base); + }); + + it("forces unlisted languages to be analyzed when disabled", () => { + expect( + pinFor(p, { enabled: false, unlistedAction: "skip" }).unlistedAction, + ).toBe("analyze"); + }); +}); + +describe("planScan", () => { + const p = policy([ + { language: "cairo", stage: "preview" }, + { language: "javascript", stage: "retired", replacement: "typescript" }, + { language: "solidity", stage: "supported" }, + { language: "typescript", stage: "supported" }, + { + language: "vyper", + stage: "deprecated", + deprecatedAt: "2026-07-01", + sunsetAt: "2027-01-31", + replacement: "solidity", + }, + ]); + const map = (entries: Record) => + new Map(Object.entries(entries)); + + it("is complete when every file is analyzable", () => { + const plan = planScan( + pinFor(p), + map({ "b.sol": "solidity", "a.ts": "typescript" }), + 10, + ); + expect(plan.status).toBe("complete"); + expect([...plan.analyze.keys()]).toEqual(["a.ts", "b.sol"]); + expect(plan.skipped).toEqual([]); + }); + + it("is partial, with reasons, when some files are skipped", () => { + const plan = planScan( + pinFor(p), + map({ "a.sol": "solidity", "z.js": "javascript", "m.cairo": "cairo" }), + 10, + ); + expect(plan.status).toBe("partial"); + expect(plan.skipped).toEqual([ + { path: "m.cairo", language: "cairo", reason: "preview_not_enabled" }, + { path: "z.js", language: "javascript", reason: "retired" }, + ]); + }); + + it("is unsupported when nothing can be analyzed", () => { + expect(planScan(pinFor(p), map({ "z.js": "javascript" }), 10).status).toBe( + "unsupported", + ); + }); + + it("is empty when there are no source files", () => { + expect(planScan(pinFor(p), new Map(), 10).status).toBe("empty"); + }); + + it("warns once per deprecated language with a file count", () => { + const plan = planScan( + pinFor(p), + map({ "a.vy": "vyper", "b.vy": "vyper", "c.sol": "solidity" }), + 10, + ); + expect(plan.warnings).toEqual([ + { + language: "vyper", + files: 2, + sunsetAt: "2027-01-31", + replacement: "solidity", + message: + 'Language "vyper" is deprecated and will no longer be analyzed from 2027-01-31. Migrate to "solidity".', + }, + ]); + }); + + it("treats unlisted languages per the pinned action", () => { + const files = map({ "a.move": "move", "b.sol": "solidity" }); + expect(planScan(pinFor(p), files, 10).analyze.get("a.move")).toBe("move"); + expect( + planScan(pinFor(p, { unlistedAction: "skip" }), files, 10).skipped, + ).toEqual([ + { path: "a.move", language: "move", reason: "unlisted_language" }, + ]); + }); + + it("normalizes language case from the engine map", () => { + expect(planScan(pinFor(p), map({ "a.sol": "Solidity" }), 10).status).toBe( + "complete", + ); + }); + + it("accepts exactly maxFiles files and rejects one more", () => { + const files = map({ "a.sol": "solidity", "b.sol": "solidity" }); + expect(planScan(pinFor(p), files, 2).analyze.size).toBe(2); + expect(() => planScan(pinFor(p), files, 1)).toThrow(/at most 1 files/); + }); + + it.each([ + ["a non-map", {} as Map], + ["an empty language", new Map([["a.sol", " "]])], + ["a non-string language", new Map([["a.sol", 7 as unknown as string]])], + ["a NUL in a path", new Map([["a\u0000.sol", "solidity"]])], + ])("rejects %s as INVALID_INPUT", (_name, languageMap) => { + try { + planScan(pinFor(p), languageMap, 10); + throw new Error("expected a throw"); + } catch (error) { + expect((error as LanguageLifecycleError).code).toBe("INVALID_INPUT"); + } + }); + + it("gives identical plans regardless of input order", () => { + const forward = planScan( + pinFor(p), + map({ "a.sol": "solidity", "b.js": "javascript", "c.vy": "vyper" }), + 10, + ); + const reversed = planScan( + pinFor(p), + new Map([ + ["c.vy", "vyper"], + ["b.js", "javascript"], + ["a.sol", "solidity"], + ]), + 10, + ); + expect([...reversed.analyze]).toEqual([...forward.analyze]); + expect(reversed).toEqual(forward); + }); +}); + +describe("helpers", () => { + it("formats UTC days", () => { + expect(utcDay(Date.UTC(2026, 9, 18, 23, 59, 59))).toBe("2026-10-18"); + expect(() => utcDay(Number.NaN)).toThrow(LanguageLifecycleError); + }); + + it("validates days", () => { + expect(isValidDay("2028-02-29")).toBe(true); + expect(isValidDay("2027-02-29")).toBe(false); + expect(isValidDay("2026-1-01")).toBe(false); + expect(isValidDay(20261018)).toBe(false); + }); + + it("compares versions numerically", () => { + expect(compareVersions("1.10.0", "1.9.0")).toBeGreaterThan(0); + expect(compareVersions("1.0.0", "1.0.0")).toBe(0); + expect(compareVersions("0.0.0", "0.0.1")).toBeLessThan(0); + }); +}); + +/** + * Frozen decisions and fingerprints around the scheduled deprecation and + * sunset boundaries. A change here means scan results for existing commits + * would no longer be reproducible, so update the fixture only deliberately. + */ +describe("regression: sunset boundaries", () => { + interface RegressionFixture { + policyDigest: string; + cases: { + asOf: string; + previewOptIns: string[]; + decisions: unknown[]; + fingerprint: string; + }[]; + detection: Record; + } + const frozen = fixture("sunset-boundary-regression.json"); + const p = normalizePolicy(fixture("sunset-boundary-policy.json"), LIMITS); + + it("keeps the canonical policy digest", () => { + expect(policyDigest(p)).toBe(frozen.policyDigest); + }); + + it.each( + frozen.cases.map((c) => [c.asOf, c.previewOptIns.join(",") || "-", c]), + )("on %s with opt-ins %s", (_asOf, _optIns, expected) => { + const pin = pinFor(p, { + asOf: expected.asOf, + previewOptIns: expected.previewOptIns, + }); + expect(pin.decisions).toEqual(expected.decisions); + expect(pin.fingerprint).toBe(expected.fingerprint); + }); + + it("keeps extension detection stable", () => { + expect( + Object.fromEntries( + detectLanguages( + [ + "Contracts/Token.SOL", + "a/b.vy", + "lib/x.cairo", + "noext", + ".vy", + "dir.sol/readme", + ], + p, + ), + ), + ).toEqual(frozen.detection); + }); +}); diff --git a/apps/api/src/language-lifecycle/language-lifecycle-policy.ts b/apps/api/src/language-lifecycle/language-lifecycle-policy.ts new file mode 100644 index 00000000..8f49beac --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-policy.ts @@ -0,0 +1,722 @@ +/** + * Pure policy logic for the supported-language lifecycle: validation, + * scheduled stage evaluation, governance checks between policy versions, + * language detection, scan pins, and scan planning. + * + * Nothing here reads the clock, the environment, or storage. Every function is + * a deterministic function of its arguments, which is what lets a scan's + * decisions be pinned and reproduced exactly on retry or re-run. + */ + +import * as crypto from "crypto"; +import { Language } from "@engine/core/analyzer-interface"; +import { + LIFECYCLE_STAGES, + LanguageDecision, + LanguageLifecycleEntry, + LanguageLifecycleError, + LanguageLifecyclePolicy, + LanguageScanPin, + LanguageScanPlan, + LanguageWarning, + LifecycleStage, + PolicyViolation, + SkippedFile, + UnlistedLanguageAction, +} from "./language-lifecycle-types"; + +const LANGUAGE_PATTERN = /^[a-z][a-z0-9_-]{0,31}$/; +/** `.sol`, or compound up to three segments such as `.d.ts`; length-capped below. */ +const EXTENSION_PATTERN = /^(?:\.[a-z0-9][a-z0-9+_-]*){1,3}$/; +const MAX_EXTENSION_LENGTH = 16; +const VERSION_PATTERN = + /^(0|[1-9]\d{0,5})\.(0|[1-9]\d{0,5})\.(0|[1-9]\d{0,5})$/; +const DAY_PATTERN = /^\d{4}-\d{2}-\d{2}$/; +const MAX_IDENTIFIER_LENGTH = 200; +const MAX_PATH_LENGTH = 4096; +const DAY_MS = 24 * 60 * 60 * 1000; + +/** + * Policy equivalent to the behaviour before lifecycle management: every + * language the engine knows is `supported`. Used until an operator publishes. + */ +export const DEFAULT_LANGUAGE_POLICY: Readonly = + Object.freeze({ + version: "0.0.0", + languages: [ + { language: Language.CAIRO, stage: "supported", extensions: [".cairo"] }, + { + language: Language.JAVASCRIPT, + stage: "supported", + extensions: [".cjs", ".js", ".mjs"], + }, + { language: Language.MOVE, stage: "supported", extensions: [".move"] }, + { language: Language.RUST, stage: "supported", extensions: [".rs"] }, + // Soroban contracts are Rust files; callers tell them apart. + { language: Language.SOROBAN, stage: "supported", extensions: [] }, + { language: Language.SOLIDITY, stage: "supported", extensions: [".sol"] }, + { + language: Language.TYPESCRIPT, + stage: "supported", + extensions: [".cts", ".mts", ".ts"], + }, + { language: Language.VYPER, stage: "supported", extensions: [".vy"] }, + ] as LanguageLifecycleEntry[], + }); + +export interface PolicyLimits { + maxLanguages: number; + minDeprecationNoticeDays: number; +} + +/** + * Validate `input` and return a canonical copy: unknown keys dropped, + * languages sorted, extensions sorted. Two policies that mean the same thing + * produce the same canonical form and therefore the same digest. + * + * @throws LanguageLifecycleError INVALID_POLICY listing every problem found + */ +export function normalizePolicy( + input: unknown, + limits: PolicyLimits, +): LanguageLifecyclePolicy { + const errors: string[] = []; + if (!isRecord(input)) { + throw invalidPolicy(["policy must be an object"]); + } + if ( + typeof input.version !== "string" || + !VERSION_PATTERN.test(input.version) + ) { + errors.push("version must be MAJOR.MINOR.PATCH"); + } + if (!Array.isArray(input.languages) || input.languages.length === 0) { + throw invalidPolicy([...errors, "languages must be a non-empty array"]); + } + if (input.languages.length > limits.maxLanguages) { + throw invalidPolicy([ + ...errors, + `languages may list at most ${limits.maxLanguages} entries`, + ]); + } + + const entries: LanguageLifecycleEntry[] = []; + const seenLanguages = new Set(); + const extensionOwner = new Map(); + + input.languages.forEach((raw: unknown, index: number) => { + const where = `languages[${index}]`; + if (!isRecord(raw)) { + errors.push(`${where} must be an object`); + return; + } + const language = raw.language; + if (typeof language !== "string" || !LANGUAGE_PATTERN.test(language)) { + errors.push(`${where}.language must be a lower-case slug`); + return; + } + const at = `language "${language}"`; + if (seenLanguages.has(language)) { + errors.push(`${at} is listed more than once`); + return; + } + seenLanguages.add(language); + + const stage = raw.stage as LifecycleStage; + if (!LIFECYCLE_STAGES.includes(stage)) { + errors.push(`${at}: stage must be one of ${LIFECYCLE_STAGES.join(", ")}`); + return; + } + + const extensions: string[] = []; + if (!Array.isArray(raw.extensions)) { + errors.push(`${at}: extensions must be an array`); + } else { + for (const ext of raw.extensions) { + if ( + typeof ext !== "string" || + ext.length > MAX_EXTENSION_LENGTH || + !EXTENSION_PATTERN.test(ext) + ) { + errors.push(`${at}: extension ${JSON.stringify(ext)} is invalid`); + continue; + } + const owner = extensionOwner.get(ext); + if (owner !== undefined && owner !== language) { + errors.push( + `${at}: extension "${ext}" is already claimed by "${owner}"`, + ); + continue; + } + if (owner === undefined) { + extensionOwner.set(ext, language); + extensions.push(ext); + } + } + } + + const entry: LanguageLifecycleEntry = { + language, + stage, + extensions: extensions.sort(compareStrings), + }; + for (const field of ["deprecatedAt", "sunsetAt"] as const) { + const value = raw[field]; + if (value === undefined) { + continue; + } + if (!isValidDay(value)) { + errors.push(`${at}: ${field} must be a YYYY-MM-DD date`); + } else { + entry[field] = value; + } + } + if (raw.replacement !== undefined) { + if ( + typeof raw.replacement !== "string" || + !LANGUAGE_PATTERN.test(raw.replacement) + ) { + errors.push(`${at}: replacement must be a lower-case slug`); + } else { + entry.replacement = raw.replacement; + } + } + + errors.push(...checkEntryDates(entry, at, limits.minDeprecationNoticeDays)); + entries.push(entry); + }); + + const byLanguage = new Map(entries.map((e) => [e.language, e])); + for (const entry of entries) { + if (entry.replacement === undefined) { + continue; + } + const target = byLanguage.get(entry.replacement); + const at = `language "${entry.language}"`; + if (entry.replacement === entry.language) { + errors.push(`${at}: replacement cannot be the language itself`); + } else if (!target) { + errors.push(`${at}: replacement "${entry.replacement}" is not listed`); + } else if (target.stage === "retired") { + errors.push(`${at}: replacement "${entry.replacement}" is retired`); + } + } + + if (errors.length > 0) { + throw invalidPolicy(errors); + } + return { + version: input.version as string, + languages: entries.sort((a, b) => compareStrings(a.language, b.language)), + }; +} + +function checkEntryDates( + entry: LanguageLifecycleEntry, + at: string, + minNoticeDays: number, +): string[] { + const errors: string[] = []; + const { stage, deprecatedAt, sunsetAt } = entry; + if (stage === "preview" && (deprecatedAt || sunsetAt)) { + errors.push( + `${at}: preview languages carry no deprecation dates; retire them directly`, + ); + } + if (stage === "deprecated" && (!deprecatedAt || !sunsetAt)) { + errors.push(`${at}: deprecated languages need deprecatedAt and sunsetAt`); + } + if (stage === "supported" && (deprecatedAt || sunsetAt)) { + if (!deprecatedAt || !sunsetAt) { + errors.push( + `${at}: a scheduled deprecation needs both deprecatedAt and sunsetAt`, + ); + } + } + if (stage !== "retired" && deprecatedAt && sunsetAt) { + const noticeDays = dayDifference(deprecatedAt, sunsetAt); + if (noticeDays < minNoticeDays) { + errors.push( + `${at}: sunsetAt must be at least ${minNoticeDays} days after deprecatedAt`, + ); + } + } + return errors; +} + +/** SHA-256 of the canonical policy (call on a `normalizePolicy` result). */ +export function policyDigest(policy: LanguageLifecyclePolicy): string { + return sha256( + JSON.stringify({ version: policy.version, languages: policy.languages }), + ); +} + +/** Stage of `entry` on day `asOf`, after applying its scheduled dates. */ +export function effectiveStage( + entry: LanguageLifecycleEntry, + asOf: string, +): LifecycleStage { + if (entry.stage === "retired" || entry.stage === "preview") { + return entry.stage; + } + if (entry.sunsetAt !== undefined && asOf >= entry.sunsetAt) { + return "retired"; + } + if (entry.deprecatedAt !== undefined && asOf >= entry.deprecatedAt) { + return "deprecated"; + } + return entry.stage; +} + +/** One decision per policy language, sorted by language. */ +export function evaluateDecisions( + policy: LanguageLifecyclePolicy, + asOf: string, + previewOptIns: readonly string[], + enabled: boolean, +): LanguageDecision[] { + const optedIn = new Set(previewOptIns); + return [...policy.languages] + .sort((a, b) => compareStrings(a.language, b.language)) + .map((entry) => { + const stage = effectiveStage(entry, asOf); + const decision: LanguageDecision = { + language: entry.language, + stage, + ...decide(stage, enabled, optedIn.has(entry.language)), + }; + if (stage === "deprecated" || stage === "retired") { + if (entry.sunsetAt !== undefined) { + decision.sunsetAt = entry.sunsetAt; + } + if (entry.replacement !== undefined) { + decision.replacement = entry.replacement; + } + } + return decision; + }); +} + +function decide( + stage: LifecycleStage, + enabled: boolean, + optedIn: boolean, +): Pick { + if (!enabled) { + return { action: "analyze", reason: "policy_disabled" }; + } + switch (stage) { + case "preview": + return optedIn + ? { action: "analyze", reason: "preview_opt_in" } + : { action: "skip", reason: "preview_not_enabled" }; + case "supported": + return { action: "analyze", reason: "supported" }; + case "deprecated": + return { action: "analyze", reason: "deprecated" }; + case "retired": + return { action: "skip", reason: "retired" }; + } +} + +/** + * Governance checks for replacing `previous` with `next` on day `asOf`. + * Both must be canonical. An empty result means the change is safe for + * tenants; anything else needs an explicit, reasoned override. + */ +export function checkPolicyTransition( + previous: LanguageLifecyclePolicy, + next: LanguageLifecyclePolicy, + asOf: string, + minNoticeDays: number, +): PolicyViolation[] { + const violations: PolicyViolation[] = []; + if (compareVersions(next.version, previous.version) <= 0) { + violations.push({ + code: "VERSION_NOT_INCREASED", + message: `version ${next.version} must be greater than ${previous.version}`, + }); + } + + const nextByLanguage = new Map(next.languages.map((e) => [e.language, e])); + for (const before of previous.languages) { + const language = before.language; + const after = nextByLanguage.get(language); + if (!after) { + violations.push({ + code: "LANGUAGE_REMOVED", + language, + message: `"${language}" was removed; retire it instead so its files are reported as skipped`, + }); + continue; + } + const stageBefore = effectiveStage(before, asOf); + const stageAfter = effectiveStage(after, asOf); + if (stageBefore === "supported" && stageAfter === "retired") { + violations.push({ + code: "SKIPPED_DEPRECATION", + language, + message: `"${language}" goes from supported to retired without a deprecation period`, + }); + continue; + } + if ( + stageBefore !== "retired" && + after.sunsetAt !== undefined && + stageAfter !== "retired" + ) { + if (before.sunsetAt === undefined) { + if (dayDifference(asOf, after.sunsetAt) < minNoticeDays) { + violations.push({ + code: "INSUFFICIENT_NOTICE", + language, + message: `"${language}" sunsets on ${after.sunsetAt}, less than ${minNoticeDays} days from ${asOf}`, + }); + } + } else if (after.sunsetAt < before.sunsetAt) { + violations.push({ + code: "SUNSET_PULLED_FORWARD", + language, + message: `"${language}" sunset moved from ${before.sunsetAt} to ${after.sunsetAt}`, + }); + } + } + } + return violations; +} + +export type CoverageIssueKind = "no_analyzer" | "unlisted_analyzer_language"; + +export interface CoverageIssue { + kind: CoverageIssueKind; + language: string; + message: string; +} + +/** + * Compare a policy with the languages analyzers actually claim (e.g. + * `AnalyzerRegistry.getSupportedLanguages()` or a compatibility matrix). + * + * A language the policy promises to analyze with no analyzer would produce + * zero findings that read as clean; an analyzed language the policy does not + * list escapes lifecycle management. Retired languages need no analyzer. + */ +export function checkPolicyCoverage( + policy: LanguageLifecyclePolicy, + analyzerLanguages: readonly string[], + asOf: string, +): CoverageIssue[] { + const available = new Set(analyzerLanguages.map(String)); + const listed = new Set(policy.languages.map((e) => e.language)); + const issues: CoverageIssue[] = []; + for (const entry of policy.languages) { + if (effectiveStage(entry, asOf) === "retired") { + continue; + } + if (!available.has(entry.language)) { + issues.push({ + kind: "no_analyzer", + language: entry.language, + message: `"${entry.language}" is ${effectiveStage(entry, asOf)} but no analyzer supports it`, + }); + } + } + for (const language of [...available].sort(compareStrings)) { + if (!listed.has(language)) { + issues.push({ + kind: "unlisted_analyzer_language", + language, + message: `an analyzer supports "${language}" but the policy does not list it`, + }); + } + } + return issues; +} + +/** + * Map each path whose extension a policy entry claims to that language. + * Matching is case-insensitive on the extension only; the longest claimed + * extension wins, and paths with no claimed extension are left out. + */ +export function detectLanguages( + paths: readonly string[], + policy: LanguageLifecyclePolicy, +): Map { + const byExtension = new Map(); + for (const entry of policy.languages) { + for (const ext of entry.extensions) { + byExtension.set(ext, entry.language); + } + } + const result = new Map(); + for (const filePath of [...paths].sort(compareStrings)) { + requirePath(filePath); + const base = filePath.slice(filePath.lastIndexOf("/") + 1).toLowerCase(); + let match: string | undefined; + let dot = base.indexOf(".", 1); + while (dot !== -1 && match === undefined) { + // Longest first: `a.test.ts` tries `.test.ts`, then `.ts`. + match = byExtension.get(base.slice(dot)); + dot = base.indexOf(".", dot + 1); + } + if (match !== undefined) { + result.set(filePath, match); + } + } + return result; +} + +export interface PinInput { + tenantId: string; + scanId: string; + policy: LanguageLifecyclePolicy; + policyDigest: string; + asOf: string; + previewOptIns: readonly string[]; + enabled: boolean; + unlistedAction: UnlistedLanguageAction; + pinnedAt: number; +} + +export function buildPin(input: PinInput): LanguageScanPin { + const decisions = evaluateDecisions( + input.policy, + input.asOf, + input.previewOptIns, + input.enabled, + ); + // With the policy off, nothing is skipped, matching pre-policy behaviour. + const unlistedAction: UnlistedLanguageAction = input.enabled + ? input.unlistedAction + : "analyze"; + const body = { + policyVersion: input.policy.version, + policyDigest: input.policyDigest, + asOf: input.asOf, + decisions, + unlistedAction, + }; + return { + tenantId: input.tenantId, + scanId: input.scanId, + ...body, + fingerprint: sha256(JSON.stringify(body)), + pinnedAt: input.pinnedAt, + }; +} + +/** + * Split a repository's files into what will and will not be analyzed under + * `pin`. `languageMap` is the engine's path → language map (see + * `AnalyzerRegistry.analyzeMultiple`); files missing from it are not source + * and are ignored, exactly as the registry ignores them. + * + * @throws LanguageLifecycleError INVALID_INPUT for malformed paths or + * languages, or more than `maxFiles` entries + */ +export function planScan( + pin: LanguageScanPin, + languageMap: ReadonlyMap, + maxFiles: number, +): LanguageScanPlan { + if (!(languageMap instanceof Map)) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "languageMap must be a Map of path to language", + ); + } + if (languageMap.size > maxFiles) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + `a scan may classify at most ${maxFiles} files`, + ); + } + const decisions = new Map(pin.decisions.map((d) => [d.language, d])); + const analyze = new Map(); + const skipped: SkippedFile[] = []; + const deprecatedCounts = new Map(); + + for (const filePath of [...languageMap.keys()].sort(compareStrings)) { + requirePath(filePath); + const raw = languageMap.get(filePath); + if (typeof raw !== "string" || raw.trim() === "") { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "every language in languageMap must be a non-empty string", + ); + } + const language = raw.trim().toLowerCase(); + const decision = decisions.get(language); + if (!decision) { + if (pin.unlistedAction === "analyze") { + analyze.set(filePath, language); + } else { + skipped.push({ path: filePath, language, reason: "unlisted_language" }); + } + continue; + } + if (decision.action === "analyze") { + analyze.set(filePath, language); + if (decision.stage === "deprecated") { + deprecatedCounts.set( + language, + (deprecatedCounts.get(language) ?? 0) + 1, + ); + } + } else { + skipped.push({ + path: filePath, + language, + reason: + decision.reason === "preview_not_enabled" + ? "preview_not_enabled" + : "retired", + }); + } + } + + const warnings: LanguageWarning[] = [...deprecatedCounts.keys()] + .sort(compareStrings) + .map((language) => { + const decision = decisions.get(language)!; + return deprecationWarning(decision, deprecatedCounts.get(language)!); + }); + + return { + status: + analyze.size === 0 + ? skipped.length === 0 + ? "empty" + : "unsupported" + : skipped.length === 0 + ? "complete" + : "partial", + analyze, + skipped, + warnings, + fingerprint: pin.fingerprint, + }; +} + +function deprecationWarning( + decision: LanguageDecision, + files: number, +): LanguageWarning { + const sunset = decision.sunsetAt + ? ` and will no longer be analyzed from ${decision.sunsetAt}` + : ""; + const replacement = decision.replacement + ? ` Migrate to "${decision.replacement}".` + : ""; + const warning: LanguageWarning = { + language: decision.language, + files, + message: `Language "${decision.language}" is deprecated${sunset}.${replacement}`, + }; + if (decision.sunsetAt) { + warning.sunsetAt = decision.sunsetAt; + } + if (decision.replacement) { + warning.replacement = decision.replacement; + } + return warning; +} + +/** UTC day (`YYYY-MM-DD`) of a millisecond timestamp. */ +export function utcDay(timestamp: number): string { + if (!Number.isFinite(timestamp)) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "timestamp must be finite", + ); + } + return new Date(timestamp).toISOString().slice(0, 10); +} + +export function isValidDay(value: unknown): value is string { + if (typeof value !== "string" || !DAY_PATTERN.test(value)) { + return false; + } + const parsed = Date.parse(`${value}T00:00:00Z`); + return Number.isFinite(parsed) && utcDayUnchecked(parsed) === value; +} + +/** Negative when `a < b`; compares `MAJOR.MINOR.PATCH` numerically. */ +export function compareVersions(a: string, b: string): number { + const left = a.split(".").map(Number); + const right = b.split(".").map(Number); + for (let i = 0; i < 3; i++) { + if (left[i] !== right[i]) { + return left[i] - right[i]; + } + } + return 0; +} + +/** Code-point order, independent of the host locale. */ +export function compareStrings(a: string, b: string): number { + return a === b ? 0 : a < b ? -1 : 1; +} + +export function requireIdentifier(value: unknown, field: string): void { + if ( + typeof value !== "string" || + value.trim() === "" || + value.length > MAX_IDENTIFIER_LENGTH || + /[\u0000-\u001f\u007f]/.test(value) + ) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + `${field} must be a non-empty string of at most ${MAX_IDENTIFIER_LENGTH} printable characters`, + ); + } +} + +export function requireLanguage(value: unknown): void { + if (typeof value !== "string" || !LANGUAGE_PATTERN.test(value)) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "language must be a lower-case slug", + ); + } +} + +function requirePath(value: unknown): void { + if ( + typeof value !== "string" || + value === "" || + value.length > MAX_PATH_LENGTH || + value.includes("\u0000") + ) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + `file paths must be non-empty strings of at most ${MAX_PATH_LENGTH} characters`, + ); + } +} + +function dayDifference(from: string, to: string): number { + return Math.round( + (Date.parse(`${to}T00:00:00Z`) - Date.parse(`${from}T00:00:00Z`)) / DAY_MS, + ); +} + +function utcDayUnchecked(timestamp: number): string { + return new Date(timestamp).toISOString().slice(0, 10); +} + +function sha256(value: string): string { + return crypto.createHash("sha256").update(value, "utf8").digest("hex"); +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function invalidPolicy(errors: string[]): LanguageLifecycleError { + return new LanguageLifecycleError( + "INVALID_POLICY", + `policy is invalid: ${errors.join("; ")}`, + errors, + ); +} diff --git a/apps/api/src/language-lifecycle/language-lifecycle-store.spec.ts b/apps/api/src/language-lifecycle/language-lifecycle-store.spec.ts new file mode 100644 index 00000000..394ed243 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-store.spec.ts @@ -0,0 +1,223 @@ +import { promises as fs } from "fs"; +import * as os from "os"; +import * as path from "path"; +import { + FileLanguageLifecycleStore, + InMemoryLanguageLifecycleStore, + LanguageLifecycleStore, +} from "./language-lifecycle-store"; +import { + LanguageScanPin, + StoredLanguagePolicy, +} from "./language-lifecycle-types"; + +function storedPolicy(version: string, digest: string): StoredLanguagePolicy { + return { + version, + languages: [ + { language: "solidity", stage: "supported", extensions: [".sol"] }, + ], + digest, + publishedAt: 1, + }; +} + +function pin( + tenantId: string, + scanId: string, + pinnedAt = 1, + fingerprint = "f", +): LanguageScanPin { + return { + tenantId, + scanId, + policyVersion: "1.0.0", + policyDigest: "d", + asOf: "2026-10-18", + decisions: [], + unlistedAction: "analyze", + fingerprint, + pinnedAt, + }; +} + +const tempDirs: string[] = []; + +afterAll(async () => { + await Promise.all( + tempDirs.map((dir) => fs.rm(dir, { recursive: true, force: true })), + ); +}); + +async function fileStore(): Promise { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "gasguard-langlc-")); + tempDirs.push(dir); + return new FileLanguageLifecycleStore(dir); +} + +describe.each<[string, () => Promise]>([ + [ + "InMemoryLanguageLifecycleStore", + async () => new InMemoryLanguageLifecycleStore(), + ], + ["FileLanguageLifecycleStore", fileStore], +])("%s", (_name, create) => { + let store: LanguageLifecycleStore; + + beforeEach(async () => { + store = await create(); + }); + + describe("policy", () => { + it("starts empty", async () => { + expect(await store.getPolicy()).toBeUndefined(); + expect(await store.listPolicyHistory()).toEqual([]); + }); + + it("writes only against the expected digest", async () => { + expect(await store.putPolicy(storedPolicy("1.0.0", "d1"), "stale")).toBe( + "conflict", + ); + expect(await store.putPolicy(storedPolicy("1.0.0", "d1"), null)).toBe( + "written", + ); + expect(await store.putPolicy(storedPolicy("1.1.0", "d2"), null)).toBe( + "conflict", + ); + expect(await store.putPolicy(storedPolicy("1.1.0", "d2"), "d1")).toBe( + "written", + ); + expect((await store.getPolicy())?.version).toBe("1.1.0"); + }); + + it("keeps an ordered history of published policies", async () => { + await store.putPolicy(storedPolicy("1.0.0", "d1"), null); + await store.putPolicy(storedPolicy("1.1.0", "d2"), "d1"); + await store.putPolicy(storedPolicy("2.0.0", "d3"), "d1"); // conflict + expect((await store.listPolicyHistory()).map((p) => p.version)).toEqual([ + "1.0.0", + "1.1.0", + ]); + }); + + it("lets exactly one of two concurrent publishes win", async () => { + await store.putPolicy(storedPolicy("1.0.0", "d1"), null); + const results = await Promise.all([ + store.putPolicy(storedPolicy("1.1.0", "a"), "d1"), + store.putPolicy(storedPolicy("1.1.0", "b"), "d1"), + ]); + expect([...results].sort()).toEqual(["conflict", "written"]); + }); + + it("returns copies that cannot mutate stored state", async () => { + await store.putPolicy(storedPolicy("1.0.0", "d1"), null); + const read = await store.getPolicy(); + read!.languages[0].stage = "retired"; + expect((await store.getPolicy())!.languages[0].stage).toBe("supported"); + }); + }); + + describe("tenant settings", () => { + it("keeps settings per tenant", async () => { + await store.putTenantSettings({ + tenantId: "tenant-a", + previewOptIns: ["cairo"], + updatedAt: 1, + }); + expect(await store.getTenantSettings("tenant-b")).toBeUndefined(); + expect( + (await store.getTenantSettings("tenant-a"))?.previewOptIns, + ).toEqual(["cairo"]); + }); + }); + + describe("pins", () => { + it("are write-once", async () => { + expect(await store.createPin(pin("t", "s", 1, "first"))).toBe("written"); + expect(await store.createPin(pin("t", "s", 2, "second"))).toBe("exists"); + expect((await store.getPin("t", "s"))?.fingerprint).toBe("first"); + }); + + it("are isolated per tenant even when scan ids collide", async () => { + await store.createPin(pin("tenant-a", "scan-1", 1, "a")); + await store.createPin(pin("tenant-b", "scan-1", 1, "b")); + expect((await store.getPin("tenant-a", "scan-1"))?.fingerprint).toBe("a"); + expect((await store.getPin("tenant-b", "scan-1"))?.fingerprint).toBe("b"); + expect(await store.getPin("tenant-c", "scan-1")).toBeUndefined(); + }); + + it("are purged strictly before the cutoff", async () => { + await store.createPin(pin("t", "old", 99)); + await store.createPin(pin("t", "edge", 100)); + await store.createPin(pin("u", "new", 101)); + expect(await store.deletePinsBefore(100)).toBe(1); + expect(await store.getPin("t", "old")).toBeUndefined(); + expect(await store.getPin("t", "edge")).toBeDefined(); + expect(await store.deletePinsBefore(100)).toBe(0); + }); + }); +}); + +describe("FileLanguageLifecycleStore", () => { + it("rejects an empty root", () => { + expect(() => new FileLanguageLifecycleStore(" ")).toThrow(TypeError); + }); + + it("never derives paths from identifiers", async () => { + const store = await fileStore(); + const root = (store as unknown as { rootDir: string }).rootDir; + await store.createPin(pin("../../escape", "../../../etc/passwd")); + await store.putTenantSettings({ + tenantId: "../../escape", + previewOptIns: [], + updatedAt: 1, + }); + const files: string[] = []; + const walk = async (dir: string) => { + for (const entry of await fs.readdir(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) { + await walk(full); + } else { + files.push(path.relative(root, full)); + } + } + }; + await walk(root); + expect(files).toHaveLength(2); + for (const file of files) { + expect(file).toMatch( + /^tenants\/[0-9a-f]{64}\/(settings|pins\/[0-9a-f]{64})\.json$/, + ); + } + expect( + (await store.getPin("../../escape", "../../../etc/passwd"))?.tenantId, + ).toBe("../../escape"); + }); + + it("refuses a pin file that does not match its key", async () => { + const store = await fileStore(); + await store.createPin(pin("tenant-a", "scan-1")); + const root = (store as unknown as { rootDir: string }).rootDir; + const tenantDir = (await fs.readdir(path.join(root, "tenants")))[0]; + const pinsDir = path.join(root, "tenants", tenantDir, "pins"); + const [file] = await fs.readdir(pinsDir); + await fs.writeFile( + path.join(pinsDir, file), + JSON.stringify(pin("tenant-b", "scan-1")), + ); + await expect(store.getPin("tenant-a", "scan-1")).rejects.toThrow( + /does not match/, + ); + }); + + it("persists across instances over the same directory", async () => { + const store = await fileStore(); + const root = (store as unknown as { rootDir: string }).rootDir; + await store.putPolicy(storedPolicy("1.0.0", "d1"), null); + await store.createPin(pin("t", "s")); + const reopened = new FileLanguageLifecycleStore(root); + expect((await reopened.getPolicy())?.digest).toBe("d1"); + expect(await reopened.getPin("t", "s")).toEqual(pin("t", "s")); + }); +}); diff --git a/apps/api/src/language-lifecycle/language-lifecycle-store.ts b/apps/api/src/language-lifecycle/language-lifecycle-store.ts new file mode 100644 index 00000000..b7a6a1ea --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-store.ts @@ -0,0 +1,364 @@ +/** + * Persistence boundary for language lifecycle policies, tenant settings, and + * scan pins. + * + * The policy is platform-scoped. Writes are optimistic: `putPolicy` only + * succeeds when the stored digest still equals the one the caller read, so two + * operators publishing at once can never silently overwrite each other. Every + * published policy is also kept in an append-only history for audit. + * + * Tenant settings and pins are tenant-scoped. Every read and write takes the + * tenant id explicitly, and implementations must never resolve a record across + * tenants, so one tenant cannot read or alter another's settings or pins even + * when scan ids collide. Pins are write-once (`createPin`), which makes retries + * idempotent. + * + * Implementations throw plain errors on storage failure; the service maps them + * to the retryable STORE_UNAVAILABLE code. + */ + +import * as crypto from "crypto"; +import { promises as fs } from "fs"; +import * as path from "path"; +import { + LanguageScanPin, + StoredLanguagePolicy, + TenantLanguageSettings, +} from "./language-lifecycle-types"; + +export type PolicyWriteResult = "written" | "conflict"; +export type PinCreateResult = "written" | "exists"; + +export interface LanguageLifecycleStore { + getPolicy(): Promise; + /** + * Make `policy` current when the stored digest equals `expectedDigest` + * (`null` means no policy may be stored yet), and append it to the history. + */ + putPolicy( + policy: StoredLanguagePolicy, + expectedDigest: string | null, + ): Promise; + /** Every published policy, oldest first. */ + listPolicyHistory(): Promise; + getTenantSettings( + tenantId: string, + ): Promise; + putTenantSettings(settings: TenantLanguageSettings): Promise; + getPin( + tenantId: string, + scanId: string, + ): Promise; + /** Store `pin` unless a pin already exists for its tenant and scan. */ + createPin(pin: LanguageScanPin): Promise; + /** Remove pins created before `cutoff`; returns how many were removed. */ + deletePinsBefore(cutoff: number): Promise; +} + +export class InMemoryLanguageLifecycleStore implements LanguageLifecycleStore { + // Stored serialized so callers can never mutate persisted state by reference. + private current: string | undefined; + private readonly history: string[] = []; + private readonly settings: Map = new Map(); + private readonly pins: Map = new Map(); + + async getPolicy(): Promise { + return this.current === undefined ? undefined : JSON.parse(this.current); + } + + async putPolicy( + policy: StoredLanguagePolicy, + expectedDigest: string | null, + ): Promise { + const stored = + this.current === undefined + ? null + : (JSON.parse(this.current) as StoredLanguagePolicy).digest; + if (stored !== expectedDigest) { + return "conflict"; + } + this.current = JSON.stringify(policy); + this.history.push(this.current); + return "written"; + } + + async listPolicyHistory(): Promise { + return this.history.map((raw) => JSON.parse(raw)); + } + + async getTenantSettings( + tenantId: string, + ): Promise { + const raw = this.settings.get(tenantId); + return raw === undefined ? undefined : JSON.parse(raw); + } + + async putTenantSettings(settings: TenantLanguageSettings): Promise { + this.settings.set(settings.tenantId, JSON.stringify(settings)); + } + + async getPin( + tenantId: string, + scanId: string, + ): Promise { + const raw = this.pins.get(pinKey(tenantId, scanId)); + return raw === undefined ? undefined : JSON.parse(raw); + } + + async createPin(pin: LanguageScanPin): Promise { + const key = pinKey(pin.tenantId, pin.scanId); + if (this.pins.has(key)) { + return "exists"; + } + this.pins.set(key, JSON.stringify(pin)); + return "written"; + } + + async deletePinsBefore(cutoff: number): Promise { + let removed = 0; + for (const [key, raw] of this.pins) { + if ((JSON.parse(raw) as LanguageScanPin).pinnedAt < cutoff) { + this.pins.delete(key); + removed++; + } + } + return removed; + } + + pinCount(): number { + return this.pins.size; + } +} + +/** + * Durable store writing JSON files under `rootDir`: + * + * policy/current.json current policy + * policy/history/.json every published policy, zero-padded order + * tenants//settings.json + * tenants//pins/.json + * + * Paths are derived from SHA-256 digests of identifiers, never from the + * identifiers themselves, so crafted ids cannot traverse outside `rootDir`, and + * each tenant's records live in their own directory. Files are written to a + * temporary name and renamed into place, so a crash mid-write leaves either the + * old record or the new one, never a torn file. + * + * Conditional writes are serialized per key within this process. Deployments + * that run several API processes against shared storage should use a database + * store that performs the digest check as a single conditional update. + */ +export class FileLanguageLifecycleStore implements LanguageLifecycleStore { + private readonly rootDir: string; + private readonly locks: Map> = new Map(); + + constructor(rootDir: string) { + if (typeof rootDir !== "string" || rootDir.trim() === "") { + throw new TypeError("rootDir must be a non-empty string"); + } + this.rootDir = path.resolve(rootDir); + } + + getPolicy(): Promise { + return readJson(this.currentPolicyPath()); + } + + putPolicy( + policy: StoredLanguagePolicy, + expectedDigest: string | null, + ): Promise { + const file = this.currentPolicyPath(); + return this.withLock(file, async () => { + const existing = await readJson(file); + if ((existing?.digest ?? null) !== expectedDigest) { + return "conflict"; + } + const historyDir = path.join(this.rootDir, "policy", "history"); + const sequence = (await readDirOrEmpty(historyDir)).filter((n) => + n.endsWith(".json"), + ).length; + // History first: a crash between the writes leaves an extra history + // entry, never a current policy missing from the audit trail. + await writeAtomic( + path.join(historyDir, `${String(sequence).padStart(8, "0")}.json`), + policy, + ); + await writeAtomic(file, policy); + return "written"; + }); + } + + async listPolicyHistory(): Promise { + const dir = path.join(this.rootDir, "policy", "history"); + const policies: StoredLanguagePolicy[] = []; + for (const name of (await readDirOrEmpty(dir)) + .filter((n) => n.endsWith(".json")) + .sort()) { + policies.push( + JSON.parse(await fs.readFile(path.join(dir, name), "utf8")), + ); + } + return policies; + } + + async getTenantSettings( + tenantId: string, + ): Promise { + const settings = await readJson( + this.settingsPath(tenantId), + ); + // Defence in depth: the digest path must hold the record it was asked for. + if (settings && settings.tenantId !== tenantId) { + throw new Error("Stored tenant settings do not match their key"); + } + return settings; + } + + putTenantSettings(settings: TenantLanguageSettings): Promise { + const file = this.settingsPath(settings.tenantId); + return this.withLock(file, () => writeAtomic(file, settings)); + } + + async getPin( + tenantId: string, + scanId: string, + ): Promise { + return this.readPin(this.pinPath(tenantId, scanId), tenantId, scanId); + } + + createPin(pin: LanguageScanPin): Promise { + const file = this.pinPath(pin.tenantId, pin.scanId); + return this.withLock(file, async () => { + if (await this.readPin(file, pin.tenantId, pin.scanId)) { + return "exists"; + } + await writeAtomic(file, pin); + return "written"; + }); + } + + async deletePinsBefore(cutoff: number): Promise { + const tenantsDir = path.join(this.rootDir, "tenants"); + let removed = 0; + for (const tenantDir of (await readDirOrEmpty(tenantsDir)).sort()) { + const dir = path.join(tenantsDir, tenantDir, "pins"); + for (const name of (await readDirOrEmpty(dir)).sort()) { + if (!name.endsWith(".json")) { + continue; + } + const file = path.join(dir, name); + const deleted = await this.withLock(file, async () => { + const pin = await readJson(file); + if (!pin || pin.pinnedAt >= cutoff) { + return false; + } + await fs.rm(file, { force: true }); + return true; + }); + if (deleted) { + removed++; + } + } + } + return removed; + } + + private async readPin( + file: string, + tenantId: string, + scanId: string, + ): Promise { + const pin = await readJson(file); + if (pin && (pin.tenantId !== tenantId || pin.scanId !== scanId)) { + throw new Error("Stored scan pin does not match its key"); + } + return pin; + } + + private currentPolicyPath(): string { + return path.join(this.rootDir, "policy", "current.json"); + } + + private settingsPath(tenantId: string): string { + return path.join( + this.rootDir, + "tenants", + digest(tenantId), + "settings.json", + ); + } + + private pinPath(tenantId: string, scanId: string): string { + return path.join( + this.rootDir, + "tenants", + digest(tenantId), + "pins", + `${digest(scanId)}.json`, + ); + } + + private withLock(key: string, task: () => Promise): Promise { + const previous = this.locks.get(key) ?? Promise.resolve(); + const run = previous.then(task, task); + const settled = run.catch(() => undefined); + this.locks.set(key, settled); + void settled.then(() => { + if (this.locks.get(key) === settled) { + this.locks.delete(key); + } + }); + return run; + } +} + +function pinKey(tenantId: string, scanId: string): string { + return `${tenantId}\u0000${scanId}`; +} + +function digest(value: string): string { + return crypto.createHash("sha256").update(value, "utf8").digest("hex"); +} + +async function writeAtomic(file: string, value: unknown): Promise { + await fs.mkdir(path.dirname(file), { recursive: true }); + const tmp = `${file}.${process.pid}.${crypto.randomBytes(6).toString("hex")}.tmp`; + try { + await fs.writeFile(tmp, JSON.stringify(value), { + encoding: "utf8", + mode: 0o600, + }); + await fs.rename(tmp, file); + } catch (error) { + await fs.rm(tmp, { force: true }).catch(() => undefined); + throw error; + } +} + +async function readJson(file: string): Promise { + let raw: string; + try { + raw = await fs.readFile(file, "utf8"); + } catch (error) { + if (isNotFound(error)) { + return undefined; + } + throw error; + } + return JSON.parse(raw) as T; +} + +async function readDirOrEmpty(dir: string): Promise { + try { + return await fs.readdir(dir); + } catch (error) { + if (isNotFound(error)) { + return []; + } + throw error; + } +} + +function isNotFound(error: unknown): boolean { + return (error as NodeJS.ErrnoException)?.code === "ENOENT"; +} diff --git a/apps/api/src/language-lifecycle/language-lifecycle-types.ts b/apps/api/src/language-lifecycle/language-lifecycle-types.ts new file mode 100644 index 00000000..5b306ee4 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-types.ts @@ -0,0 +1,212 @@ +/** + * Shared types for the supported-language lifecycle policy. + * + * Every language GasGuard can analyze moves through a documented lifecycle: + * + * preview → supported → deprecated → retired + * + * `preview` languages run only for tenants that opted in. `supported` and + * `deprecated` languages run for everyone, deprecated ones with a warning that + * names the sunset date and any replacement. `retired` languages never run; + * their files are reported as skipped, so a retired language can never look + * like a clean result. + * + * The policy is platform-scoped operator data with a version. What a scan + * actually uses is a `LanguageScanPin`, fixed the first time the scan resolves + * its languages, so retries see the same decisions even if the policy, the + * date, or the tenant's opt-ins change in between. + */ + +export const LIFECYCLE_STAGES = [ + "preview", + "supported", + "deprecated", + "retired", +] as const; +export type LifecycleStage = (typeof LIFECYCLE_STAGES)[number]; + +/** One language's entry in the policy. Dates are UTC days, `YYYY-MM-DD`. */ +export interface LanguageLifecycleEntry { + /** Lower-case slug matching the engine's language names, e.g. `solidity`. */ + language: string; + /** Declared stage. Dates below can move it forward without a new policy. */ + stage: LifecycleStage; + /** + * File extensions (with the dot, lower-case) used by `detectLanguages`. May + * be empty for languages that share an extension with another language and + * are detected by the caller instead (e.g. `soroban` vs `rust`). + */ + extensions: string[]; + /** + * Day from which the language counts as deprecated. Required for + * `deprecated`; on a `supported` entry it schedules the deprecation. + */ + deprecatedAt?: string; + /** + * Day from which the language counts as retired. Required whenever + * `deprecatedAt` is set, so every deprecation announces its end. + */ + sunsetAt?: string; + /** Language tenants should move to, when there is one. */ + replacement?: string; +} + +/** Operator input to `publishPolicy`. */ +export interface LanguageLifecyclePolicy { + /** `MAJOR.MINOR.PATCH`; each publish must be strictly greater. */ + version: string; + /** Sorted by `language` once validated. */ + languages: LanguageLifecycleEntry[]; +} + +/** A validated policy as persisted, with bookkeeping. */ +export interface StoredLanguagePolicy extends LanguageLifecyclePolicy { + /** SHA-256 of the canonical policy; part of every scan fingerprint. */ + digest: string; + publishedAt: number; + /** Operator reason, required when transition checks were overridden. */ + reason?: string; + /** Transition violations that `force` overrode, for the audit trail. */ + overriddenViolations?: PolicyViolationCode[]; +} + +/** Tenant-scoped preferences. Never readable by another tenant. */ +export interface TenantLanguageSettings { + tenantId: string; + /** Preview languages this tenant opted into. Sorted and unique. */ + previewOptIns: string[]; + updatedAt: number; +} + +export type LanguageAction = "analyze" | "skip"; + +export type DecisionReason = + | "supported" + | "deprecated" + | "preview_opt_in" + | "preview_not_enabled" + | "retired" + /** The policy is switched off: every language runs as before the policy. */ + | "policy_disabled"; + +/** Effective treatment of one language for one scan. */ +export interface LanguageDecision { + language: string; + /** Stage after applying scheduled dates at `asOf`. */ + stage: LifecycleStage; + action: LanguageAction; + reason: DecisionReason; + sunsetAt?: string; + replacement?: string; +} + +/** What a scan does with languages the policy does not list. */ +export type UnlistedLanguageAction = "analyze" | "skip"; + +/** + * Language decisions fixed for one scan the first time it resolves, so + * retries analyze the same languages. + */ +export interface LanguageScanPin { + tenantId: string; + scanId: string; + /** Policy version and digest in force when the pin was created. */ + policyVersion: string; + policyDigest: string; + /** UTC day the scheduled dates were evaluated at. */ + asOf: string; + /** One per policy language, sorted by `language`. */ + decisions: LanguageDecision[]; + unlistedAction: UnlistedLanguageAction; + /** SHA-256 over everything above except ids; belongs in result cache keys. */ + fingerprint: string; + pinnedAt: number; +} + +export type SkipReason = + | "retired" + | "preview_not_enabled" + | "unlisted_language"; + +export interface SkippedFile { + path: string; + language: string; + reason: SkipReason; +} + +export interface LanguageWarning { + language: string; + /** Number of files in the language that were analyzed anyway. */ + files: number; + sunsetAt?: string; + replacement?: string; + message: string; +} + +export type LanguagePlanStatus = + /** Every source file will be analyzed. */ + | "complete" + /** Some source files were skipped; the rest will be analyzed. */ + | "partial" + /** Source files exist but none can be analyzed. */ + | "unsupported" + /** No file had a recognised language. */ + | "empty"; + +/** Result of applying a pin to a repository's files. Deterministic. */ +export interface LanguageScanPlan { + status: LanguagePlanStatus; + /** Path to language for files that will be analyzed, in path order. */ + analyze: Map; + /** Sorted by path. */ + skipped: SkippedFile[]; + /** One per deprecated language present, sorted by language. */ + warnings: LanguageWarning[]; + fingerprint: string; +} + +export type PolicyViolationCode = + | "VERSION_NOT_INCREASED" + | "LANGUAGE_REMOVED" + | "SKIPPED_DEPRECATION" + | "INSUFFICIENT_NOTICE" + | "SUNSET_PULLED_FORWARD"; + +export interface PolicyViolation { + code: PolicyViolationCode; + language?: string; + message: string; +} + +export type LanguageLifecycleErrorCode = + | "INVALID_INPUT" + | "INVALID_POLICY" + | "TRANSITION_REJECTED" + | "UNSUPPORTED_REPOSITORY" + | "CONFLICT" + | "STORE_UNAVAILABLE"; + +const RETRYABLE_CODES: ReadonlySet = new Set([ + "CONFLICT", + "STORE_UNAVAILABLE", +]); + +export class LanguageLifecycleError extends Error { + readonly code: LanguageLifecycleErrorCode; + /** True when the same call may succeed if retried unchanged. */ + readonly retryable: boolean; + /** Present on INVALID_POLICY and TRANSITION_REJECTED. */ + readonly violations: readonly (PolicyViolation | string)[]; + + constructor( + code: LanguageLifecycleErrorCode, + message: string, + violations: readonly (PolicyViolation | string)[] = [], + ) { + super(message); + this.name = "LanguageLifecycleError"; + this.code = code; + this.retryable = RETRYABLE_CODES.has(code); + this.violations = violations; + } +} diff --git a/apps/api/src/language-lifecycle/language-lifecycle-worker.ts b/apps/api/src/language-lifecycle/language-lifecycle-worker.ts new file mode 100644 index 00000000..4953f19a --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle-worker.ts @@ -0,0 +1,68 @@ +/** + * Worker-side adapter for the language lifecycle policy. + * + * `runScanWithLanguagePolicy` wraps one scan attempt: it resolves (or + * re-reads) the scan's pinned language decisions, plans the repository's + * files, and runs the scan over the files that may be analyzed. The scan + * callback receives the filtered language map, which drops straight into + * `AnalyzerRegistry.analyzeMultiple`. + * + * Failure semantics: + * - STORE_UNAVAILABLE from resolution propagates before any analysis runs and + * is retryable, so the queue's normal retry policy applies. + * - INVALID_INPUT and UNSUPPORTED_REPOSITORY are not retryable: the same + * commit and pin would fail the same way on every attempt. + * - A `partial` plan runs; the skipped files and deprecation warnings travel + * with the result so reports can show them instead of implying a clean scan. + */ + +import { + LanguageLifecycleService, + ScanLanguageResolution, +} from "./language-lifecycle.service"; +import { + LanguageLifecycleError, + LanguageScanPlan, +} from "./language-lifecycle-types"; + +export interface LanguageScanAttempt { + tenantId: string; + scanId: string; + /** Engine path → language map for the repository at the scanned commit. */ + languageMap: ReadonlyMap; + /** UTC day for scheduled dates, e.g. the commit date. Default: today. */ + asOf?: string; +} + +export interface LanguageScopedResult { + result: T; + plan: LanguageScanPlan; + resolution: ScanLanguageResolution; +} + +export async function runScanWithLanguagePolicy( + service: LanguageLifecycleService, + attempt: LanguageScanAttempt, + scan: ( + analyzableLanguageMap: Map, + plan: LanguageScanPlan, + ) => Promise, +): Promise> { + const resolution = await service.resolveForScan({ + tenantId: attempt.tenantId, + scanId: attempt.scanId, + asOf: attempt.asOf, + }); + const plan = service.planScan(resolution.pin, attempt.languageMap); + if (plan.status === "unsupported") { + const languages = [...new Set(plan.skipped.map((f) => f.language))].join( + ", ", + ); + throw new LanguageLifecycleError( + "UNSUPPORTED_REPOSITORY", + `no file in this repository is in an analyzable language (found: ${languages})`, + ); + } + const result = await scan(new Map(plan.analyze), plan); + return { result, plan, resolution }; +} diff --git a/apps/api/src/language-lifecycle/language-lifecycle.integration.spec.ts b/apps/api/src/language-lifecycle/language-lifecycle.integration.spec.ts new file mode 100644 index 00000000..2498c2fc --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle.integration.spec.ts @@ -0,0 +1,438 @@ +/** + * Integration tests across the language lifecycle boundaries: operator-facing + * service, worker adapter, the durable file store (including service restarts + * over the same storage), and the engine's real `AnalyzerRegistry`. + */ + +import { createHash } from "crypto"; +import { promises as fs, readFileSync } from "fs"; +import * as os from "os"; +import * as path from "path"; +import { + AnalysisResult, + Analyzer, + AnalyzerConfig, + Language, + Rule, + Severity, +} from "@engine/core/analyzer-interface"; +import { AnalyzerRegistry } from "@engine/core/analyzer-registry"; +import { LanguageLifecycleConfig } from "./language-lifecycle-config"; +import { + LanguageLifecycleLogFields, + LanguageLifecycleLogLevel, + LanguageLifecycleLogger, +} from "./language-lifecycle-metrics"; +import { detectLanguages, normalizePolicy } from "./language-lifecycle-policy"; +import { FileLanguageLifecycleStore } from "./language-lifecycle-store"; +import { + LanguageLifecycleError, + LanguageLifecyclePolicy, + LanguageScanPlan, + SkippedFile, + LanguageWarning, +} from "./language-lifecycle-types"; +import { LanguageLifecycleService } from "./language-lifecycle.service"; +import { runScanWithLanguagePolicy } from "./language-lifecycle-worker"; + +interface RepositoryFixture { + tenantId: string; + repository: string; + commit: string; + commitDate: string; + scanId: string; + files: string[]; + sorobanPrefixes: string[]; +} + +interface ExpectedFixture { + policyDigest: string; + fingerprint: string; + status: string; + analyze: Record; + skipped: SkippedFile[]; + warnings: LanguageWarning[]; +} + +function fixture(name: string): T { + return JSON.parse( + readFileSync(path.join(__dirname, "__fixtures__", name), "utf8"), + ) as T; +} + +const governed = fixture("governed-policy.json"); +const repo = fixture("soroban-dex-repository.json"); +const expected = fixture("soroban-dex-expected.json"); +const DAY = 24 * 60 * 60 * 1000; + +class CapturingLogger implements LanguageLifecycleLogger { + readonly lines: string[] = []; + log( + level: LanguageLifecycleLogLevel, + event: string, + fields: LanguageLifecycleLogFields, + ): void { + this.lines.push(JSON.stringify({ level, event, ...fields })); + } +} + +/** + * Stand-in analyzer: one finding per file, a pure function of the commit, the + * path, and the analyzer, as real analysis must be. Records what it was given. + */ +class StubAnalyzer implements Analyzer { + readonly received: string[] = []; + constructor(private readonly language: Language) {} + getName() { + return `${this.language}-stub`; + } + getVersion() { + return "1.0.0"; + } + async analyze(code: string, filePath: string): Promise { + return this.analyzeMultiple(new Map([[filePath, code]])); + } + async analyzeMultiple( + files: Map, + _config?: AnalyzerConfig, + ): Promise { + const findings = [...files].map(([file, code]) => { + this.received.push(file); + return { + ruleId: `${this.language}-stub-rule`, + message: createHash("sha256") + .update(`${file}\u0000${code}`) + .digest("hex") + .slice(0, 12), + severity: Severity.LOW, + location: { file, startLine: 1, endLine: 1 }, + }; + }); + return { + findings, + filesAnalyzed: files.size, + analysisTime: 0, + analyzerVersion: "1.0.0", + summary: { + critical: 0, + high: 0, + medium: 0, + low: findings.length, + info: 0, + }, + }; + } + supportsLanguage(language: Language | string) { + return language === this.language; + } + getSupportedLanguages() { + return [this.language]; + } + getRules(): Rule[] { + return []; + } + getRule() { + return undefined; + } + validateConfig() { + return []; + } + async initialize() {} + async dispose() {} +} + +let rootDir: string; +let logger: CapturingLogger; +let now: number; +let registry: AnalyzerRegistry; +let analyzers: Map; + +/** A fresh service over the same directory, i.e. a restarted process. */ +function boot(config: Partial = {}) { + return new LanguageLifecycleService({ + store: new FileLanguageLifecycleStore(rootDir), + logger, + clock: () => now, + config, + }); +} + +/** The worker's view of the repository: path → language at the commit. */ +function repositoryLanguageMap(files = repo.files): Map { + const policy = normalizePolicy(governed, { + maxLanguages: 64, + minDeprecationNoticeDays: 0, + }); + const map = detectLanguages(files, policy); + for (const file of map.keys()) { + if (repo.sorobanPrefixes.some((prefix) => file.startsWith(prefix))) { + map.set(file, Language.SOROBAN); + } + } + return map; +} + +/** Repository content is a pure function of the commit and path. */ +function contents(languageMap: Map): Map { + return new Map( + [...languageMap.keys()].map((file) => [file, `${repo.commit}:${file}`]), + ); +} + +async function scanRepository( + service: LanguageLifecycleService, + scanId = repo.scanId, + tenantId = repo.tenantId, + languageMap = repositoryLanguageMap(), +) { + return runScanWithLanguagePolicy( + service, + { tenantId, scanId, languageMap, asOf: repo.commitDate }, + async (analyzable, plan: LanguageScanPlan) => { + const result = await registry.analyzeMultiple( + contents(analyzable), + analyzable, + ); + return { + findings: result.findings.map( + (f) => `${f.location.file}|${f.ruleId}|${f.message}`, + ), + skipped: plan.skipped, + warnings: plan.warnings, + }; + }, + ); +} + +beforeEach(async () => { + rootDir = await fs.mkdtemp(path.join(os.tmpdir(), "gasguard-langlc-int-")); + logger = new CapturingLogger(); + now = Date.UTC(2026, 9, 18, 9); + registry = new AnalyzerRegistry(); + analyzers = new Map(); + // A javascript and a cairo analyzer stay registered: the policy, not the + // registry, decides they do not run. + for (const language of [ + Language.CAIRO, + Language.JAVASCRIPT, + Language.RUST, + Language.SOLIDITY, + Language.SOROBAN, + Language.TYPESCRIPT, + Language.VYPER, + ]) { + const analyzer = new StubAnalyzer(language); + analyzers.set(language, analyzer); + registry.register(analyzer); + } + await boot().publishPolicy(governed, { + force: true, + reason: "javascript analyzer withdrawn", + }); +}); + +afterEach(async () => { + await fs.rm(rootDir, { recursive: true, force: true }); +}); + +describe("realistic repository: Soroban DEX with EVM and Starknet bridges", () => { + it("analyzes supported and deprecated languages and reports the rest", async () => { + const { result, plan, resolution } = await scanRepository(boot()); + + expect(resolution.pin.policyDigest).toBe(expected.policyDigest); + expect(plan.fingerprint).toBe(expected.fingerprint); + expect(plan.status).toBe(expected.status); + expect(Object.fromEntries(plan.analyze)).toEqual(expected.analyze); + expect(result.skipped).toEqual(expected.skipped); + expect(result.warnings).toEqual(expected.warnings); + + // Only planned files reached the engine, each through its own analyzer. + const received = [...analyzers.values()].flatMap((a) => a.received).sort(); + expect(received).toEqual(Object.keys(expected.analyze).sort()); + expect(analyzers.get(Language.JAVASCRIPT)!.received).toEqual([]); + expect(analyzers.get(Language.CAIRO)!.received).toEqual([]); + expect(result.findings).toHaveLength(Object.keys(expected.analyze).length); + }); + + it("produces identical findings for the same commit, config and policy", async () => { + const first = await scanRepository(boot(), "scan-a"); + now += 5 * DAY; + const second = await scanRepository(boot(), "scan-b"); + expect(second.result).toEqual(first.result); + expect(second.plan.fingerprint).toBe(first.plan.fingerprint); + }); + + it("keeps a retried scan on its pin across restarts, policy changes and the sunset", async () => { + const languageMap = repositoryLanguageMap(); + // First attempt fails after pinning, before the analyzers finish. + await expect( + runScanWithLanguagePolicy( + boot(), + { tenantId: repo.tenantId, scanId: repo.scanId, languageMap }, + async () => { + throw new Error("worker lost its lease"); + }, + ), + ).rejects.toThrow("worker lost its lease"); + + // Meanwhile: the tenant opts into cairo, and vyper's sunset passes. + const operator = boot(); + await operator.setPreviewOptIns(repo.tenantId, ["cairo"]); + now = Date.UTC(2027, 1, 15); + + const retry = await scanRepository(boot()); + expect(retry.resolution.status).toBe("reused"); + expect(retry.plan.analyze.get("bridge/evm/legacy/Vault.vy")).toBe("vyper"); + expect(retry.plan.analyze.has("bridge/starknet/src/bridge.cairo")).toBe( + false, + ); + + // A new scan of the same repository sees today's decisions. + const fresh = await runScanWithLanguagePolicy( + boot(), + { tenantId: repo.tenantId, scanId: "scan-after-sunset", languageMap }, + async (analyzable) => [...analyzable.keys()], + ); + expect(fresh.result).not.toContain("bridge/evm/legacy/Vault.vy"); + expect(fresh.result).toContain("bridge/starknet/src/bridge.cairo"); + expect(fresh.plan.skipped).toContainEqual({ + path: "bridge/evm/legacy/Vault.vy", + language: "vyper", + reason: "retired", + }); + }); +}); + +describe("tenant isolation", () => { + it("applies one tenant's opt-ins only to that tenant, even with colliding scan ids", async () => { + const service = boot(); + await service.setPreviewOptIns("tenant-starknet", ["cairo"]); + const optedIn = await scanRepository(boot(), "scan-1", "tenant-starknet"); + const other = await scanRepository(boot(), "scan-1", repo.tenantId); + expect(optedIn.plan.analyze.has("bridge/starknet/src/bridge.cairo")).toBe( + true, + ); + expect(other.plan.analyze.has("bridge/starknet/src/bridge.cairo")).toBe( + false, + ); + expect(other.resolution.status).toBe("created"); + }); +}); + +describe("failure semantics", () => { + it("fails a repository with no analyzable language as non-retryable, without running analyzers", async () => { + const scan = jest.fn(); + const error = await runScanWithLanguagePolicy( + boot(), + { + tenantId: repo.tenantId, + scanId: "legacy-js", + languageMap: repositoryLanguageMap([ + "scripts/a.js", + "scripts/b.mjs", + "README.md", + ]), + }, + scan, + ).catch((e: LanguageLifecycleError) => e); + expect(error).toMatchObject({ + code: "UNSUPPORTED_REPOSITORY", + retryable: false, + }); + expect((error as Error).message).toContain("javascript"); + expect(scan).not.toHaveBeenCalled(); + }); + + it("runs a repository with no source files, as the registry did before", async () => { + const { plan, result } = await runScanWithLanguagePolicy( + boot(), + { tenantId: repo.tenantId, scanId: "docs-only", languageMap: new Map() }, + async (analyzable) => + (await registry.analyzeMultiple(new Map(), analyzable)).findings, + ); + expect(plan.status).toBe("empty"); + expect(result).toEqual([]); + }); + + it("reports an unreachable store as retryable before any analysis", async () => { + const blocker = path.join(rootDir, "not-a-directory"); + await fs.writeFile(blocker, ""); + const broken = new LanguageLifecycleService({ + store: new FileLanguageLifecycleStore(path.join(blocker, "store")), + logger, + clock: () => now, + }); + const scan = jest.fn(); + const error = await runScanWithLanguagePolicy( + broken, + { + tenantId: repo.tenantId, + scanId: "s", + languageMap: repositoryLanguageMap(), + }, + scan, + ).catch((e: LanguageLifecycleError) => e); + expect(error).toMatchObject({ code: "STORE_UNAVAILABLE", retryable: true }); + expect(scan).not.toHaveBeenCalled(); + expect(logger.lines.join("\n")).not.toContain(rootDir); + }); +}); + +describe("backward compatibility", () => { + it("analyzes every language when the kill switch is off", async () => { + const { plan } = await scanRepository(boot({ enabled: false })); + expect(plan.status).toBe("complete"); + expect(plan.analyze.size).toBe(repositoryLanguageMap().size); + }); + + it("analyzes everything under the built-in default before any policy is published", async () => { + await fs.rm(path.join(rootDir, "policy"), { recursive: true, force: true }); + const { plan, resolution } = await scanRepository(boot(), "pre-policy"); + expect(resolution.pin.policyVersion).toBe("0.0.0"); + expect(plan.status).toBe("complete"); + }); +}); + +describe("analyzer coverage", () => { + it("flags policy languages that no registered analyzer supports", async () => { + const issues = await boot().checkCoverage( + registry.getSupportedLanguages().map(String), + ); + expect(issues).toEqual([ + expect.objectContaining({ kind: "no_analyzer", language: "move" }), + ]); + }); +}); + +describe("observability", () => { + it("never logs file paths, repository names, or content", async () => { + await scanRepository(boot()); + const text = logger.lines.join("\n"); + expect(text).toContain('"event":"scan_planned"'); + for (const forbidden of [ + "contracts/amm", + "Vault.vy", + repo.repository, + repo.commit, + ]) { + expect(text).not.toContain(forbidden); + } + }); + + it("exports metrics with bounded labels", async () => { + const service = boot(); + await scanRepository(service); + const policy = await service.getPolicy(); + const text = service.metrics.toPrometheus(policy, repo.commitDate); + expect(text).toContain( + 'gasguard_language_files_skipped_total{language="cairo",reason="preview_not_enabled"} 1', + ); + expect(text).toContain( + 'gasguard_language_deprecated_files_total{language="vyper"} 1', + ); + expect(text).toContain( + 'gasguard_language_stage{language="javascript",stage="retired"} 1', + ); + expect(text).not.toContain(repo.tenantId); + }); +}); diff --git a/apps/api/src/language-lifecycle/language-lifecycle.service.spec.ts b/apps/api/src/language-lifecycle/language-lifecycle.service.spec.ts new file mode 100644 index 00000000..abaa0c55 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle.service.spec.ts @@ -0,0 +1,459 @@ +import { readFileSync } from "fs"; +import * as path from "path"; +import { LanguageLifecycleConfig } from "./language-lifecycle-config"; +import { + LanguageLifecycleLogFields, + LanguageLifecycleLogLevel, + LanguageLifecycleLogger, +} from "./language-lifecycle-metrics"; +import { InMemoryLanguageLifecycleStore } from "./language-lifecycle-store"; +import { + LanguageLifecycleError, + LanguageLifecyclePolicy, +} from "./language-lifecycle-types"; +import { LanguageLifecycleService } from "./language-lifecycle.service"; + +const TODAY = Date.UTC(2026, 9, 18, 12); +const governed = JSON.parse( + readFileSync( + path.join(__dirname, "__fixtures__", "governed-policy.json"), + "utf8", + ), +) as LanguageLifecyclePolicy; + +class CapturingLogger implements LanguageLifecycleLogger { + readonly entries: { + level: string; + event: string; + fields: LanguageLifecycleLogFields; + }[] = []; + log( + level: LanguageLifecycleLogLevel, + event: string, + fields: LanguageLifecycleLogFields, + ): void { + this.entries.push({ level, event, fields }); + } + text(): string { + return JSON.stringify(this.entries); + } +} + +let store: InMemoryLanguageLifecycleStore; +let logger: CapturingLogger; +let now: number; + +function service(config: Partial = {}) { + return new LanguageLifecycleService({ + store, + logger, + clock: () => now, + config, + }); +} + +async function expectCode(promise: Promise, code: string) { + await expect(promise).rejects.toMatchObject({ + name: "LanguageLifecycleError", + code, + }); +} + +/** Governed policy, but with javascript deprecated rather than retired. */ +function publishable(version = "1.0.0"): LanguageLifecyclePolicy { + return { + version, + languages: governed.languages.map((entry) => + entry.language === "javascript" + ? { + language: "javascript", + stage: "deprecated", + extensions: entry.extensions, + deprecatedAt: "2026-10-18", + sunsetAt: "2027-03-01", + replacement: "typescript", + } + : entry, + ), + }; +} + +beforeEach(() => { + store = new InMemoryLanguageLifecycleStore(); + logger = new CapturingLogger(); + now = TODAY; +}); + +describe("getPolicy", () => { + it("falls back to the built-in default, which supports every language", async () => { + const policy = await service().getPolicy(); + expect(policy.version).toBe("0.0.0"); + expect(policy.publishedAt).toBe(0); + expect(policy.languages.every((l) => l.stage === "supported")).toBe(true); + }); +}); + +describe("publishPolicy", () => { + it("publishes a valid policy and records it in the history", async () => { + const svc = service(); + const published = await svc.publishPolicy(publishable()); + expect(published.result).toBe("published"); + expect(published.policy).toMatchObject({ + version: "1.0.0", + publishedAt: TODAY, + }); + expect(published.policy.digest).toMatch(/^[0-9a-f]{64}$/); + expect((await svc.getPolicy()).digest).toBe(published.policy.digest); + expect(await svc.listPolicyHistory()).toHaveLength(1); + expect(svc.metrics.snapshot().policyPublishes).toEqual({ published: 1 }); + }); + + it("is idempotent for the policy that is already current", async () => { + const svc = service(); + const first = await svc.publishPolicy(publishable()); + now += 1000; + const again = await svc.publishPolicy(publishable()); + expect(again.result).toBe("unchanged"); + expect(again.policy).toEqual(first.policy); + expect(await svc.listPolicyHistory()).toHaveLength(1); + }); + + it("rejects an invalid policy without touching the store", async () => { + await expectCode( + service().publishPolicy({ version: "1.0.0", languages: [] }), + "INVALID_POLICY", + ); + expect(await store.getPolicy()).toBeUndefined(); + }); + + it("rejects retiring a language the default supported, listing the violation", async () => { + // The governed fixture retires javascript, which the default supports. + const error = await service() + .publishPolicy(governed) + .catch((e: LanguageLifecycleError) => e); + expect(error).toBeInstanceOf(LanguageLifecycleError); + expect((error as LanguageLifecycleError).code).toBe("TRANSITION_REJECTED"); + expect((error as LanguageLifecycleError).retryable).toBe(false); + expect((error as LanguageLifecycleError).violations).toEqual([ + expect.objectContaining({ + code: "SKIPPED_DEPRECATION", + language: "javascript", + }), + ]); + }); + + it("allows a forced publish with a reason and keeps an audit record", async () => { + const svc = service(); + const published = await svc.publishPolicy(governed, { + force: true, + reason: " javascript analyzer removed in engine 3.0 ", + }); + expect(published.overridden.map((v) => v.code)).toEqual([ + "SKIPPED_DEPRECATION", + ]); + expect(published.policy.reason).toBe( + "javascript analyzer removed in engine 3.0", + ); + expect(published.policy.overriddenViolations).toEqual([ + "SKIPPED_DEPRECATION", + ]); + expect(svc.metrics.snapshot().policyPublishes).toEqual({ forced: 1 }); + expect( + logger.entries.find((e) => e.event === "policy_published")?.level, + ).toBe("warn"); + }); + + it("requires a reason to force", async () => { + await expectCode( + service().publishPolicy(governed, { force: true }), + "INVALID_INPUT", + ); + await expectCode( + service().publishPolicy(governed, { force: true, reason: " " }), + "INVALID_INPUT", + ); + }); + + it("never lets force bypass a non-increasing version", async () => { + const svc = service(); + await svc.publishPolicy(publishable("1.0.0")); + const changed = publishable("1.0.0"); + changed.languages = changed.languages.filter((l) => l.language !== "move"); + await expectCode( + svc.publishPolicy(changed, { force: true, reason: "r" }), + "TRANSITION_REJECTED", + ); + }); + + it("reports a concurrent publish as a retryable CONFLICT", async () => { + const svc = service(); + await svc.publishPolicy(publishable("1.0.0")); + const real = store.putPolicy.bind(store); + jest.spyOn(store, "putPolicy").mockImplementationOnce(async (policy) => { + // Another operator publishes between our read and our write. + const current = await store.getPolicy(); + await real( + { ...current!, version: "1.0.5", digest: "other" }, + current!.digest, + ); + return real(policy, current!.digest); + }); + const error = await svc + .publishPolicy(publishable("1.1.0")) + .catch((e: LanguageLifecycleError) => e); + expect(error).toMatchObject({ code: "CONFLICT", retryable: true }); + }); +}); + +describe("tenant settings", () => { + it("stores sorted, unique opt-ins per tenant", async () => { + const svc = service(); + const settings = await svc.setPreviewOptIns("tenant-a", [ + "move", + "cairo", + "move", + ]); + expect(settings).toEqual({ + tenantId: "tenant-a", + previewOptIns: ["cairo", "move"], + updatedAt: TODAY, + }); + expect((await svc.getTenantSettings("tenant-b")).previewOptIns).toEqual([]); + }); + + it.each([ + ["an empty tenant", "", ["cairo"]], + ["a non-array", "tenant-a", "cairo" as unknown as string[]], + ["a bad slug", "tenant-a", ["Cairo"]], + ])("rejects %s", async (_name, tenantId, languages) => { + await expectCode( + service().setPreviewOptIns(tenantId, languages), + "INVALID_INPUT", + ); + }); + + it("rejects more opt-ins than maxLanguages", async () => { + await expectCode( + service({ maxLanguages: 1 }).setPreviewOptIns("t", ["a", "b"]), + "INVALID_INPUT", + ); + }); + + it("describes the policy for one tenant only", async () => { + const svc = service(); + await svc.publishPolicy(governed, { force: true, reason: "test" }); + await svc.setPreviewOptIns("tenant-a", ["cairo"]); + const a = await svc.describeForTenant("tenant-a"); + const b = await svc.describeForTenant("tenant-b"); + const cairo = (view: typeof a) => + view.languages.find((l) => l.language === "cairo"); + expect(cairo(a)).toMatchObject({ + action: "analyze", + reason: "preview_opt_in", + }); + expect(cairo(b)).toMatchObject({ + action: "skip", + reason: "preview_not_enabled", + }); + expect(JSON.stringify(b)).not.toContain("tenant-a"); + }); +}); + +describe("resolveForScan", () => { + it("creates a pin once and reuses it on retry", async () => { + const svc = service(); + const first = await svc.resolveForScan({ tenantId: "t", scanId: "s" }); + expect(first.status).toBe("created"); + expect(first.pin.asOf).toBe("2026-10-18"); + const retry = await svc.resolveForScan({ tenantId: "t", scanId: "s" }); + expect(retry.status).toBe("reused"); + expect(retry.pin).toEqual(first.pin); + expect(svc.metrics.snapshot()).toMatchObject({ + pinsCreated: 1, + pinsReused: 1, + }); + }); + + it("keeps a pinned scan's decisions when the policy, date and opt-ins change", async () => { + const svc = service(); + const first = await svc.resolveForScan({ tenantId: "t", scanId: "s" }); + await svc.publishPolicy(governed, { force: true, reason: "test" }); + await svc.setPreviewOptIns("t", ["cairo"]); + now += 400 * 24 * 60 * 60 * 1000; + const retry = await svc.resolveForScan({ + tenantId: "t", + scanId: "s", + asOf: "2030-01-01", + }); + expect(retry.pin).toEqual(first.pin); + }); + + it("gives the same fingerprint for the same commit date, config and policy", async () => { + const svc = service(); + await svc.publishPolicy(governed, { force: true, reason: "test" }); + const a = await svc.resolveForScan({ + tenantId: "t", + scanId: "s1", + asOf: "2026-10-15", + }); + now += 3 * 24 * 60 * 60 * 1000; + const b = await svc.resolveForScan({ + tenantId: "t", + scanId: "s2", + asOf: "2026-10-15", + }); + expect(b.pin.fingerprint).toBe(a.pin.fingerprint); + }); + + it("isolates pins between tenants with colliding scan ids", async () => { + const svc = service(); + await svc.publishPolicy(governed, { force: true, reason: "test" }); + await svc.setPreviewOptIns("tenant-a", ["cairo"]); + const a = await svc.resolveForScan({ + tenantId: "tenant-a", + scanId: "scan-1", + }); + const b = await svc.resolveForScan({ + tenantId: "tenant-b", + scanId: "scan-1", + }); + expect(b.status).toBe("created"); + expect(b.pin.fingerprint).not.toBe(a.pin.fingerprint); + expect(b.pin.tenantId).toBe("tenant-b"); + }); + + it("uses the winner's pin when a concurrent attempt pins first", async () => { + const svc = service(); + const winner = await svc.resolveForScan({ tenantId: "t", scanId: "race" }); + jest.spyOn(store, "getPin").mockResolvedValueOnce(undefined); + const loser = await svc.resolveForScan({ tenantId: "t", scanId: "race" }); + expect(loser).toEqual({ pin: winner.pin, status: "reused" }); + }); + + it.each([ + [{ tenantId: "", scanId: "s" }], + [{ tenantId: "t", scanId: "x".repeat(201) }], + [{ tenantId: "t", scanId: "bad\nid" }], + [{ tenantId: "t", scanId: "s", asOf: "2026-13-01" }], + [undefined as unknown as { tenantId: string; scanId: string }], + ])("rejects %j as INVALID_INPUT", async (request) => { + await expectCode(service().resolveForScan(request), "INVALID_INPUT"); + }); + + it("maps store failures to retryable STORE_UNAVAILABLE without leaking details", async () => { + const svc = service(); + jest + .spyOn(store, "getPin") + .mockRejectedValueOnce( + new Error("ECONNREFUSED postgres://admin:hunter2@db"), + ); + const error = await svc + .resolveForScan({ tenantId: "t", scanId: "s" }) + .catch((e: LanguageLifecycleError) => e); + expect(error).toMatchObject({ code: "STORE_UNAVAILABLE", retryable: true }); + expect((error as Error).message).not.toContain("hunter2"); + expect(logger.text()).not.toContain("hunter2"); + expect(svc.metrics.snapshot().errors).toEqual({ STORE_UNAVAILABLE: 1 }); + expect(await store.getPin("t", "s")).toBeUndefined(); + }); + + describe("kill switch", () => { + it("analyzes every language for new scans when disabled", async () => { + await service().publishPolicy(governed, { force: true, reason: "test" }); + const disabled = service({ + enabled: false, + unlistedLanguageAction: "skip", + }); + const { pin } = await disabled.resolveForScan({ + tenantId: "t", + scanId: "s", + }); + expect(pin.decisions.every((d) => d.action === "analyze")).toBe(true); + expect(pin.unlistedAction).toBe("analyze"); + }); + + it("still honours pins created while enabled", async () => { + await service().publishPolicy(governed, { force: true, reason: "test" }); + const enabledPin = ( + await service().resolveForScan({ tenantId: "t", scanId: "s" }) + ).pin; + const retry = await service({ enabled: false }).resolveForScan({ + tenantId: "t", + scanId: "s", + }); + expect(retry.pin).toEqual(enabledPin); + }); + }); +}); + +describe("planScan", () => { + it("logs counts and codes but never file paths", async () => { + const svc = service(); + await svc.publishPolicy(governed, { force: true, reason: "test" }); + const { pin } = await svc.resolveForScan({ tenantId: "t", scanId: "s" }); + const plan = svc.planScan( + pin, + new Map([ + ["private/customer-secret/Vault.vy", "vyper"], + ["private/customer-secret/old.js", "javascript"], + ]), + ); + expect(plan.status).toBe("partial"); + const entry = logger.entries.find((e) => e.event === "scan_planned")!; + expect(entry.level).toBe("warn"); + expect(entry.fields).toMatchObject({ + status: "partial", + analyzedCount: 1, + skippedCount: 1, + deprecatedLanguages: "vyper", + }); + expect(logger.text()).not.toContain("customer-secret"); + expect(svc.metrics.snapshot().filesSkipped).toEqual({ + "javascript:retired": 1, + }); + }); + + it("counts invalid input as an error", async () => { + const svc = service({ maxFilesPerScan: 1 }); + const { pin } = await svc.resolveForScan({ tenantId: "t", scanId: "s" }); + expect(() => + svc.planScan( + pin, + new Map([ + ["a.sol", "solidity"], + ["b.sol", "solidity"], + ]), + ), + ).toThrow(LanguageLifecycleError); + expect(svc.metrics.snapshot().errors).toEqual({ INVALID_INPUT: 1 }); + }); +}); + +describe("checkCoverage", () => { + it("compares the current policy with analyzer languages", async () => { + const svc = service(); + await svc.publishPolicy(governed, { force: true, reason: "test" }); + const issues = await svc.checkCoverage([ + "solidity", + "rust", + "soroban", + "typescript", + ]); + expect(issues.map((i) => `${i.kind}:${i.language}`)).toEqual([ + "no_analyzer:cairo", + "no_analyzer:move", + "no_analyzer:vyper", + ]); + }); +}); + +describe("purgeExpiredPins", () => { + it("removes pins older than the retention period", async () => { + const svc = service({ pinRetentionMs: 1000 }); + await svc.resolveForScan({ tenantId: "t", scanId: "old" }); + now += 2000; + await svc.resolveForScan({ tenantId: "t", scanId: "new" }); + expect(await svc.purgeExpiredPins()).toBe(1); + expect(store.pinCount()).toBe(1); + }); +}); diff --git a/apps/api/src/language-lifecycle/language-lifecycle.service.ts b/apps/api/src/language-lifecycle/language-lifecycle.service.ts new file mode 100644 index 00000000..87dd3060 --- /dev/null +++ b/apps/api/src/language-lifecycle/language-lifecycle.service.ts @@ -0,0 +1,526 @@ +/** + * Supported-language lifecycle policy. + * + * Operators publish a versioned policy that places each language in a + * lifecycle stage (preview, supported, deprecated, retired), optionally with + * scheduled deprecation and sunset dates. Tenants may opt into preview + * languages. Workers call `resolveForScan` before analysing a repository and + * `planScan` to split its files into what will be analyzed and what is skipped + * and why. + * + * Determinism: a scan's language decisions are pinned the first time it + * resolves, including the day the scheduled dates were evaluated at, so a + * retry always analyzes the same languages. Pass `asOf` (for example the + * commit date) to make re-runs of the same commit reproduce exactly; the + * pin's `fingerprint` belongs in any cache key for scan results. + * + * Backward compatibility: until a policy is published the built-in default + * marks every engine language as supported, and languages a policy does not + * list are analyzed unless `unlistedLanguageAction` is `skip`. Disabling the + * policy (`LANGUAGE_LIFECYCLE_ENABLED=false`) analyzes everything again for + * new scans while honouring existing pins. + */ + +import { + DEFAULT_LANGUAGE_LIFECYCLE_CONFIG, + LanguageLifecycleConfig, + resolveLanguageLifecycleConfig, +} from "./language-lifecycle-config"; +import { + LanguageLifecycleLogger, + LanguageLifecycleMetrics, + consoleLanguageLifecycleLogger, + sanitizeLogFields, +} from "./language-lifecycle-metrics"; +import { + CoverageIssue, + DEFAULT_LANGUAGE_POLICY, + buildPin, + checkPolicyCoverage, + checkPolicyTransition, + compareStrings, + evaluateDecisions, + isValidDay, + normalizePolicy, + planScan, + policyDigest, + requireIdentifier, + requireLanguage, + utcDay, +} from "./language-lifecycle-policy"; +import { LanguageLifecycleStore } from "./language-lifecycle-store"; +import { + LanguageDecision, + LanguageLifecycleError, + LanguageScanPin, + LanguageScanPlan, + PolicyViolation, + StoredLanguagePolicy, + TenantLanguageSettings, +} from "./language-lifecycle-types"; + +const MAX_REASON_LENGTH = 500; + +export interface LanguageLifecycleServiceOptions { + store: LanguageLifecycleStore; + config?: Partial; + logger?: LanguageLifecycleLogger; + metrics?: LanguageLifecycleMetrics; + clock?: () => number; +} + +export interface PublishOptions { + /** + * Publish despite transition violations other than VERSION_NOT_INCREASED. + * Requires `reason`, which is kept in the policy history. + */ + force?: boolean; + reason?: string; +} + +export interface PublishResult { + policy: StoredLanguagePolicy; + /** `unchanged` when the same policy was already current (idempotent). */ + result: "published" | "unchanged"; + /** Violations that `force` overrode; empty for a clean publish. */ + overridden: PolicyViolation[]; +} + +export interface ScanLanguageRequest { + tenantId: string; + scanId: string; + /** + * UTC day to evaluate scheduled dates at, `YYYY-MM-DD`. Defaults to today. + * Ignored when the scan is already pinned. + */ + asOf?: string; +} + +export interface ScanLanguageResolution { + pin: LanguageScanPin; + /** `reused` when the scan was already pinned (a retry). */ + status: "created" | "reused"; +} + +/** Tenant-facing view of the policy. Never names other tenants. */ +export interface TenantLanguageView { + policyVersion: string; + asOf: string; + languages: LanguageDecision[]; + previewOptIns: string[]; +} + +export class LanguageLifecycleService { + readonly config: LanguageLifecycleConfig; + readonly metrics: LanguageLifecycleMetrics; + private readonly store: LanguageLifecycleStore; + private readonly logger: LanguageLifecycleLogger; + private readonly clock: () => number; + + constructor(options: LanguageLifecycleServiceOptions) { + this.config = resolveLanguageLifecycleConfig( + options.config ?? DEFAULT_LANGUAGE_LIFECYCLE_CONFIG, + ); + this.store = options.store; + // Sanitize here as well as in the default logger, so a custom logger can + // never receive more than identifiers, versions, counts, and codes. + const logger = options.logger ?? consoleLanguageLifecycleLogger; + this.logger = { + log: (level, event, fields) => + logger.log(level, event, sanitizeLogFields(fields)), + }; + this.metrics = options.metrics ?? new LanguageLifecycleMetrics(); + this.clock = options.clock ?? Date.now; + } + + /** + * The current policy, or the built-in default when none was published. + * + * @throws LanguageLifecycleError STORE_UNAVAILABLE + */ + getPolicy(): Promise { + return this.guard("get_policy", () => this.currentPolicy()); + } + + /** + * Validate and publish a new policy. Publishing the policy that is already + * current is a no-op, so retrying a publish after a timeout is safe. + * + * @throws LanguageLifecycleError INVALID_INPUT, INVALID_POLICY, + * TRANSITION_REJECTED, CONFLICT or STORE_UNAVAILABLE + */ + publishPolicy( + input: unknown, + options: PublishOptions = {}, + ): Promise { + return this.guard("publish_policy", async () => { + const reason = this.optionalReason(options.reason); + if (options.force && reason === undefined) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "a forced publish requires a reason", + ); + } + const next = normalizePolicy(input, this.config); + const digest = policyDigest(next); + const stored = await this.store.getPolicy(); + const current = stored ?? defaultStoredPolicy(); + if (stored && stored.digest === digest) { + this.metrics.recordPublish("unchanged"); + return { policy: stored, result: "unchanged", overridden: [] }; + } + + const asOf = utcDay(this.clock()); + const violations = checkPolicyTransition( + current, + next, + asOf, + this.config.minDeprecationNoticeDays, + ); + for (const violation of violations) { + this.metrics.recordViolation(violation.code); + } + const unforceable = violations.filter( + (v) => v.code === "VERSION_NOT_INCREASED", + ); + if (violations.length > 0 && (!options.force || unforceable.length > 0)) { + throw new LanguageLifecycleError( + "TRANSITION_REJECTED", + `policy ${next.version} was rejected: ${violations + .map((v) => v.message) + .join("; ")}`, + violations, + ); + } + + const policy: StoredLanguagePolicy = { + ...next, + digest, + publishedAt: this.clock(), + }; + if (reason !== undefined) { + policy.reason = reason; + } + if (violations.length > 0) { + policy.overriddenViolations = violations.map((v) => v.code); + } + const write = await this.store.putPolicy(policy, stored?.digest ?? null); + if (write === "conflict") { + throw new LanguageLifecycleError( + "CONFLICT", + "the policy changed while publishing; re-read it and try again", + ); + } + const kind = violations.length > 0 ? "forced" : "published"; + this.metrics.recordPublish(kind); + this.logger.log( + violations.length > 0 ? "warn" : "info", + "policy_published", + { + policyVersion: policy.version, + previousVersion: current.version, + policyDigest: digest, + languageCount: policy.languages.length, + forced: violations.length > 0, + overridden: violations.map((v) => v.code).join(","), + }, + ); + return { policy, result: "published", overridden: violations }; + }); + } + + /** Every published policy, oldest first. */ + listPolicyHistory(): Promise { + return this.guard("list_policy_history", () => + this.store.listPolicyHistory(), + ); + } + + /** @throws LanguageLifecycleError INVALID_INPUT or STORE_UNAVAILABLE */ + getTenantSettings(tenantId: string): Promise { + return this.guard("get_tenant_settings", async () => { + requireIdentifier(tenantId, "tenantId"); + return this.tenantSettings(tenantId); + }); + } + + /** + * Replace the tenant's preview opt-ins. A language may be opted into before + * it is listed as preview; the opt-in only has an effect while it is. + * + * @throws LanguageLifecycleError INVALID_INPUT or STORE_UNAVAILABLE + */ + setPreviewOptIns( + tenantId: string, + languages: string[], + ): Promise { + return this.guard("set_preview_opt_ins", async () => { + requireIdentifier(tenantId, "tenantId"); + if (!Array.isArray(languages)) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "languages must be an array", + ); + } + languages.forEach(requireLanguage); + const previewOptIns = [...new Set(languages)].sort(compareStrings); + if (previewOptIns.length > this.config.maxLanguages) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + `at most ${this.config.maxLanguages} preview opt-ins are allowed`, + ); + } + const settings: TenantLanguageSettings = { + tenantId, + previewOptIns, + updatedAt: this.clock(), + }; + await this.store.putTenantSettings(settings); + this.logger.log("info", "preview_opt_ins_updated", { + tenantId, + optInCount: previewOptIns.length, + }); + return settings; + }); + } + + /** + * What the policy means for one tenant on `asOf` (default today), without + * pinning anything. Safe to return to that tenant. + */ + describeForTenant( + tenantId: string, + asOf?: string, + ): Promise { + return this.guard("describe_for_tenant", async () => { + requireIdentifier(tenantId, "tenantId"); + const day = this.asOfOrToday(asOf); + const policy = await this.currentPolicy(); + const settings = await this.tenantSettings(tenantId); + return { + policyVersion: policy.version, + asOf: day, + languages: evaluateDecisions( + policy, + day, + settings.previewOptIns, + this.config.enabled, + ), + previewOptIns: settings.previewOptIns, + }; + }); + } + + /** + * Pin (or re-read) the language decisions for one scan. + * + * @throws LanguageLifecycleError INVALID_INPUT or STORE_UNAVAILABLE + * (retryable; nothing is pinned) + */ + resolveForScan( + request: ScanLanguageRequest, + ): Promise { + return this.guard("resolve_for_scan", async () => { + requireIdentifier(request?.tenantId, "tenantId"); + requireIdentifier(request.scanId, "scanId"); + const { tenantId, scanId } = request; + const asOf = this.asOfOrToday(request.asOf); + + const existing = await this.store.getPin(tenantId, scanId); + if (existing) { + return this.reused(existing); + } + + const policy = await this.currentPolicy(); + const settings = await this.tenantSettings(tenantId); + const pin = buildPin({ + tenantId, + scanId, + policy, + policyDigest: policy.digest, + asOf, + previewOptIns: settings.previewOptIns, + enabled: this.config.enabled, + unlistedAction: this.config.unlistedLanguageAction, + pinnedAt: this.clock(), + }); + if ((await this.store.createPin(pin)) === "exists") { + // A concurrent attempt of the same scan pinned first; use its pin. + const winner = await this.store.getPin(tenantId, scanId); + if (winner) { + return this.reused(winner); + } + throw new Error("pin vanished after a create conflict"); + } + this.metrics.recordPin("created"); + this.logger.log("info", "scan_pinned", { + tenantId, + scanId, + policyVersion: pin.policyVersion, + asOf: pin.asOf, + fingerprint: pin.fingerprint, + enabled: this.config.enabled, + }); + return { pin, status: "created" }; + }); + } + + /** + * Classify a repository's files under `pin`. Pure apart from metrics and + * logging: the same pin and language map always give the same plan. + * + * @throws LanguageLifecycleError INVALID_INPUT + */ + planScan( + pin: LanguageScanPin, + languageMap: ReadonlyMap, + ): LanguageScanPlan { + let plan: LanguageScanPlan; + try { + plan = planScan(pin, languageMap, this.config.maxFilesPerScan); + } catch (error) { + if (error instanceof LanguageLifecycleError) { + this.metrics.recordError(error.code); + this.logger.log("warn", "plan_scan_failed", { + tenantId: pin?.tenantId, + scanId: pin?.scanId, + code: error.code, + }); + } + throw error; + } + this.metrics.recordPlan(plan); + this.logger.log( + plan.status === "complete" ? "info" : "warn", + "scan_planned", + { + tenantId: pin.tenantId, + scanId: pin.scanId, + status: plan.status, + analyzedCount: plan.analyze.size, + skippedCount: plan.skipped.length, + deprecatedLanguages: plan.warnings.map((w) => w.language).join(","), + fingerprint: plan.fingerprint, + }, + ); + return plan; + } + + /** + * Compare the current policy with the languages analyzers claim, e.g. + * `AnalyzerRegistry.getSupportedLanguages()`. + */ + checkCoverage( + analyzerLanguages: readonly string[], + asOf?: string, + ): Promise { + return this.guard("check_coverage", async () => + checkPolicyCoverage( + await this.currentPolicy(), + analyzerLanguages, + this.asOfOrToday(asOf), + ), + ); + } + + /** Remove pins older than `pinRetentionMs`; returns how many were removed. */ + purgeExpiredPins(): Promise { + return this.guard("purge_pins", async () => { + const removed = await this.store.deletePinsBefore( + this.clock() - this.config.pinRetentionMs, + ); + this.logger.log("info", "pins_purged", { removed }); + return removed; + }); + } + + private reused(pin: LanguageScanPin): ScanLanguageResolution { + this.metrics.recordPin("reused"); + this.logger.log("info", "scan_pin_reused", { + tenantId: pin.tenantId, + scanId: pin.scanId, + policyVersion: pin.policyVersion, + fingerprint: pin.fingerprint, + }); + return { pin, status: "reused" }; + } + + private async currentPolicy(): Promise { + return (await this.store.getPolicy()) ?? defaultStoredPolicy(); + } + + private async tenantSettings( + tenantId: string, + ): Promise { + return ( + (await this.store.getTenantSettings(tenantId)) ?? { + tenantId, + previewOptIns: [], + updatedAt: 0, + } + ); + } + + private asOfOrToday(asOf: string | undefined): string { + if (asOf === undefined) { + return utcDay(this.clock()); + } + if (!isValidDay(asOf)) { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "asOf must be a YYYY-MM-DD date", + ); + } + return asOf; + } + + private optionalReason(reason: unknown): string | undefined { + if (reason === undefined) { + return undefined; + } + if (typeof reason !== "string" || reason.trim() === "") { + throw new LanguageLifecycleError( + "INVALID_INPUT", + "reason must be a non-empty string", + ); + } + return reason.trim().slice(0, MAX_REASON_LENGTH); + } + + /** + * Run `operation`, mapping unexpected (storage) errors to the retryable + * STORE_UNAVAILABLE code. Store error messages are never logged or + * returned, since they may contain paths or connection details. + */ + private async guard( + operation: string, + task: () => Promise, + ): Promise { + try { + return await task(); + } catch (error) { + const mapped = + error instanceof LanguageLifecycleError + ? error + : new LanguageLifecycleError( + "STORE_UNAVAILABLE", + "the language lifecycle store is unavailable; retry later", + ); + this.metrics.recordError(mapped.code); + this.logger.log(mapped.retryable ? "error" : "warn", "operation_failed", { + operation, + code: mapped.code, + retryable: mapped.retryable, + }); + throw mapped; + } + } +} + +function defaultStoredPolicy(): StoredLanguagePolicy { + const policy = normalizePolicy(DEFAULT_LANGUAGE_POLICY, { + maxLanguages: Number.MAX_SAFE_INTEGER, + minDeprecationNoticeDays: 0, + }); + return { ...policy, digest: policyDigest(policy), publishedAt: 0 }; +} diff --git a/docs/LANGUAGE_LIFECYCLE_POLICY.md b/docs/LANGUAGE_LIFECYCLE_POLICY.md new file mode 100644 index 00000000..fb9bff40 --- /dev/null +++ b/docs/LANGUAGE_LIFECYCLE_POLICY.md @@ -0,0 +1,306 @@ +# Supported-Language Lifecycle Policy + +Part of the **Release and Governance** workstream. + +GasGuard analyzes several smart-contract and tooling languages. Today a language is either handled by +an analyzer or silently produces zero findings. That second outcome looks the same as a clean +result. There is also no way to announce that a language is going away, to try a new one with +selected tenants, or to stop analyzing one without breaking someone's scans without warning. The +language lifecycle policy fixes all of this. Each language gets a documented stage with scheduled +dates. Scans pin the decisions they used, and anything a scan does not analyze is reported with a +reason. + +| Piece | Location | +| --- | --- | +| Types and error codes | `apps/api/src/language-lifecycle/language-lifecycle-types.ts` | +| Configuration (`LANGUAGE_LIFECYCLE_*`) | `apps/api/src/language-lifecycle/language-lifecycle-config.ts` | +| Pure policy logic: validation, stages, governance, detection, planning | `apps/api/src/language-lifecycle/language-lifecycle-policy.ts` | +| Persistence boundary (in-memory and file stores) | `apps/api/src/language-lifecycle/language-lifecycle-store.ts` | +| Operator, tenant and worker API | `apps/api/src/language-lifecycle/language-lifecycle.service.ts` | +| Worker adapter | `apps/api/src/language-lifecycle/language-lifecycle-worker.ts` | +| Metrics and structured logs | `apps/api/src/language-lifecycle/language-lifecycle-metrics.ts` | +| Fixtures (realistic repository, sunset-boundary regression) | `apps/api/src/language-lifecycle/__fixtures__/` | + +The module depends only on the engine's `Language` enum. Its output is the same path-to-language +map that `AnalyzerRegistry.analyzeMultiple` already takes, so the registry and analyzers are +unchanged. Existing scan paths behave exactly as before until a worker opts in through +`runScanWithLanguagePolicy`. + +## Lifecycle + +``` +preview ──► supported ──► deprecated ──► retired + │ ▲ + └───────────────────────────────────────┘ (preview may retire directly) +``` + +| Stage | Analyzed? | Reported as | +| --- | --- | --- | +| `preview` | Only for tenants that opted in | Skipped (`preview_not_enabled`) for everyone else | +| `supported` | Yes | — | +| `deprecated` | Yes | One warning per language with file count, `sunsetAt`, and `replacement` | +| `retired` | No | Skipped (`retired`), with `replacement` in the decision | + +A policy entry declares a stage and may schedule the next steps by date (UTC days, `YYYY-MM-DD`): + +```json +{ + "version": "1.2.0", + "languages": [ + { "language": "cairo", "stage": "preview", "extensions": [".cairo"] }, + { "language": "solidity", "stage": "supported", "extensions": [".sol"] }, + { "language": "soroban", "stage": "supported", "extensions": [] }, + { + "language": "vyper", "stage": "supported", "extensions": [".vy"], + "deprecatedAt": "2026-11-01", "sunsetAt": "2027-02-01", "replacement": "solidity" + } + ] +} +``` + +The effective stage on day `asOf` is computed as follows: + +- `preview` and `retired` never move on their own. +- From `deprecatedAt` (inclusive), the language is `deprecated`. +- From `sunsetAt` (inclusive), the language is `retired`. + +So an operator can publish a whole deprecation schedule once, and no change is needed on the day +itself. + +### Validation rules (`INVALID_POLICY`) + +- `version` is `MAJOR.MINOR.PATCH`, and there is at least one language and at most `maxLanguages`. +- `language` is a lower-case slug (`^[a-z][a-z0-9_-]{0,31}$`), listed once. Use the engine's names + (`Language` enum) so the registry agrees. +- `extensions` are lower-case with a leading dot, up to three segments (`.d.ts`), and at most 16 + characters. An extension belongs to one language only. A language may list none; for example, + `soroban` shares `.rs` with `rust`, and the caller tells them apart. +- `deprecated` needs both `deprecatedAt` and `sunsetAt`. A scheduled deprecation on a `supported` + entry also needs both. `preview` carries no dates. +- `sunsetAt − deprecatedAt ≥ minDeprecationNoticeDays`. +- `replacement` must be another listed language that is not retired. + +Every problem is reported at once in `error.violations`. Canonicalization drops unknown keys and +sorts languages and extensions, so two policies that mean the same thing have the same `digest`. + +### Governance between versions (`TRANSITION_REJECTED`) + +When a policy is published, it is compared with the current one (or with the built-in default if +none has been published yet), evaluated on the publish day: + +| Code | Rule | Forceable | +| --- | --- | --- | +| `VERSION_NOT_INCREASED` | The new version must be strictly greater | No | +| `LANGUAGE_REMOVED` | Languages are retired, never deleted, so their files keep being reported | Yes | +| `SKIPPED_DEPRECATION` | `supported` cannot become `retired` without a deprecation period (`preview` can) | Yes | +| `INSUFFICIENT_NOTICE` | A newly announced `sunsetAt` must be at least `minDeprecationNoticeDays` after the publish day; this blocks back-dating `deprecatedAt` | Yes | +| `SUNSET_PULLED_FORWARD` | An announced sunset may move later, never earlier | Yes | + +`publishPolicy(policy, { force: true, reason })` overrides forceable violations. The reason and the +overridden codes are stored in the policy history, the publish is logged at `warn`, and it is +counted as `forced`. Reinstating a deprecated or retired language is always allowed. + +## Determinism + +Scan results must be identical for the same commit, configuration, and rule-pack version. The +language decisions for a scan are a pure function of: + +- the policy's canonical `digest` and `version`, +- the `asOf` day the dates are evaluated at, +- the tenant's preview opt-ins, +- `enabled` and `unlistedLanguageAction`. + +The first `resolveForScan` writes these decisions to a write-once **pin**, keyed by +`(tenantId, scanId)`. Retries read the pin back. Publishing a new policy, a sunset passing, or the +tenant changing its opt-ins does not change a pinned scan. `pin.fingerprint` is a SHA-256 over +everything above except the ids. Include it in result cache keys, next to the rule-pack fingerprint. + +Two notes for reproducible re-runs: + +- Pass `asOf` (for example, the commit date) so that two different scans of the same commit agree + even if they run on different days. If you omit it, `asOf` is the UTC day the scan was first + pinned. +- `planScan` sorts by code point (not locale), so the same pin and language map always give an + identical plan regardless of input order. + +The regression fixture `sunset-boundary-regression.json` freezes the decisions and fingerprints on +the day before, on, and after each boundary. A diff there means existing results would no longer +reproduce. + +## Interface + +### Operator + +```ts +const service = new LanguageLifecycleService({ + store: new FileLanguageLifecycleStore("/var/lib/gasguard/language-lifecycle"), + config: loadLanguageLifecycleConfigFromEnv(), +}); + +await service.publishPolicy(policy); // INVALID_POLICY | TRANSITION_REJECTED | CONFLICT +await service.publishPolicy(policy, { force: true, reason }); // audited override +await service.getPolicy(); // current, or built-in default (version 0.0.0) +await service.listPolicyHistory(); // every published policy, oldest first +await service.checkCoverage(registry.getSupportedLanguages()); // policy vs analyzers +``` + +`publishPolicy` is idempotent: publishing the policy that is already current returns +`{ result: "unchanged" }` and writes nothing, so a timed-out publish can be retried safely. +`checkCoverage` reports `no_analyzer` when a non-retired language has no registered analyzer, which +would otherwise produce zero findings that look clean. It reports `unlisted_analyzer_language` when +an analyzer handles a language that the policy does not manage. + +### Tenant + +```ts +await service.setPreviewOptIns(tenantId, ["cairo"]); +await service.describeForTenant(tenantId, asOf?); // { policyVersion, asOf, languages: decisions, previewOptIns } +``` + +A tenant may opt into a language before it becomes `preview`. The opt-in takes effect only while +the language is `preview`. `describeForTenant` contains nothing about other tenants and is safe to +return from a tenant-scoped API. + +### Worker + +```ts +const { result, plan } = await runScanWithLanguagePolicy( + service, + { tenantId, scanId, languageMap, asOf: commitDate }, + (analyzable, plan) => registry.analyzeMultiple(files, analyzable), +); +// plan.status: complete | partial | unsupported | empty +// plan.skipped: [{ path, language, reason }] plan.warnings: [{ language, files, sunsetAt, replacement, message }] +``` + +`languageMap` is the engine's path-to-language map. Files that are not in it are not source files, +and are ignored exactly as the registry ignores them. `detectLanguages(paths, policy)` builds one +from extensions: the match is case-insensitive, the longest claimed extension wins. A bare dotfile such as `.vy` is not +treated as an extension, and directory names never match. Callers then override ambiguous files (for example, Rust crates that +depend on `soroban-sdk`). + +### Housekeeping + +`purgeExpiredPins()` removes pins older than `pinRetentionMs`. Run it periodically. A retry that +arrives after its pin was purged is re-pinned under the decisions in force at that time. + +## Configuration and backward compatibility + +| Option | Env | Default | Meaning | +| --- | --- | --- | --- | +| `enabled` | `LANGUAGE_LIFECYCLE_ENABLED` | `true` | Kill switch. `false` analyzes every language for new scans (`policy_disabled`). Existing pins are still honoured. | +| `minDeprecationNoticeDays` | `LANGUAGE_LIFECYCLE_MIN_NOTICE_DAYS` | `90` | Minimum deprecation window, both inside an entry and from the publish day. | +| `unlistedLanguageAction` | `LANGUAGE_LIFECYCLE_UNLISTED_ACTION` | `analyze` | What happens to files in languages the policy does not list: `analyze` (as before) or `skip` (`unlisted_language`). | +| `maxFilesPerScan` | `LANGUAGE_LIFECYCLE_MAX_FILES` | `100000` | Upper bound on files that `planScan` classifies. | +| `maxLanguages` | `LANGUAGE_LIFECYCLE_MAX_LANGUAGES` | `64` | Upper bound on policy entries and preview opt-ins. | +| `pinRetentionMs` | `LANGUAGE_LIFECYCLE_PIN_RETENTION_MS` | 7 days | Pin retention for `purgeExpiredPins`. | + +Unset or blank variables keep their defaults. A variable that is set but malformed throws a +`RangeError` at startup. + +The defaults are backward compatible: + +- Until a policy is published, the built-in `DEFAULT_LANGUAGE_POLICY` (version `0.0.0`) marks every + engine language `supported`. +- Unlisted languages are analyzed. +- The kill switch restores the pre-policy behaviour immediately. + +## Failure semantics + +Every error is a `LanguageLifecycleError` with a `code` and a `retryable` flag. + +| Code | Retryable | When | +| --- | --- | --- | +| `INVALID_INPUT` | No | Bad tenant/scan id (empty, over 200 chars, or containing control characters), bad `asOf`, bad opt-ins, malformed `languageMap` (non-Map, empty/non-string language, empty/NUL/over-4096-char path), more than `maxFilesPerScan` files, or a forced publish without a reason | +| `INVALID_POLICY` | No | The policy fails validation; `violations` lists every problem | +| `TRANSITION_REJECTED` | No | Governance violation; `violations` lists codes and languages | +| `UNSUPPORTED_REPOSITORY` | No | The repository has source files, but none in an analyzable language. The worker throws this before running any analyzer, and the message names the languages found | +| `CONFLICT` | Yes | Another publish landed between read and write (optimistic concurrency on the policy digest) | +| `STORE_UNAVAILABLE` | Yes | Any storage failure. The store's own message is never surfaced or logged, because it may contain paths or credentials | + +### Unsupported input and partial failure + +- `partial` (some files skipped) still runs the scan. `plan.skipped` and `plan.warnings` travel with + the result so that reports can show "N files in retired language X were not analyzed" instead of + implying a clean result. +- `unsupported` fails the scan with a non-retryable error, because retrying the same commit and pin + would fail the same way. +- `empty` (no recognised source) runs the scan with an empty map. This matches the registry's + previous behaviour. + +### Retries + +- `STORE_UNAVAILABLE` from resolution happens before any analysis, so the queue's normal retry + policy applies and nothing has been pinned. +- After a successful pin, a failed attempt can be retried freely. The retry reuses the pin, even + across process restarts, policy publishes, and sunsets. +- If two attempts of the same scan race, the store's write-once pin decides. The loser reads and + uses the winner's pin. + +## Tenant isolation + +- Opt-ins and pins are stored per tenant and always read with an explicit tenant id. The file store + puts each tenant under `tenants//` and refuses records whose contents do not + match their key. +- On-disk paths are SHA-256 digests of identifiers and never the identifiers themselves, so crafted + ids cannot traverse directories. +- Colliding scan ids across tenants get independent pins. +- The policy itself is platform data and contains no tenant identifiers. + +## Metrics and logs + +Prometheus text from `service.metrics.toPrometheus(policy?, asOf?)`: + +| Metric | Labels | +| --- | --- | +| `gasguard_language_pins_total` | `result` (created, reused) | +| `gasguard_language_plans_total` | `status` | +| `gasguard_language_files_skipped_total` | `language`, `reason` | +| `gasguard_language_deprecated_files_total` | `language` | +| `gasguard_language_policy_publishes_total` | `result` (published, unchanged, forced) | +| `gasguard_language_policy_violations_total` | `code` | +| `gasguard_language_lifecycle_errors_total` | `code` | +| `gasguard_language_stage` (gauge) | `language`, `stage` | + +Labels come from closed sets plus policy-controlled language slugs. Tenant ids, scan ids, +repositories, and paths never become labels. + +Logs are one JSON object per line with `component: "language-lifecycle"`: + +- **Events:** `policy_published`, `preview_opt_ins_updated`, `scan_pinned`, `scan_pin_reused`, + `scan_planned`, `plan_scan_failed`, `operation_failed`, `pins_purged`. +- **Fields:** identifiers, versions, digests, fingerprints, counts, and codes only. +- **Sanitization:** `sanitizeLogFields` drops any key matching + payload/content/source/body/path/file/secret/token/password/key/credential/authorization. This + applies to custom loggers too. + +Actionable signals: + +- **`deprecated_files_total` rising for a language:** these tenants need migration outreach before + `sunsetAt`. +- **`files_skipped_total{reason="retired"}` after a sunset:** these tenants are now getting partial + results. +- **`plans_total{status="unsupported"}`:** these scans failed outright. +- **`errors_total{code="STORE_UNAVAILABLE"}`:** storage health. + +## Limitations + +- `FileLanguageLifecycleStore` serializes writes within one process. Multi-process deployments on + shared storage should implement `LanguageLifecycleStore` over a database that does the digest + check in one conditional update. +- Detection by extension cannot separate languages that share one (`.rs` for rust and soroban). + Callers must supply those languages in `languageMap`. +- No HTTP controller is included. The service methods map directly onto operator and tenant + endpoints, behind the existing auth guards. + +## Verification + +```bash +npx jest apps/api/src/language-lifecycle +``` + +- **Unit tests:** config, policy validation and governance, detection, planning, stores (both + implementations), metrics and log hygiene, and the service. +- **Integration tests:** the realistic Soroban DEX repository fixture runs end to end through the + worker, the file store (with restarts), and the engine's real `AnalyzerRegistry`. +- **Regression tests:** frozen sunset-boundary decisions and fingerprints.