diff --git a/CHANGELOG.md b/CHANGELOG.md index eac759c1..c5a03d24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,9 @@ The format is based on Keep a Changelog. - Extended `cds maturity` with Observed, Secured, and Resilient tiers; observability now advances a profile from Automated to Observed, and Resilient requires backup coverage for every stateful module (#1059). +- Documented how CDS security capabilities map to NIST SP 800-207, clarified + the contract-based meaning and limits of the Secured maturity tier, and + filed the separate workload-identity gap (#778, #1068). - Published the Kubernetes target docs and example profiles (issue #77): `docs/kubernetes.md` gains a "Migrating from Docker Compose" guide (same profile, `--target helm` render flag, what translates and what does not), diff --git a/cli/maturity.py b/cli/maturity.py index 5b221301..fddc9044 100644 --- a/cli/maturity.py +++ b/cli/maturity.py @@ -264,4 +264,8 @@ def format_maturity_report(report: dict[str, Any]) -> str: lines.append( "Note: GitOps is reported but does not affect the maturity tiers." ) + lines.append( + "Note: Tier results reflect declared contract presence, not effective " + "runtime enforcement or NIST SP 800-207 conformity." + ) return "\n".join(lines) diff --git a/docs/haven-parity-plan.md b/docs/haven-parity-plan.md index 0d75fc50..39ffe24d 100644 --- a/docs/haven-parity-plan.md +++ b/docs/haven-parity-plan.md @@ -155,11 +155,13 @@ that provide it are still open, tracked by TLS certs for modules that consume a `tls-cert` contract. - `service-mesh` contract + reference module (Istio-shaped): mutual TLS and traffic policy between module instances, opt-in per profile. -- `identity-broker` contract + reference module (Keycloak-shaped): distinct - from `modules/secrets/vault` (secret storage) — this is authentication/SSO, - a different capability that Haven+ also treats as a separate component - (Keycloak vs. Vault-equivalents). Tracked by issue - [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370). +- Identity/SSO: distinct from `modules/secrets/vault` (secret storage) — this + is authentication, a different capability that Haven+ also treats as a + separate component. The Keycloak identity module landed under + [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370); + its shared `oidc-provider` contract and first consumer remain tracked by + [#681](https://github.com/RonaldHensbergen/composable-data-stack/issues/681) + and [#1055](https://github.com/RonaldHensbergen/composable-data-stack/issues/1055). - Extend `cli/resources/rule-schema.json` / `rule-set.json` with security rules that apply to the new contracts (e.g., "profile exposes a service externally without a `tls-cert` binding" the same way existing rules flag @@ -220,7 +222,7 @@ Define a **profile-facing maturity model**, distinct from the roadmap's existing stable/experimental component-maturity tags (which describe *CDS's own* component readiness, not a *deployed profile's* compliance level): -| Tier | Criteria (draft) | +| Tier | Criteria | | --- | --- | | **Basic** | Profile provides a database and cache | | **Automated** | Basic, plus identity, TLS/reverse proxy, and a backup provider | @@ -228,16 +230,13 @@ own* component readiness, not a *deployed profile's* compliance level): | **Secured** | Observed, plus a TLS-certificate or service-mesh provider, or an identity-broker provider alongside a separate OIDC provider | | **Resilient** | Secured, plus backup coverage for every stateful module instance | -Implementation sketch: - -- Add `cds maturity ` (or fold into `cds validate`/`cds test` output) - that inspects the resolved Plan's contract bindings and reports the highest - tier satisfied, plus which bindings are missing for the next tier — mirrors - how `cds security` already reports rule violations without blocking - compile stages by default. -- This is purely additive: it reads the Plan, it does not change plan/render - semantics, and a profile with zero platform-layer bindings still validates - and renders exactly as it does today (tier: Basic). +This model is implemented by `cds maturity` ([#769](https://github.com/RonaldHensbergen/composable-data-stack/issues/769), +extended by [#1059](https://github.com/RonaldHensbergen/composable-data-stack/issues/1059)). +It reads recognized contracts from the resolved Plan and does not change +plan/render semantics. The **Secured** tier is evidence of declared contract +presence, not proof of effective control enforcement or NIST conformity; see +the [Zero Trust Architecture mapping](zero-trust-architecture.md) for the +principle-by-principle mapping and its limits. ## 4. Current status vs. tracked work (as of 2026-10-31) @@ -246,12 +245,12 @@ gap is narrower than section 2 suggested when this document was drafted: | Workstream | Status | Tracking | | --- | --- | --- | -| 3.1 Kubernetes rendering target | **Done, pending merge** — implemented end-to-end with real k3d E2E verification | PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608); originating issues [#65](https://github.com/RonaldHensbergen/composable-data-stack/issues/65), [#73](https://github.com/RonaldHensbergen/composable-data-stack/issues/73), [#74](https://github.com/RonaldHensbergen/composable-data-stack/issues/74), [#77](https://github.com/RonaldHensbergen/composable-data-stack/issues/77) should close on merge | -| 3.1 follow-up: NetworkPolicy generation | Open, not started | [#75](https://github.com/RonaldHensbergen/composable-data-stack/issues/75) (partially — also covered secrets/PVC, which #608 did deliver) | +| 3.1 Kubernetes rendering target | **Done** — implemented end-to-end with real k3d E2E verification | PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608); originating issues [#65](https://github.com/RonaldHensbergen/composable-data-stack/issues/65), [#73](https://github.com/RonaldHensbergen/composable-data-stack/issues/73), [#74](https://github.com/RonaldHensbergen/composable-data-stack/issues/74), [#77](https://github.com/RonaldHensbergen/composable-data-stack/issues/77) | +| 3.1 follow-up: NetworkPolicy generation | **Done** — per-module default-deny ingress/egress policies are rendered for Helm | [#75](https://github.com/RonaldHensbergen/composable-data-stack/issues/75), implemented with the Kubernetes security resources in PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608) | | 3.1 follow-up: CI cluster validation (kind/k3d apply in CI, not just `helm lint`/`template`) | Open, not started | [#76](https://github.com/RonaldHensbergen/composable-data-stack/issues/76) | | 3.2 GitOps | **In progress** — `gitops-target` contract + FluxCD reference module (`modules-experimental/gitops/fluxcd`) rendering Flux bootstrap manifests over the Helm chart | [#767](https://github.com/RonaldHensbergen/composable-data-stack/issues/767) | | 3.3 Observability platform module | Contract/schema done; reference module open | Design closed via [#174](https://github.com/RonaldHensbergen/composable-data-stack/issues/174); module tracked by [#369](https://github.com/RonaldHensbergen/composable-data-stack/issues/369), [#662](https://github.com/RonaldHensbergen/composable-data-stack/issues/662); OpenTelemetry trace-sink/collector tracked by [#777](https://github.com/RonaldHensbergen/composable-data-stack/issues/777); OpenLineage `lineage-sink` contract and Dagster wiring delivered by [#779](https://github.com/RonaldHensbergen/composable-data-stack/issues/779) (PR #831), with an experimental Marquez reference backend (`modules-experimental/observability/marquez`) | -| 3.4 Security platform modules | Pod/container hardening rules done; TLS/mesh/IAM open | Hardening in PR #608 (`cli/k8s_security.py`); TLS tracked by [#205](https://github.com/RonaldHensbergen/composable-data-stack/issues/205), [#577](https://github.com/RonaldHensbergen/composable-data-stack/issues/577), [#579](https://github.com/RonaldHensbergen/composable-data-stack/issues/579), [#580](https://github.com/RonaldHensbergen/composable-data-stack/issues/580); identity tracked by [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370); service mesh (mTLS) tracked by [#776](https://github.com/RonaldHensbergen/composable-data-stack/issues/776); consolidated Zero-Trust framing tracked by [#778](https://github.com/RonaldHensbergen/composable-data-stack/issues/778) | +| 3.4 Security platform modules | Kubernetes pod/container hardening and default-deny NetworkPolicy done; TLS policy exists; OIDC contract and service mesh remain open | Hardening and NetworkPolicy in PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608) / [#75](https://github.com/RonaldHensbergen/composable-data-stack/issues/75); TLS tracked by [#205](https://github.com/RonaldHensbergen/composable-data-stack/issues/205), [#577](https://github.com/RonaldHensbergen/composable-data-stack/issues/577), [#579](https://github.com/RonaldHensbergen/composable-data-stack/issues/579), [#580](https://github.com/RonaldHensbergen/composable-data-stack/issues/580); identity module delivered via [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370), with the shared OIDC contract and consumer tracked by [#681](https://github.com/RonaldHensbergen/composable-data-stack/issues/681) and [#1055](https://github.com/RonaldHensbergen/composable-data-stack/issues/1055); service mesh (mTLS) tracked by [#776](https://github.com/RonaldHensbergen/composable-data-stack/issues/776); workload identity by [#1068](https://github.com/RonaldHensbergen/composable-data-stack/issues/1068); consolidated Zero-Trust framing by [#778](https://github.com/RonaldHensbergen/composable-data-stack/issues/778) | | 3.5 Backup and restore | Not started, but scoped into 4 issues already | [#210](https://github.com/RonaldHensbergen/composable-data-stack/issues/210) (contracts), [#665](https://github.com/RonaldHensbergen/composable-data-stack/issues/665) (Postgres), [#668](https://github.com/RonaldHensbergen/composable-data-stack/issues/668) (generic file/volume), [#669](https://github.com/RonaldHensbergen/composable-data-stack/issues/669) (object storage) | | 3.6 Database operators | Filed, not started | [#768](https://github.com/RonaldHensbergen/composable-data-stack/issues/768) | | 3.7 Maturity model | **Implemented** — Basic/Automated/Observed/Secured/Resilient tiers are reported by `cds maturity` (#769, #1059) | [#769](https://github.com/RonaldHensbergen/composable-data-stack/issues/769), [#1059](https://github.com/RonaldHensbergen/composable-data-stack/issues/1059) | @@ -262,34 +261,31 @@ Notable duplicates closed in favor of the issues above: [#661](https://github.com/RonaldHensbergen/composable-data-stack/issues/661) (Keycloak identity/RBAC module → closed in favor of #370). -Practical implication: once PR #608 merges, this plan's single biggest -prerequisite (3.1) is satisfied, and the highest-leverage next filings are -**3.2 (GitOps)**, **3.6 (database operators)**, and **3.7 (maturity model)** -— the three workstreams with no existing issue at all — plus following up on -the two 3.1 gaps (#75's NetworkPolicy scope, #76's CI cluster validation -scope) that PR #608 did not close. +Practical implication: PR #608 has completed the Kubernetes rendering +baseline, including NetworkPolicy generation. GitOps (3.2) and the maturity +model (3.7) are now implemented; the database-operator pattern (3.6) and +CI-based cluster validation (#76) remain open. ## 5. Sequencing The workstreams have real dependencies; suggested order, updated for the current status in section 4: -1. **3.1 follow-ups (NetworkPolicy, CI cluster validation)** — close out the - two gaps PR #608 left in an otherwise-complete workstream before building - on top of it. +1. **3.1 CI cluster validation** — close the remaining validation gap tracked + by [#76](https://github.com/RonaldHensbergen/composable-data-stack/issues/76); + NetworkPolicy generation is delivered under #75 / PR #608. 2. **3.3 Observability platform module** — no Kubernetes dependency, and closes an already-designed but unimplemented contract ([docs/observability.md](observability.md)); issues #369/#662 are ready to pick up now. -3. **3.7 Maturity model** — the Basic, Automated, Observed, Secured, and - Resilient report tiers are implemented independently of the remaining - provider implementations; tiers reflect contracts in the resolved plan, - not operational compliance or production suitability. -4. **3.4 Security platform modules** (issues #205/#370/#577/#579/#580) and - **3.2 GitOps** (#767, in progress) — parallelizable now that 3.1 has landed. -5. **3.5 Backup** (issues #210/#665/#668/#669) and **3.6 Database operators** - (no issue yet) — lowest priority; highest Kubernetes-specific complexity, - least reusable by Compose-only users. +3. **3.4 Security platform modules** — continue the OIDC contract/consumer + (#681/#1055), service mesh (#776), and workload identity (#1068) follow-ups; + the TLS contract work has landed. See the [Zero Trust mapping](zero-trust-architecture.md) + for how these capabilities relate without claiming full NIST conformity. +4. **3.5 Backup** (issues #210/#665/#668/#669) and **3.6 Database operators** + (#768) — Kubernetes-specific work with less reuse for Compose-only users. +5. **3.7 Maturity model** and **3.2 GitOps** are implemented (#769/#1059 + and #767 respectively); their status is documented above. ## 6. Non-goals @@ -356,3 +352,6 @@ scoped separately from Haven parity: Haven parity is "does CDS produce a Common Ground-shaped platform", while the NL alignment milestone is "does CDS help operators of that platform meet Dutch data-sharing and security-law obligations." +The NIST Zero Trust mapping in +[docs/zero-trust-architecture.md](zero-trust-architecture.md) complements +this alignment by documenting security-control evidence and its limits. diff --git a/docs/roadmap.md b/docs/roadmap.md index 2f1f9f6e..7310139a 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -92,8 +92,8 @@ These work but may have breaking changes in upcoming releases: - 📋 **Stabilize Vault-backed profile** — validate and harden `local-dagster-postgres-superset-vault` for regular use - 📋 **`cds update`** — refresh profiles, modules, and contracts previously fetched with `cds get` (#348) -- 📋 **Kubernetes platform baseline** — close the gap to the full plan in [docs/haven-parity-plan.md](haven-parity-plan.md): self-contained `cds up --target helm` (#690), K8s secrets/configmaps/network policies (#75), Kubernetes docs/migration guide (#77) -- 🚧 **Haven parity — GitOps rendering/module support** — in progress: `gitops-target` contract + FluxCD reference module rendering Flux bootstrap manifests over the Helm chart output (#767); the remaining previously untracked workstream is the Kubernetes-native database operator pattern (#768); the Basic/Automated/Observed/Secured/Resilient profile maturity report is implemented across #769 and #1059 +- 📋 **Kubernetes platform baseline** — close the gap to the full plan in [docs/haven-parity-plan.md](haven-parity-plan.md): self-contained `cds up --target helm` (#690), K8s secrets/configmaps/network policies (#75), Kubernetes docs/migration guide (#77); see the [Zero Trust mapping](zero-trust-architecture.md) for the security boundaries and remaining gaps +- 🚧 **Haven parity — Kubernetes database operators** — GitOps rendering is delivered: `gitops-target` contract + FluxCD reference module render Flux bootstrap manifests over the Helm chart output (#767); the Kubernetes-native database operator pattern remains open (#768); the Basic/Automated/Observed/Secured/Resilient profile maturity report is implemented across #769 and #1059 - 📋 **Zero-Trust and OpenTelemetry** — previously unaddressed platform/standards gaps identified during the Dutch public-sector alignment review: service-mesh contract + Istio reference module (#776), consolidated Zero-Trust Architecture framing over existing TLS/identity/mesh/network-policy work (#778), `trace-sink` contract + OpenTelemetry Collector reference module (#777) --- diff --git a/docs/zero-trust-architecture.md b/docs/zero-trust-architecture.md new file mode 100644 index 00000000..f5546632 --- /dev/null +++ b/docs/zero-trust-architecture.md @@ -0,0 +1,71 @@ +# Zero Trust Architecture mapping + +This document maps Composable Data Stack (CDS) capabilities and tracked work +to the core principles of [NIST SP 800-207, Zero Trust +Architecture](https://csrc.nist.gov/pubs/sp/800/207/final). It consolidates +existing platform work; it does not claim that CDS or a profile is NIST +conformant. + +Zero Trust is an operational architecture, not a set of contracts whose +presence proves enforcement. CDS can validate declared configuration and +render platform resources, but operators remain responsible for their +identity providers, cluster, runtime, network, and policies. In particular, +`cds maturity` reports declared contract presence, not effective runtime +controls or a security certification. + +## Mapping to the core principles + +| Principle | CDS capabilities and tracking | Limits | +| --- | --- | --- | +| **Verify explicitly** — authenticate and authorize access using relevant identity and context rather than network location alone. | Database and cache authentication policies reject insecure trust modes in production profiles ([#206](https://github.com/RonaldHensbergen/composable-data-stack/issues/206)). Shared TLS requirements and reverse-proxy metadata let consumers declare and validate transport expectations ([#205](https://github.com/RonaldHensbergen/composable-data-stack/issues/205), [#577](https://github.com/RonaldHensbergen/composable-data-stack/issues/577), [#579](https://github.com/RonaldHensbergen/composable-data-stack/issues/579), [#580](https://github.com/RonaldHensbergen/composable-data-stack/issues/580)). The identity module work began under [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370); its shared OIDC contract and consumer wiring remain tracked by [#681](https://github.com/RonaldHensbergen/composable-data-stack/issues/681) and [#1055](https://github.com/RonaldHensbergen/composable-data-stack/issues/1055). | Declared TLS does not prove that every connection is encrypted or that certificates are trusted. OIDC provider wiring is not yet a general per-service authorization policy, and a network boundary does not authenticate a workload. | +| **Use least-privilege access** — grant only the access needed to a resource. | The Helm renderer emits a per-module default-deny ingress/egress NetworkPolicy, then permits contract-derived traffic, same-module traffic, and DNS; custom allow rules are explicit. This work was delivered under [#75](https://github.com/RonaldHensbergen/composable-data-stack/issues/75). Kubernetes pod hardening and opt-in RBAC are covered by [PR #608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608). The optional service-mesh capability for mTLS and traffic policy is tracked by [#776](https://github.com/RonaldHensbergen/composable-data-stack/issues/776). | NetworkPolicy and pod hardening apply to the Helm target, not Docker Compose. The default-deny policy limits network reachability; it does not itself establish peer identity or application authorization. A service mesh is not yet implemented. | +| **Assume breach** — minimize blast radius, continuously assess access, and retain evidence for detection and response. | Per-module network boundaries and hardened Kubernetes workloads reduce some lateral movement. The local audit trail records rendered/applied command outcomes and module/image versions ([#737](https://github.com/RonaldHensbergen/composable-data-stack/issues/737)). | The audit trail is local command history, not centralized or continuous runtime monitoring. It does not collect workload activity or provide incident response. | + +These are partial, target-dependent controls. The mapping is intended to make +the evidence and its limits reviewable, not to imply that a profile satisfying +one row satisfies the whole NIST architecture. + +## How `cds maturity` uses the mapping + +`cds maturity` uses the following profile-facing tiers, based on recognized +contracts in the resolved plan: + +| Tier | Contract evidence required | +| --- | --- | +| **Basic** | A database and cache provider | +| **Automated** | Basic, plus identity, a reverse proxy, and a backup provider | +| **Observed** | Automated, plus at least one logging, metrics, or tracing provider | +| **Secured** | Observed, plus a TLS-certificate provider, a service-mesh provider, or an identity broker backed by a separate OIDC provider | +| **Resilient** | Secured, plus backup coverage for every stateful module instance | + +The **Secured** threshold therefore indicates additional security-control +contract evidence on top of an Observed profile. It is not a score for all +three NIST principles, and does not attest that a provider is correctly +configured or that a control is effective. The tier names and contract +requirements are defined by [`cli/maturity.py`](../cli/maturity.py); the CLI +also states this limitation in its text report. + +## Remaining gaps and follow-ups + +- **Workload identity and attestation:** There is no provider-neutral contract + for cryptographically attested workload identity or trust-domain evidence. + This is distinct from user-facing OIDC identity ([#681](https://github.com/RonaldHensbergen/composable-data-stack/issues/681)) + and from network-level mTLS. A focused follow-up, [#1068](https://github.com/RonaldHensbergen/composable-data-stack/issues/1068), + tracks the contract and an initial Kubernetes reference. +- **OIDC authorization wiring:** The Keycloak `oidc-provider` contract and + its first consumer remain open in [#681](https://github.com/RonaldHensbergen/composable-data-stack/issues/681) + and [#1055](https://github.com/RonaldHensbergen/composable-data-stack/issues/1055). + Even when implemented, provider binding alone will not represent every + per-resource authorization policy. +- **Service-mesh enforcement:** mTLS and traffic-policy support remain open + under [#776](https://github.com/RonaldHensbergen/composable-data-stack/issues/776). +- **Target coverage:** The default-deny NetworkPolicy is generated for Helm; + Docker Compose has no equivalent CDS-generated per-module network policy. + Do not infer cross-target segmentation from the maturity report. +- **Continuous verification:** The local audit log in [docs/audit-log.md](audit-log.md) + is not a runtime event stream. Continuous workload monitoring and response + remain operator responsibilities. + +The status and background for this work are also tracked in +[docs/haven-parity-plan.md](haven-parity-plan.md), workstreams 3.4 and 3.7, +and in [docs/roadmap.md](roadmap.md). diff --git a/tests/test_maturity.py b/tests/test_maturity.py index e2d79c29..3bd07d42 100644 --- a/tests/test_maturity.py +++ b/tests/test_maturity.py @@ -294,6 +294,10 @@ def test_text_output_lists_tier_and_provider_modules(self) -> None: self.assertIn("Observed: not met", text) self.assertIn("Secured: not met", text) self.assertIn("Resilient: not met", text) + self.assertIn( + "not effective runtime enforcement or NIST SP 800-207 conformity", + text, + ) class MaturityCommandTest(unittest.TestCase):