Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ The format is based on Keep a Changelog.
- Extended `cds maturity` with Observed, Secured, and Resilient tiers;
observability now advances a profile from Automated to Observed, and
Resilient requires backup coverage for every stateful module (#1059).
- Documented how CDS security capabilities map to NIST SP 800-207, clarified
the contract-based meaning and limits of the Secured maturity tier, and
filed the separate workload-identity gap (#778, #1068).
- Published the Kubernetes target docs and example profiles (issue #77):
`docs/kubernetes.md` gains a "Migrating from Docker Compose" guide (same
profile, `--target helm` render flag, what translates and what does not),
Expand Down
4 changes: 4 additions & 0 deletions cli/maturity.py
Original file line number Diff line number Diff line change
Expand Up @@ -264,4 +264,8 @@ def format_maturity_report(report: dict[str, Any]) -> str:
lines.append(
"Note: GitOps is reported but does not affect the maturity tiers."
)
lines.append(
"Note: Tier results reflect declared contract presence, not effective "
"runtime enforcement or NIST SP 800-207 conformity."
)
return "\n".join(lines)
73 changes: 36 additions & 37 deletions docs/haven-parity-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,11 +155,13 @@ that provide it are still open, tracked by
TLS certs for modules that consume a `tls-cert` contract.
- `service-mesh` contract + reference module (Istio-shaped): mutual TLS and
traffic policy between module instances, opt-in per profile.
- `identity-broker` contract + reference module (Keycloak-shaped): distinct
from `modules/secrets/vault` (secret storage) — this is authentication/SSO,
a different capability that Haven+ also treats as a separate component
(Keycloak vs. Vault-equivalents). Tracked by issue
[#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370).
- Identity/SSO: distinct from `modules/secrets/vault` (secret storage) — this
is authentication, a different capability that Haven+ also treats as a
separate component. The Keycloak identity module landed under
[#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370);
its shared `oidc-provider` contract and first consumer remain tracked by
[#681](https://github.com/RonaldHensbergen/composable-data-stack/issues/681)
and [#1055](https://github.com/RonaldHensbergen/composable-data-stack/issues/1055).
- Extend `cli/resources/rule-schema.json` / `rule-set.json` with security
rules that apply to the new contracts (e.g., "profile exposes a service
externally without a `tls-cert` binding" the same way existing rules flag
Expand Down Expand Up @@ -220,24 +222,21 @@ Define a **profile-facing maturity model**, distinct from the roadmap's
existing stable/experimental component-maturity tags (which describe *CDS's
own* component readiness, not a *deployed profile's* compliance level):

| Tier | Criteria (draft) |
| Tier | Criteria |
| --- | --- |
| **Basic** | Profile provides a database and cache |
| **Automated** | Basic, plus identity, TLS/reverse proxy, and a backup provider |
| **Observed** | Automated, plus at least one logging, metrics, or tracing provider |
| **Secured** | Observed, plus a TLS-certificate or service-mesh provider, or an identity-broker provider alongside a separate OIDC provider |
| **Resilient** | Secured, plus backup coverage for every stateful module instance |

Implementation sketch:

- Add `cds maturity <profile>` (or fold into `cds validate`/`cds test` output)
that inspects the resolved Plan's contract bindings and reports the highest
tier satisfied, plus which bindings are missing for the next tier — mirrors
how `cds security` already reports rule violations without blocking
compile stages by default.
- This is purely additive: it reads the Plan, it does not change plan/render
semantics, and a profile with zero platform-layer bindings still validates
and renders exactly as it does today (tier: Basic).
This model is implemented by `cds maturity` ([#769](https://github.com/RonaldHensbergen/composable-data-stack/issues/769),
extended by [#1059](https://github.com/RonaldHensbergen/composable-data-stack/issues/1059)).
It reads recognized contracts from the resolved Plan and does not change
plan/render semantics. The **Secured** tier is evidence of declared contract
presence, not proof of effective control enforcement or NIST conformity; see
the [Zero Trust Architecture mapping](zero-trust-architecture.md) for the
principle-by-principle mapping and its limits.

## 4. Current status vs. tracked work (as of 2026-10-31)

Expand All @@ -246,12 +245,12 @@ gap is narrower than section 2 suggested when this document was drafted:

| Workstream | Status | Tracking |
| --- | --- | --- |
| 3.1 Kubernetes rendering target | **Done, pending merge** — implemented end-to-end with real k3d E2E verification | PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608); originating issues [#65](https://github.com/RonaldHensbergen/composable-data-stack/issues/65), [#73](https://github.com/RonaldHensbergen/composable-data-stack/issues/73), [#74](https://github.com/RonaldHensbergen/composable-data-stack/issues/74), [#77](https://github.com/RonaldHensbergen/composable-data-stack/issues/77) should close on merge |
| 3.1 follow-up: NetworkPolicy generation | Open, not started | [#75](https://github.com/RonaldHensbergen/composable-data-stack/issues/75) (partially — also covered secrets/PVC, which #608 did deliver) |
| 3.1 Kubernetes rendering target | **Done** — implemented end-to-end with real k3d E2E verification | PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608); originating issues [#65](https://github.com/RonaldHensbergen/composable-data-stack/issues/65), [#73](https://github.com/RonaldHensbergen/composable-data-stack/issues/73), [#74](https://github.com/RonaldHensbergen/composable-data-stack/issues/74), [#77](https://github.com/RonaldHensbergen/composable-data-stack/issues/77) |
| 3.1 follow-up: NetworkPolicy generation | **Done** — per-module default-deny ingress/egress policies are rendered for Helm | [#75](https://github.com/RonaldHensbergen/composable-data-stack/issues/75), implemented with the Kubernetes security resources in PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608) |
| 3.1 follow-up: CI cluster validation (kind/k3d apply in CI, not just `helm lint`/`template`) | Open, not started | [#76](https://github.com/RonaldHensbergen/composable-data-stack/issues/76) |
| 3.2 GitOps | **In progress** — `gitops-target` contract + FluxCD reference module (`modules-experimental/gitops/fluxcd`) rendering Flux bootstrap manifests over the Helm chart | [#767](https://github.com/RonaldHensbergen/composable-data-stack/issues/767) |
| 3.3 Observability platform module | Contract/schema done; reference module open | Design closed via [#174](https://github.com/RonaldHensbergen/composable-data-stack/issues/174); module tracked by [#369](https://github.com/RonaldHensbergen/composable-data-stack/issues/369), [#662](https://github.com/RonaldHensbergen/composable-data-stack/issues/662); OpenTelemetry trace-sink/collector tracked by [#777](https://github.com/RonaldHensbergen/composable-data-stack/issues/777); OpenLineage `lineage-sink` contract and Dagster wiring delivered by [#779](https://github.com/RonaldHensbergen/composable-data-stack/issues/779) (PR #831), with an experimental Marquez reference backend (`modules-experimental/observability/marquez`) |
| 3.4 Security platform modules | Pod/container hardening rules done; TLS/mesh/IAM open | Hardening in PR #608 (`cli/k8s_security.py`); TLS tracked by [#205](https://github.com/RonaldHensbergen/composable-data-stack/issues/205), [#577](https://github.com/RonaldHensbergen/composable-data-stack/issues/577), [#579](https://github.com/RonaldHensbergen/composable-data-stack/issues/579), [#580](https://github.com/RonaldHensbergen/composable-data-stack/issues/580); identity tracked by [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370); service mesh (mTLS) tracked by [#776](https://github.com/RonaldHensbergen/composable-data-stack/issues/776); consolidated Zero-Trust framing tracked by [#778](https://github.com/RonaldHensbergen/composable-data-stack/issues/778) |
| 3.4 Security platform modules | Kubernetes pod/container hardening and default-deny NetworkPolicy done; TLS policy exists; OIDC contract and service mesh remain open | Hardening and NetworkPolicy in PR [#608](https://github.com/RonaldHensbergen/composable-data-stack/pull/608) / [#75](https://github.com/RonaldHensbergen/composable-data-stack/issues/75); TLS tracked by [#205](https://github.com/RonaldHensbergen/composable-data-stack/issues/205), [#577](https://github.com/RonaldHensbergen/composable-data-stack/issues/577), [#579](https://github.com/RonaldHensbergen/composable-data-stack/issues/579), [#580](https://github.com/RonaldHensbergen/composable-data-stack/issues/580); identity module delivered via [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370), with the shared OIDC contract and consumer tracked by [#681](https://github.com/RonaldHensbergen/composable-data-stack/issues/681) and [#1055](https://github.com/RonaldHensbergen/composable-data-stack/issues/1055); service mesh (mTLS) tracked by [#776](https://github.com/RonaldHensbergen/composable-data-stack/issues/776); workload identity by [#1068](https://github.com/RonaldHensbergen/composable-data-stack/issues/1068); consolidated Zero-Trust framing by [#778](https://github.com/RonaldHensbergen/composable-data-stack/issues/778) |
| 3.5 Backup and restore | Not started, but scoped into 4 issues already | [#210](https://github.com/RonaldHensbergen/composable-data-stack/issues/210) (contracts), [#665](https://github.com/RonaldHensbergen/composable-data-stack/issues/665) (Postgres), [#668](https://github.com/RonaldHensbergen/composable-data-stack/issues/668) (generic file/volume), [#669](https://github.com/RonaldHensbergen/composable-data-stack/issues/669) (object storage) |
| 3.6 Database operators | Filed, not started | [#768](https://github.com/RonaldHensbergen/composable-data-stack/issues/768) |
| 3.7 Maturity model | **Implemented** — Basic/Automated/Observed/Secured/Resilient tiers are reported by `cds maturity` (#769, #1059) | [#769](https://github.com/RonaldHensbergen/composable-data-stack/issues/769), [#1059](https://github.com/RonaldHensbergen/composable-data-stack/issues/1059) |
Expand All @@ -262,34 +261,31 @@ Notable duplicates closed in favor of the issues above:
[#661](https://github.com/RonaldHensbergen/composable-data-stack/issues/661)
(Keycloak identity/RBAC module → closed in favor of #370).

Practical implication: once PR #608 merges, this plan's single biggest
prerequisite (3.1) is satisfied, and the highest-leverage next filings are
**3.2 (GitOps)**, **3.6 (database operators)**, and **3.7 (maturity model)**
— the three workstreams with no existing issue at all — plus following up on
the two 3.1 gaps (#75's NetworkPolicy scope, #76's CI cluster validation
scope) that PR #608 did not close.
Practical implication: PR #608 has completed the Kubernetes rendering
baseline, including NetworkPolicy generation. GitOps (3.2) and the maturity
model (3.7) are now implemented; the database-operator pattern (3.6) and
CI-based cluster validation (#76) remain open.

## 5. Sequencing

The workstreams have real dependencies; suggested order, updated for the
current status in section 4:

1. **3.1 follow-ups (NetworkPolicy, CI cluster validation)** — close out the
two gaps PR #608 left in an otherwise-complete workstream before building
on top of it.
1. **3.1 CI cluster validation** — close the remaining validation gap tracked
by [#76](https://github.com/RonaldHensbergen/composable-data-stack/issues/76);
NetworkPolicy generation is delivered under #75 / PR #608.
2. **3.3 Observability platform module** — no Kubernetes dependency, and
closes an already-designed but unimplemented contract
([docs/observability.md](observability.md)); issues #369/#662 are ready
to pick up now.
3. **3.7 Maturity model** — the Basic, Automated, Observed, Secured, and
Resilient report tiers are implemented independently of the remaining
provider implementations; tiers reflect contracts in the resolved plan,
not operational compliance or production suitability.
4. **3.4 Security platform modules** (issues #205/#370/#577/#579/#580) and
**3.2 GitOps** (#767, in progress) — parallelizable now that 3.1 has landed.
5. **3.5 Backup** (issues #210/#665/#668/#669) and **3.6 Database operators**
(no issue yet) — lowest priority; highest Kubernetes-specific complexity,
least reusable by Compose-only users.
3. **3.4 Security platform modules** — continue the OIDC contract/consumer
(#681/#1055), service mesh (#776), and workload identity (#1068) follow-ups;
the TLS contract work has landed. See the [Zero Trust mapping](zero-trust-architecture.md)
for how these capabilities relate without claiming full NIST conformity.
4. **3.5 Backup** (issues #210/#665/#668/#669) and **3.6 Database operators**
(#768) — Kubernetes-specific work with less reuse for Compose-only users.
5. **3.7 Maturity model** and **3.2 GitOps** are implemented (#769/#1059
and #767 respectively); their status is documented above.

## 6. Non-goals

Expand Down Expand Up @@ -356,3 +352,6 @@ scoped separately from
Haven parity: Haven parity is "does CDS produce a Common Ground-shaped
platform", while the NL alignment milestone is "does CDS help operators of
that platform meet Dutch data-sharing and security-law obligations."
The NIST Zero Trust mapping in
[docs/zero-trust-architecture.md](zero-trust-architecture.md) complements
this alignment by documenting security-control evidence and its limits.
4 changes: 2 additions & 2 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ These work but may have breaking changes in upcoming releases:

- 📋 **Stabilize Vault-backed profile** — validate and harden `local-dagster-postgres-superset-vault` for regular use
- 📋 **`cds update`** — refresh profiles, modules, and contracts previously fetched with `cds get` (#348)
- 📋 **Kubernetes platform baseline** — close the gap to the full plan in [docs/haven-parity-plan.md](haven-parity-plan.md): self-contained `cds up --target helm` (#690), K8s secrets/configmaps/network policies (#75), Kubernetes docs/migration guide (#77)
- 🚧 **Haven parity — GitOps rendering/module support** — in progress: `gitops-target` contract + FluxCD reference module rendering Flux bootstrap manifests over the Helm chart output (#767); the remaining previously untracked workstream is the Kubernetes-native database operator pattern (#768); the Basic/Automated/Observed/Secured/Resilient profile maturity report is implemented across #769 and #1059
- 📋 **Kubernetes platform baseline** — close the gap to the full plan in [docs/haven-parity-plan.md](haven-parity-plan.md): self-contained `cds up --target helm` (#690), K8s secrets/configmaps/network policies (#75), Kubernetes docs/migration guide (#77); see the [Zero Trust mapping](zero-trust-architecture.md) for the security boundaries and remaining gaps
- 🚧 **Haven parity — Kubernetes database operators** — GitOps rendering is delivered: `gitops-target` contract + FluxCD reference module render Flux bootstrap manifests over the Helm chart output (#767); the Kubernetes-native database operator pattern remains open (#768); the Basic/Automated/Observed/Secured/Resilient profile maturity report is implemented across #769 and #1059
- 📋 **Zero-Trust and OpenTelemetry** — previously unaddressed platform/standards gaps identified during the Dutch public-sector alignment review: service-mesh contract + Istio reference module (#776), consolidated Zero-Trust Architecture framing over existing TLS/identity/mesh/network-policy work (#778), `trace-sink` contract + OpenTelemetry Collector reference module (#777)

---
Expand Down
Loading
Loading