Skip to content

cf2cnp 0.5.0 → 0.7.0: download_url behind a proxy, many flows per request, ClusterMesh-aware selectors, L7 rules, DNS visibility, review before generating, merge, descriptions from the rules — and Cilium's own policy types with the agent's validation - #3

Open
ephico2real2 wants to merge 44 commits into
onzack:mainfrom
ephico2real2:feat/external-url-multi-flow

Conversation

@ephico2real2

@ephico2real2 ephico2real2 commented Sep 13, 2026 •

Copy link
Copy Markdown

Closes #2.

@R-Studio

This PR is the fork's develop branch (https://github.com/ephico2real2/cf2cnp/tree/develop), released as 0.5.0 → 0.7.0. Everything in it was built for a Cilium 1.20.1 ClusterMesh and proven there; each row below links to the recorded demo (transcript, Hubble verdicts, screenshots) that exercises it. If you prefer smaller PRs, every feature is also its own branch (enh/E1 … enh/E10) and I will open them one by one on request.

1. Features

Feature How to use it What it does Proven in
download_url behind a TLS-terminating proxy (the bug in #2) --external-url / CF2CNP_EXTERNAL_URL, or the proxy's Forwarded / X-Forwarded-Proto / X-Forwarded-Host The JSON answer's download_url is https://… behind an HTTPS gateway instead of http://. Precedence: the flag, then RFC 7239 Forwarded, then X-Forwarded-*, then r.TLS/r.Host; a garbage proto is ignored demo 26 Part 14
Many flows per request POST a JSON object, a JSON array, or NDJSON (hubble observe -o json as is) Flows to the same workload become one policy with one rule per peer; identical rules are kept once demo 27 Part 4
Policy names that cannot collide automatic The name is a function of every identifying label the selector carries (shop, shop-frontend, shop-blue-frontend); two components of one app no longer produce two objects with one name that kubectl apply would replace demo 27 Part 4
Labels on every policy automatic app.kubernetes.io/managed-by: cf2cnp plus the selector's labels, so kubectl get cnp -l … finds them demo 27 Part 5
?name=<name> query parameter Names the single resulting policy (sanitised); a 400 if the flows make more than one demo 26 Part 5
Cluster-aware selectors (ClusterMesh) automatic When the flow's source and destination clusters differ, the peer selector carries io.cilium.k8s.policy.cluster: <cluster>. Since Cilium 1.19 a selector without it matches the local cluster only, so the old output dropped the real cross-cluster caller and admitted a same-labelled local pod demo 29 Part 5, Part 6
Layer-7 rules --l7, ?l7=true, the page's box HTTP method + path and DNS names from flows that carry l7: one port rule per port that has L7 records, paths anchored and escaped with an optional query string (Envoy matches the whole :path), DNS names kept exactly as observed demo 30 Part 3, Part 4
DNS visibility on demand --dns-visibility, ?dnsVisibility=true, the page's box Adds the kube-dns L7 DNS rule (matchPattern: "*") to a world policy whose flows carry no names, so the next flows carry destination_names and the next generation writes toFQDNs instead of a CIDR demo 31 Parts 1–3
Review before generating ?exclude=<label>=<value> (repeatable; a comma-joined set must match entirely), the page's peer checklist Drops the flows of a peer you did not intend — a dropped stranger is otherwise a rule, because cf2cnp reads flows, not verdicts. Everything excluded is a 400 demo 32 Part 1
cf2cnp merge cf2cnp merge --existing policy.yaml --input flows.ndjson [-o out.yaml] [--l7] Adds the rules new flows produce to an existing policy file and nothing else: key order, comments, quoting and indentation kept (a pull request shows the added rule only), idempotent, refuses another target (namespace, name or selector) demo 32 Part 2, Part 2b
Descriptions from the rules automatic Allow ingress to shop/frontend in cf2cnp-lab27: from pos on TCP/80 (HTTP GET /, GET /checkout); from kiosk on TCP/80 — subject and peers as the selector names them, a peer's namespace or cluster when it differs, ports, L7 rules; cut with "and N more" past six peers demo 35 Part 4
The page / A summary of the pasted flows (direction, verdict, peer → workload:port), the peer checklist, the two boxes, a policy-name field, an access-token field, Copy / Download / Load example / Clear, and the kubectl apply hint. No backticks in the script (the page is a Go raw string) demo 27 Part 6, demo 30 Part 5
JSON answer Accept: application/json or X-Grafana-Action Gains flows, policies and yaml; download_url, filename, message unchanged, and a single flow's cache id is still its UUID, so the hubble-observer dashboard's Download link keeps working demo 27 Part 7
Release binaries the GitHub release of every v* tag linux/darwin × amd64/arm64 archives and a checksums file, for pipelines that run cf2cnp merge without a Go toolchain demo 32 Part 0, Part 3

| Cilium's own policy types (0.7.0) | automatic | The document is github.com/cilium/cilium/pkg/policy/api at the pinned Cilium version (v1.20.1): every field of the CiliumNetworkPolicy spec exists, the agent's own Sanitize() runs on every generated policy (on a copy, so the file stays as a user writes it), the CRD of the same version is embedded and checked byte for byte in CI, and cf2cnp version prints the spec version the build supports. Output is byte-identical to 0.6.3 for the 14 recorded captures except two deliberate changes (below) | demo 35 Part 4 regenerated; docs/CRD-SPEC-FORENSICS.md |
| cf2cnp validate (0.7.0) | cf2cnp validate policy.yaml … | Every document of every file through the agent's checks plus the object's: Sanitize, ObjectMeta (DNS-1123 names, label keys), the CRD's protocol enum, apiVersion/kind, no nodeSelector on a namespaced policy, a namespace on a clusterwide one, specs; exit = the number refused | review record |
| fromCIDR for external sources (0.7.0) | automatic | An ingress flow from reserved:world (or world-ipv4/world-ipv6 on dual-stack) with an address becomes fromCIDR: [<addr>/32] (/128 IPv6) — what the receiver saw — instead of fromEntities: [world] | docs/CRD-SPEC-FORENSICS.md §2.5 |
| The DNS resolver rule from the flows (0.7.0) | --dns-profile auto\|kubernetes\|openshift, --dns-resolver <ns>[/<label>=<v>]:<port>[/<proto>], ?dnsProfile=, ?dnsResolver=, serve and chart defaults (dns.profile, dns.resolver) | The rule toFQDNs and --dns-visibility need names the resolver the workload actually asked (kube-dns by k8s-app, the CoreDNS Helm chart by app.kubernetes.io/name, NodeLocal DNSCache, OpenShift's openshift-dns on 5353 with no label — Cilium's DNS guide), protocol ANY as Cilium's examples write it (a truncated UDP answer retries over TCP) | review record |
| merge --l7 (0.7.0) | cf2cnp merge … --l7 | The flag the policy-PR template already passed | review record |

2. Chart (0.4.0 → 0.7.0)

Value Default What it does Proven in
externalURL "" --external-url for the pod demo 26 Part 14
extraArgs, extraEnv [] Appended to serve --port <containerPort>; the container's args are now rendered, so a flag can be given at all —
cors.allowedOrigins [] (any origin, as before) Only the listed origins are echoed in Access-Control-Allow-Origin (with Vary: Origin); a foreign origin gets no CORS header demo 33 Part 2
auth.token, auth.existingSecret "" A bearer token on /generate and /download — never on the page, /health or preflight; compared in constant time after hashing, so length does not leak; scheme case-insensitive demo 33 Part 3
dns.profile, dns.resolver, networkPolicy.dnsEgress (0.7.0) auto, "", {} The DNS resolver a request gets when it names none (an OpenShift cluster sets dns.profile: openshift once); the chart's own policy's DNS egress follows the profile, and refuses to render when a custom resolver is named without a matching dnsEgress review record
networkPolicy.enabled, networkPolicy.fromEndpoints false, [] A CiliumNetworkPolicy for the pod: ingress from reserved:ingress (a Gateway) and reserved:host (the kubelet's probes) on the container port, plus the listed endpoints; egress to kube-dns only demo 33 Part 2

3. Bug fixes

Fix Release Found by
download_url said http:// behind an HTTPS proxy (r.TLS is nil after TLS termination) 0.5.0 #2
Two flows into one service produced two objects with the same metadata.name; in file mode the second file overwrote the first (TestGeneratePolicies_OneFilePerWorkload fails on main) 0.5.0 demo 26
Two components sharing app.kubernetes.io/name produced one policy name — the second kubectl apply replaced the first 0.5.0 demo 27 Part 1
A {} body (no flow) was accepted and produced an empty policy 0.5.1 review
The proxy's Forwarded host was used unsanitised 0.5.1 review
An L7 rule seen on one port was attached to every port of the peer pair (an HTTP rule on 8080 restricted 5432) 0.6.0 review
The anchored path regex did not allow a query string, so ?id=42 would have been blocked once enforced 0.6.0 review
merge re-serialised the whole file (alphabetical keys, 4-space indentation): a PR diff was the file, not the rule 0.6.1 demo 32 Part 2
A pod's own DNS lookups plus the DNS-visibility rule left kube-dns 53/UDP twice in one policy 0.6.1 demo 31 Part 1
The description said only the namespaces (Allow ingress traffic from ns to ns for the shop) 0.6.2 / 0.6.3 operator review of the page
An IPv6 world peer got a /32 0.7.0 forensics
A dual-stack world peer (reserved:world-ipv4 / -ipv6) was neither world nor an entity: an egress to it produced no peer at all 0.7.0 review (Cursor, Codex)
The DNS resolver rule was hard-coded (kube-system, k8s-app: kube-dns, 53/UDP): on OpenShift and with the CoreDNS Helm chart it selected nothing and cut the pod off from DNS once enforced; UDP alone denied the TCP retry of a truncated answer 0.7.0 plan 003; review
merge had no --l7 although the policy-PR template passed it (unknown flag) 0.7.0 review (Codex)

4. Tests

  • Unit tests: 46 at 0.6.3, 70 at 0.7.0 (14 golden captures byte-identical to 0.6.3 except two deliberate changes named in internal/testdata/golden/README.md; the Sanitize, admission, dual-stack, DNS-resolver and document-decoding cases), go test ./... green at every tag; CI also checks the embedded CRD byte for byte against the pinned module, validates every golden with kubectl-validate, keeps the Kubernetes client machinery out of the binary (40 MB ceiling), and runs the policy-PR template's two calls. By area: parsing (TestParseFlowsFromBytes_*, _RejectsNonFlows, ParseFlowsFromDirectory_ManyPerFile), the proxy URL (TestDownloadURL_PrecedenceAndHostSanity, _Scheme, TestYolo_JSONUsesBaseURL), naming and merging across flows (TestBuildPolicies_NameIsAFunctionOfTheSelector, _MergesSameWorkload, _KeepsDistinctWorkloads, _DedupesIdenticalRules, TestGeneratePolicies_OneFilePerWorkload), ClusterMesh (TestBuildPolicies_CrossClusterPeerNamesItsCluster, _SameClusterHasNoClusterLabel, _SameLabelsTwoClustersTwoRules, _ClusterLabelWhenClusterNameEmpty), L7 (TestBuildPolicies_L7HTTPRules, _L7DNSRules, _L7RulesArePerPort, TestL7Rules_PathIsEscapedAndAllowsAQuery, _KeepsTheQueryAsObserved), DNS visibility (TestBuildPolicies_DNSVisibilityCompanion, TestDNSVisibility_OneKubeDNSRule), exclude (TestGenerate_ExcludePeers, TestExclude_WholeLabelSetAndSingleLabel), merge (TestMergeDocument_AddsOnlyWhatIsMissing, _KeepsTheDocumentAsItWas, _KeepsOrderCommentsAndStyle, _PortIntEqualsPortString, _RefusesAnotherTarget, _CreatesTheMissingList), hardening (TestCORS_AllowedOrigins, TestAuthToken, TestAuthToken_SchemeCaseAndLengths, TestGenerate_BodyLimit, _BadInputs), the Grafana contract (TestGenerate_GrafanaActionKeepsFlowUUIDAndServesDownload), descriptions (TestDescribe, TestDescribe_CutsLongLists).
  • Chart: helm lint clean; rendered with every option on and with none, the CiliumNetworkPolicy and the Secret checked.
  • Live, on Cilium 1.20.1 (kind, ClusterMesh of two clusters, Gateway API, kube-prometheus-stack, Loki): demos 26–35 in cilium-implementation-poc — every command recorded, every verdict read back from Hubble with the deciding policy's name, screenshots of the page, the Grafana action and the dashboards. The independent review record of the 0.6.0 set (two reviewers, 15 claims, every refutation re-measured) is docs/REVIEW_ENH-001.md.

5. Compatibility

  • 0.7.0 is a drop-in for a 0.6.3 deployment: the same flags and output shape, the new flags optional; regenerating a toFQDNs or DNS-visibility policy writes the resolver rule as 53/ANY instead of 53/UDP (an applied UDP rule stays valid). Two review passes by two reviewers: docs/REVIEW_ENH-003.md.

  • Single-cluster, L4-only output is byte-identical to main for one flow (the cluster label, L7 rules and the DNS rule appear only when the flow carries the data or the option is set).

  • The upstream dashboard's Download link keeps working: a single flow's cache id is still the flow UUID.

  • Nothing in the chart changes when the new values are left at their defaults.

…ad_url; accept many flows per request and merge them per workload; ?name=; chart externalURL/extraArgs/extraEnv; UI summary, copy, download

download_url was built from r.TLS and r.Host. Behind a TLS-terminating ingress or Gateway the listener
has no TLS, so the JSON answer said http://<host>/download/<id> for a host that only serves https
(measured behind Cilium Gateway API: Envoy forwards X-Forwarded-Proto: https, the answer said http).
baseURL() now takes, in order: --external-url (env CF2CNP_EXTERNAL_URL), RFC 7239 Forwarded
(proto=, host=), X-Forwarded-Proto / X-Forwarded-Host, then r.TLS and r.Host.

/generate accepts one flow, a JSON array, or one flow per line (hubble observe -o json), and
policies that select the same workload are merged into one object with one rule per peer: two flows
into the same service from two peers used to produce two objects with the same metadata.name, so the
second kubectl apply replaced the first; in file mode the second file overwrote the first. ?name=
names the (single) resulting policy. The JSON answer carries flows, policies and the yaml itself.

Chart 0.5.0: externalURL, extraArgs, extraEnv; the container args are rendered so a flag can be set.
The page: a summary of the pasted flows before generating, a policy-name field, Copy YAML /
Download YAML / Load example / Clear, and the apply hint.

Tests: internal/flow (single, NDJSON, array, errors, directory), internal/policy (merge, dedupe,
distinct workloads, name override, one file per workload, multi-document), internal/server (every
scheme case, the Grafana path and its cache id, many flows, ?name=, bad inputs, YOLO). Fixtures are
real Hubble 1.20 flows.
… host must be a bare host[:port], an empty object is not a flow, an 8 MiB body cap and server timeouts

baseURL applied X-Forwarded-* after Forwarded, so the de-facto headers silently won over the RFC 7239
ones the comment said came first; a host header carrying a path, userinfo, whitespace or CRLF went
into download_url verbatim. Now the order matches the comment and validHost ignores anything that is
not host[:port]. parseFlow returned a zero flow for {} / null / an object without "flow", which became
a policy with empty selectors; it refuses those. /generate reads at most 8 MiB (413 above) and the
server has read/write/header timeouts. Tests for each; 500 flows across 20 workloads → 20 policies,
140 rules, 0.09 s; the single-flow YAML stays byte-identical to main.
@ephico2real2

Copy link
Copy Markdown
Author

Second review pass on this branch (head 2eb0f6b), with the findings applied:

  • baseURL now honours the documented order — RFC 7239 Forwarded before X-Forwarded-* — and a forwarded host must be a bare host[:port]: a value with a path, userinfo, whitespace or CRLF is ignored rather than written into download_url (tests: 8 header cases).
  • ParseFlowsFromBytes refuses {}, null and objects without a flow — json.Decoder produced a zero flow for those without error, which became a policy with empty selectors.
  • /generate reads at most 8 MiB (413 above it, measured) and the server has header/read/write timeouts.

Measured against a build of main: the YAML for a single flow is byte-identical (three fixtures, same md5); 500 flows across 20 workloads, 7 peers and 3 ports → 20 policies, 140 rules, 45 KB, 0.09 s in one request, where main answers invalid character '{' after top-level value. Chart 0.5.0 renders identically to 0.4.0 at defaults except the version label and the args block reproducing the image's CMD.

Known limitation, deliberately not changed here: two workloads in one namespace that share app.kubernetes.io/name and differ only by app.kubernetes.io/component still produce two policies with the same metadata.name (they do not merge — their selectors differ — and kubectl apply of the second replaces the first). Folding the component into the name would change every generated name that carries that label; that is a naming decision for the maintainers rather than something to slip into this PR.

…abels are carried as metadata; release 0.5.0 from the fork

Kubernetes identifies an object by group, kind, namespace and name (docs: Object Names and IDs), and
the recommended labels distinguish workloads by name, instance and component (docs: Recommended
Labels). Two workloads that share app.kubernetes.io/name and differ by component or instance produced
two CiliumNetworkPolicies named alike; kubectl apply of the second replaced the first, silently. The
name now folds instance and component into it in that order (shop, shop-frontend, shop-blue-frontend),
so equal names imply equal selectors; a workload with only a name keeps the name it had. Every policy
carries app.kubernetes.io/managed-by=cf2cnp and the selector's name/instance/component labels, so
`kubectl get cnp -l app.kubernetes.io/name=shop` lists a workload's policies whatever they are named.

Chart appVersion 0.5.0 (the fork's image release); chart-releaser workflow publishing the chart to the
gh-pages branch and a GitHub release while the upstream PR is pending.
@ephico2real2

Copy link
Copy Markdown
Author

One more change on the branch (head bcf06c8), because the earlier "known limitation" is a correctness bug rather than a naming preference: Kubernetes identifies an object by group, kind, namespace and name (Object Names and IDs), and the recommended labels distinguish workloads by name, instance and component. Two workloads that share app.kubernetes.io/name and differ by component produced two policies named alike, and kubectl apply of the second replaced the first.

  • generatePolicyName now folds instance and component into the name in that order (shop, shop-frontend, shop-blue-frontend), so equal names imply equal selectors. A workload with only app.kubernetes.io/name keeps the name it always had (test).
  • Every policy carries app.kubernetes.io/managed-by: cf2cnp and the selector's name / instance / component labels, so kubectl get cnp -l app.kubernetes.io/name=shop lists a workload's policies whatever they are named. This is the one visible change to single-flow output versus main (two label lines); the spec is unchanged.
  • Chart appVersion is 0.5.0 on this branch because the fork releases its own image while this is pending; set it to whatever you release.

ephico2real2 added a commit to ephico2real2/hubble-observer that referenced this pull request Sep 13, 2026
… name, then instance and component (shop/frontend → shop/backend); chart and appVersion 0.5.1
… cluster is named by io.cilium.k8s.policy.cluster (since Cilium 1.19 a selector without it matches the local cluster only); a fixture from a measured payments@poc2 → api@poc1 flow
…ames also carries the kube-dns L7 DNS rule (the one every toFQDNs policy has), so the DNS proxy records names and the next generation writes toFQDNs; comment says so
…mes from REQUEST-type l7 records; the port then goes through the proxy; fixtures measured on the bank (a Gateway GET, a DNS request) and the derived reply side
…ows whose peer carries the label; the page lists every parsed peer as a checkbox and sends the unticked ones as excludes
…iliumNetworkPolicy (unstructured: every field survives, only missing rules are added, idempotent, refuses another target); README
…ist, any by default), --auth-token (bearer on /generate and /download, constant-time, preflight and /health open), the page keeps a token in sessionStorage; chart: cors.allowedOrigins, auth.token/existingSecret, networkPolicy (a CiliumNetworkPolicy for the pod)
…archives and checksums attached to the tag's release, for pipelines that run cf2cnp merge
…er cluster is named by io.cilium.k8s.policy.cluster in fromEndpoints/toEndpoints (since Cilium 1.19 a selector without it matches the local cluster only); the aggregator keys on the peer's cluster; fixture: a measured payments@poc2 → redis-0@poc1:6379 request; tests on it, on single-cluster fixtures (unchanged), and on two same-labelled peers in two clusters
…reach the pod as the host identity (measured: 26 of 35 flows into the cf2cnp pod), the Gateway as reserved:ingress
…y.cluster label when cluster_name is empty; the peer's cluster is named whenever it is known and differs from the local side
…e cluster domain twice) is cut to the real name; the measured fixture asked for accounts.bank.svc.cluster.local.bank.svc.cluster.local.
…en port: 80 and the generated port: "80" are one rule, not a duplicate
…na action arrives through the Gateway as reserved:ingress, and a selector for a pod in another namespace needs io.kubernetes.pod.namespace (documented)
…exactly (name, component, instance, then app, k8s-app, name, component, instance); a test reads the served page to keep them in sync
… the page sends name+component+instance as one comma-joined exclude that must match entirely, so unticking shop/frontend keeps shop/backend; a bare key=value still matches every peer carrying it
…he token compare hashes both sides before the constant-time compare, so length no longer leaks
…on — one port rule per port with rules, the rest together (an HTTP rule on 8080 no longer restricts 5432); the path regex allows an optional query string (Envoy matches the whole :path); DNS queries are kept exactly as observed — the search-list expansion included — because an L7 DNS policy denies every name it does not list
…icts with E1 resolved as the union; the test file rebuilt as develop + E1's tests + E2's tests
…ed on the E1+E2 code: the DNS rule uses L7Rules, the page sends l7 and dnsVisibility together
…es (E2), DNS visibility (E3), exclude peers (E4), merge (E5), hardening (E6), binaries (E10)
…r, comments, quoting, 2-space indentation — a pull request shows the added rules and nothing else; demo 32 finding), and a pod's own DNS lookups no longer leave kube-dns 53/UDP twice beside the DNS-visibility rule (demo 31 finding, #1)
@ephico2real2

ephico2real2 commented Sep 13, 2026 •

Copy link
Copy Markdown
Author

(Superseded: the PR description above now lists every feature, chart value, bug fix and test, each linked to the demo that proves it.)

…ct, peers as the selector names them (namespace and cluster when they differ), ports, L7 rules — instead of a generic sentence about namespaces (operator finding on the page)
@ephico2real2 ephico2real2 changed the title serve: correct download_url behind a TLS-terminating proxy (Forwarded / X-Forwarded-* / --external-url); many flows per request merged per workload; ?name=; chart externalURL cf2cnp 0.5.0 → 0.6.3: proxy-aware download_url, many flows per request, cluster-aware selectors, L7 rules, DNS visibility, exclude, merge, hardening, descriptions from the rules, release binaries Sep 13, 2026
… 1.20.1 CiliumNetworkPolicy CRD, field by field with the source each could have; the world-source mis-modelling read from the parser; three probes against github.com/cilium/cilium@v1.20.1's policy API (zero replace lines, 18 MB with the CRD file embedded, 94 MB if the k8s client is imported; Sanitize mutates and must run on a copy; the schema and Sanitize catch different errors); the design: Cilium's types behind one envelope, two validation layers, the supported spec printed and released
…habetical; the kustomize/kyaml canonical-order pattern; cf2cnp owns the field order, byte identity kept) and the DNS resolver rule derived from flows with kubernetes/openshift profiles (Cilium's DNS guide; OpenShift's dns-default on 5353)
…s are github.com/cilium/cilium/pkg/policy/api v1.20.1 behind one envelope (20 hand-written fields → the whole CiliumNetworkPolicy spec); output rendered in cf2cnp's field order (internal/render; sigs.k8s.io/yaml is alphabetical) and pinned by 14 golden cases from the PoC's demos, byte for byte against 0.6.3's answers (one deliberate change, recorded); Cilium's Sanitize on a deep copy of every generated and merged policy (the agent's own refusals, e.g. fromEndpoints+fromCIDR mixed, as a 400 / exit 1); the CRD of the pinned module embedded from its file (go generate, CI asserts identity; the k8s client packages stay out: 18 MB, not 94); cf2cnp version names the supported spec, cf2cnp validate runs the agent's checks on any file; fromCIDR for an external source with an address (0.6.x wrote fromEntities: [world]); the DNS resolver rule derived from the observed DNS flows with kubernetes / openshift profiles and --dns-resolver (Cilium's DNS guide: openshift-dns, no k8s-app label, 5353), on the API and the page; generate accepts a flow file; a CI workflow (vet, tests, CRD identity, kubectl-validate over the goldens, the dependency line); Renovate rule for the Cilium module; Go 1.26; forensics §2.5 corrected
…h a 400 (they were a 500 on the API); /128 for IPv6 addresses on both sides (0.6.x wrote /32 for every address); the Sanitize test asserts the words, not Cilium's field order (CI caught the reverse order on Linux)
…ath helpers ride along with Cilium's policy API via pkg/command) and adds a 40 MB size ceiling as the real check
…olver derivation narrowed, profiles 53/ANY, validate decodes documents and checks the name and specs, serve takes DNS defaults (chart dns.profile / dns.resolver), Renovate note

- parser: reserved:world-ipv4 / reserved:world-ipv6 (dual-stack, pkg/labels/cidr.go getWorldLabel) are world and the
  world entity; two fixtures and TestParse_DualStackWorldLabels (a dual-stack world peer parsed as nothing before)
- dns: looksLikeDNS takes k8s-app kube-dns / coredns / node-local-dns or the openshift-dns namespace, no longer every
  kube-system peer on 53; the kubernetes profile and a --dns-resolver without a protocol are ANY (Cilium's
  dns-matchname.yaml; the forensics §6 design said ANY, the code wrote UDP); derived resolvers keep the observed
  protocols — 14 goldens unchanged
- validate: DecodePolicyDocuments (the YAML decoder finds the document boundaries; "\n---" cut literal blocks);
  Validate refuses a metadata.name that is not a DNS-1123 subdomain and validates `specs` rule by rule; the envelope
  carries Specs
- serve --dns-profile / --dns-resolver (env CF2CNP_DNS_PROFILE / CF2CNP_DNS_RESOLVER), checked at start; the
  request's parameters win; its own variables (cobra resets a shared one to the last registered default — measured);
  chart values dns.profile / dns.resolver → args, and the chart's policy DNS egress follows the profile
  (networkPolicy.dnsEgress for a custom resolver), 53/ANY
- TestEmbeddedCRDVersionMatchesModule; renovate.json says hosted Renovate does not run postUpgradeTasks
…he test binary on the GitHub runner reports no build-info Deps (measured, CI run 34774484089)
…a, the CRD's protocol enum, no nodeSelector on a namespaced policy), the derived DNS resolver is ANY, specs[] render in the rule order

- Validate: the API server's ObjectMeta checks (k8s.io/apimachinery/pkg/api/validation: name and namespace DNS-1123,
  label keys and values, annotations; a namespaced policy without a namespace is checked with kubectl's default),
  the CRD's upper-case protocol enum on every ports[] of ingress/ingressDeny/egress/egressDeny (Sanitize upper-cases
  on its copy, the schema sees the file), and the agent's "rule cannot have NodeSelector" for a namespaced policy;
  `specs` rule by rule. TestValidate_MatchesAdmission (five documents accepted before, refused now; the clusterwide
  nodeSelector and the missing namespace still pass)
- DeriveDNSResolver writes ANY whatever the lookups used: a UDP-only rule denies the TCP retry of a truncated answer
  under default-deny egress; Cilium's examples write ANY. dropShadowedDNSRule drops the plain twin whatever its
  protocol. Two goldens regenerated (31-pos-fqdn, 31-pos-dns-visibility: protocol ANY, description ANY/53), the
  reason in the golden README; kubectl-validate passes on both. TestDerivedDNSResolver_IsANY
- render: "specs" shares the rule order with "spec" (specs[] fell to the lexicographic tail). TestRender_SpecsUseRuleOrder
- README, forensics §6 and the dns tests say ANY for derived resolvers too
…bel the parser kept (CoreDNS Helm chart: app.kubernetes.io/name=coredns), the chart refuses a policy that would cut the pod off from a named resolver, merge names why a specs-only document is refused

- dns: resolverIDLabels looks at k8s-app, app.kubernetes.io/name and app for kube-dns / coredns / node-local-dns;
  the rule names that label. extractLabels keeps the app.kubernetes.io/* labels and drops k8s-app beside them, so
  the first fix (k8s-app only) never derived a Helm-chart CoreDNS and fell back to a kube-dns rule selecting none of
  its pods. Fixture egress-pos-to-helm-coredns.json; TestDeriveDNSResolver_HelmCoreDNSLabels through the parser
- chart: `fail` when dns.resolver is set with networkPolicy.enabled and no networkPolicy.dnsEgress (measured: the
  else branch rendered kube-dns 53/ANY for a pod whose server names another resolver)
- merge: a `specs`-only existing document is refused with a message that says so (merge writes spec.* only);
  TestMergeDocument_SpecsOnlyIsRefusedByName
…ate already passed it: "unknown flag"), Validate refuses a wrong apiVersion or kind and a namespace on a clusterwide policy with the reason

- cmd: mergeCmd --l7 → WithL7, as generate; CI runs the template's two calls (generate, then merge --l7) and greps
  the http rule
- Validate: a decoded document must be cilium.io/v2 and a Cilium policy kind (customresource/validator.go refuses
  the apiVersion; a policy built in memory carries neither); a clusterwide policy with a namespace is refused with
  "Kubernetes discards it on a cluster-scoped object" (rest/meta.go clears it: the file would apply and mean
  something else than it says). Three cases added to TestValidate_MatchesAdmission
@ephico2real2 ephico2real2 changed the title cf2cnp 0.5.0 → 0.6.3: proxy-aware download_url, many flows per request, cluster-aware selectors, L7 rules, DNS visibility, exclude, merge, hardening, descriptions from the rules, release binaries cf2cnp 0.5.0 → 0.7.0: download_url behind a proxy, many flows per request, ClusterMesh-aware selectors, L7 rules, DNS visibility, review before generating, merge, descriptions from the rules — and Cilium's own policy types with the agent's validation Sep 13, 2026
…and review record; CI writes a job summary: tests by package, the 14 golden captures one by one, every golden against the CRD schema, and cf2cnp validate's count
@ephico2real2

Copy link
Copy Markdown
Author

Running-test captures. Beyond the per-demo transcripts linked above, cf2cnp runs on every CI run of the reference lab (a fresh two-cluster Cilium 1.20.1 ClusterMesh on kind, GitHub Actions), and the results are captured, newest first:

cf2cnp-specific: cf2cnp.png (the tool's output), grafana-policy-verdicts-cf2cnp-lab.png (the generated policy under audit / enforce), hubble-ui-cf2cnp-lab.png. These links always point at the latest run, so they stay current through merge.

ephico2real2 added a commit to ephico2real2/hubble-observer that referenced this pull request Sep 15, 2026
…ack#10, onzack#11, onzack#13 (via onzack#15) and onzack#14, the version bump, his wording

The fork's develop is now upstream 2.7.0 plus what it still needs ahead of upstream: the cf2cnp 0.7.0 dependency from the
fork's Helm repository (onzack/cf2cnp#3 pending), the reviewed dashboard panels (the open PR), and containerName with its
second-release example — which the maintainer removed on his side because the dashboard hard-codes the container name
(a1b6f47, "would broke the dashboard"); the proper fix, a dashboard variable for the container, follows as its own PR.
The vendored subchart archives and Chart.lock are gone like upstream's (`helm dependency build` fetches them; .gitignore).
@ephico2real2

ephico2real2 commented Sep 15, 2026 •

Copy link
Copy Markdown
Author

@R-Studio with hubble-observer 2.7.0 out, this is the other half: the branch is the fork's develop at cc99fcb — releases 0.5.0 → 0.7.0 — and it merges clean (main is an ancestor of the branch; 110 files, +19,975 −582, no conflicts).

Why it matters for the chart you just released: hubble-observer 2.7.0's cf2cnp dependency is oci://ghcr.io/onzack/helm-charts at "*", which resolves to 0.4.0 today. The dashboard's Generate action and the policy-verdicts dashboard's Loki row post many flows per request, use exclude=, l7=true and dnsVisibility=true, and read a download_url that is https:// behind the Gateway — all of that is this PR (0.5.0 onward). Until it is merged and a chart is published from here, the observer's own subchart cannot do what its dashboard offers; the reference lab pins the fork's 0.7.0 chart for that reason.

How it is tested:

  • the fork's CI on every push (unit tests by package, the 14 golden captures byte for byte, every golden against Cilium's embedded CRD, cf2cnp validate on all of them): https://github.com/ephico2real2/cf2cnp/actions
  • the reference lab's CI (a fresh two-cluster Cilium 1.20.1 ClusterMesh on kind) runs the whole loop on every run — raw flows captured from Hubble, policies generated through the API, cf2cnp validate on the 0.7.0 binary, kubectl apply --dry-run=server, the agent's Valid condition, applied, the applications re-tested (a 403 from the proxy for an unobserved path, toFQDNs dropping a name never seen, six policies for a five-namespace platform), and cf2cnp merge idempotent: https://github.com/ephico2real2/cilium-implementation-poc/actions/runs/34980519349 (the "cf2cnp chapters" step and the report on the run page)
  • the walkthroughs, every command recorded, one per feature: demos 26–35 in https://github.com/ephico2real2/cilium-implementation-poc/tree/main/demos; the reviews in docs/REVIEW_ENH-001.md and docs/REVIEW_ENH-003.md there

If you prefer smaller PRs, each feature is its own branch (enh/E1 … enh/E10) and I will open them one by one on request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

download_url says http:// behind a TLS-terminating proxy or Gateway (r.TLS is nil; X-Forwarded-Proto / Forwarded are not read)

1 participant