Repository navigation
cf2cnp 0.5.0 → 0.7.0: download_url behind a proxy, many flows per request, ClusterMesh-aware selectors, L7 rules, DNS visibility, review before generating, merge, descriptions from the rules — and Cilium's own policy types with the agent's validation - #3
ephico2real2 wants to merge 44 commits into
Conversation
…ad_url; accept many flows per request and merge them per workload; ?name=; chart externalURL/extraArgs/extraEnv; UI summary, copy, download download_url was built from r.TLS and r.Host. Behind a TLS-terminating ingress or Gateway the listener has no TLS, so the JSON answer said http://<host>/download/<id> for a host that only serves https (measured behind Cilium Gateway API: Envoy forwards X-Forwarded-Proto: https, the answer said http). baseURL() now takes, in order: --external-url (env CF2CNP_EXTERNAL_URL), RFC 7239 Forwarded (proto=, host=), X-Forwarded-Proto / X-Forwarded-Host, then r.TLS and r.Host. /generate accepts one flow, a JSON array, or one flow per line (hubble observe -o json), and policies that select the same workload are merged into one object with one rule per peer: two flows into the same service from two peers used to produce two objects with the same metadata.name, so the second kubectl apply replaced the first; in file mode the second file overwrote the first. ?name= names the (single) resulting policy. The JSON answer carries flows, policies and the yaml itself. Chart 0.5.0: externalURL, extraArgs, extraEnv; the container args are rendered so a flag can be set. The page: a summary of the pasted flows before generating, a policy-name field, Copy YAML / Download YAML / Load example / Clear, and the apply hint. Tests: internal/flow (single, NDJSON, array, errors, directory), internal/policy (merge, dedupe, distinct workloads, name override, one file per workload, multi-document), internal/server (every scheme case, the Grafana path and its cache id, many flows, ?name=, bad inputs, YOLO). Fixtures are real Hubble 1.20 flows.
… host must be a bare host[:port], an empty object is not a flow, an 8 MiB body cap and server timeouts
baseURL applied X-Forwarded-* after Forwarded, so the de-facto headers silently won over the RFC 7239
ones the comment said came first; a host header carrying a path, userinfo, whitespace or CRLF went
into download_url verbatim. Now the order matches the comment and validHost ignores anything that is
not host[:port]. parseFlow returned a zero flow for {} / null / an object without "flow", which became
a policy with empty selectors; it refuses those. /generate reads at most 8 MiB (413 above) and the
server has read/write/header timeouts. Tests for each; 500 flows across 20 workloads → 20 policies,
140 rules, 0.09 s; the single-flow YAML stays byte-identical to main.
|
Second review pass on this branch (head 2eb0f6b), with the findings applied:
Measured against a build of Known limitation, deliberately not changed here: two workloads in one namespace that share |
…abels are carried as metadata; release 0.5.0 from the fork Kubernetes identifies an object by group, kind, namespace and name (docs: Object Names and IDs), and the recommended labels distinguish workloads by name, instance and component (docs: Recommended Labels). Two workloads that share app.kubernetes.io/name and differ by component or instance produced two CiliumNetworkPolicies named alike; kubectl apply of the second replaced the first, silently. The name now folds instance and component into it in that order (shop, shop-frontend, shop-blue-frontend), so equal names imply equal selectors; a workload with only a name keeps the name it had. Every policy carries app.kubernetes.io/managed-by=cf2cnp and the selector's name/instance/component labels, so `kubectl get cnp -l app.kubernetes.io/name=shop` lists a workload's policies whatever they are named. Chart appVersion 0.5.0 (the fork's image release); chart-releaser workflow publishing the chart to the gh-pages branch and a GitHub release while the upstream PR is pending.
|
One more change on the branch (head bcf06c8), because the earlier "known limitation" is a correctness bug rather than a naming preference: Kubernetes identifies an object by group, kind, namespace and name (Object Names and IDs), and the recommended labels distinguish workloads by
|
… name, then instance and component (shop/frontend → shop/backend); chart and appVersion 0.5.1
… cluster is named by io.cilium.k8s.policy.cluster (since Cilium 1.19 a selector without it matches the local cluster only); a fixture from a measured payments@poc2 → api@poc1 flow
…ames also carries the kube-dns L7 DNS rule (the one every toFQDNs policy has), so the DNS proxy records names and the next generation writes toFQDNs; comment says so
…mes from REQUEST-type l7 records; the port then goes through the proxy; fixtures measured on the bank (a Gateway GET, a DNS request) and the derived reply side
…ows whose peer carries the label; the page lists every parsed peer as a checkbox and sends the unticked ones as excludes
…iliumNetworkPolicy (unstructured: every field survives, only missing rules are added, idempotent, refuses another target); README
…ist, any by default), --auth-token (bearer on /generate and /download, constant-time, preflight and /health open), the page keeps a token in sessionStorage; chart: cors.allowedOrigins, auth.token/existingSecret, networkPolicy (a CiliumNetworkPolicy for the pod)
…archives and checksums attached to the tag's release, for pipelines that run cf2cnp merge
…er cluster is named by io.cilium.k8s.policy.cluster in fromEndpoints/toEndpoints (since Cilium 1.19 a selector without it matches the local cluster only); the aggregator keys on the peer's cluster; fixture: a measured payments@poc2 → redis-0@poc1:6379 request; tests on it, on single-cluster fixtures (unchanged), and on two same-labelled peers in two clusters
…reach the pod as the host identity (measured: 26 of 35 flows into the cf2cnp pod), the Gateway as reserved:ingress
…y.cluster label when cluster_name is empty; the peer's cluster is named whenever it is known and differs from the local side
…e cluster domain twice) is cut to the real name; the measured fixture asked for accounts.bank.svc.cluster.local.bank.svc.cluster.local.
…en port: 80 and the generated port: "80" are one rule, not a duplicate
…na action arrives through the Gateway as reserved:ingress, and a selector for a pod in another namespace needs io.kubernetes.pod.namespace (documented)
…exactly (name, component, instance, then app, k8s-app, name, component, instance); a test reads the served page to keep them in sync
… the page sends name+component+instance as one comma-joined exclude that must match entirely, so unticking shop/frontend keeps shop/backend; a bare key=value still matches every peer carrying it
…he token compare hashes both sides before the constant-time compare, so length no longer leaks
…on — one port rule per port with rules, the rest together (an HTTP rule on 8080 no longer restricts 5432); the path regex allows an optional query string (Envoy matches the whole :path); DNS queries are kept exactly as observed — the search-list expansion included — because an L7 DNS policy denies every name it does not list
…icts with E1 resolved as the union; the test file rebuilt as develop + E1's tests + E2's tests
…ed on the E1+E2 code: the DNS rule uses L7Rules, the page sends l7 and dnsVisibility together
…es (E2), DNS visibility (E3), exclude peers (E4), merge (E5), hardening (E6), binaries (E10)
…r, comments, quoting, 2-space indentation — a pull request shows the added rules and nothing else; demo 32 finding), and a pod's own DNS lookups no longer leave kube-dns 53/UDP twice beside the DNS-visibility rule (demo 31 finding, #1)
|
(Superseded: the PR description above now lists every feature, chart value, bug fix and test, each linked to the demo that proves it.) |
…ct, peers as the selector names them (namespace and cluster when they differ), ports, L7 rules — instead of a generic sentence about namespaces (operator finding on the page)
…name/component/instance from the selector)
… 1.20.1 CiliumNetworkPolicy CRD, field by field with the source each could have; the world-source mis-modelling read from the parser; three probes against github.com/cilium/cilium@v1.20.1's policy API (zero replace lines, 18 MB with the CRD file embedded, 94 MB if the k8s client is imported; Sanitize mutates and must run on a copy; the schema and Sanitize catch different errors); the design: Cilium's types behind one envelope, two validation layers, the supported spec printed and released
…habetical; the kustomize/kyaml canonical-order pattern; cf2cnp owns the field order, byte identity kept) and the DNS resolver rule derived from flows with kubernetes/openshift profiles (Cilium's DNS guide; OpenShift's dns-default on 5353)
…s are github.com/cilium/cilium/pkg/policy/api v1.20.1 behind one envelope (20 hand-written fields → the whole CiliumNetworkPolicy spec); output rendered in cf2cnp's field order (internal/render; sigs.k8s.io/yaml is alphabetical) and pinned by 14 golden cases from the PoC's demos, byte for byte against 0.6.3's answers (one deliberate change, recorded); Cilium's Sanitize on a deep copy of every generated and merged policy (the agent's own refusals, e.g. fromEndpoints+fromCIDR mixed, as a 400 / exit 1); the CRD of the pinned module embedded from its file (go generate, CI asserts identity; the k8s client packages stay out: 18 MB, not 94); cf2cnp version names the supported spec, cf2cnp validate runs the agent's checks on any file; fromCIDR for an external source with an address (0.6.x wrote fromEntities: [world]); the DNS resolver rule derived from the observed DNS flows with kubernetes / openshift profiles and --dns-resolver (Cilium's DNS guide: openshift-dns, no k8s-app label, 5353), on the API and the page; generate accepts a flow file; a CI workflow (vet, tests, CRD identity, kubectl-validate over the goldens, the dependency line); Renovate rule for the Cilium module; Go 1.26; forensics §2.5 corrected
…h a 400 (they were a 500 on the API); /128 for IPv6 addresses on both sides (0.6.x wrote /32 for every address); the Sanitize test asserts the words, not Cilium's field order (CI caught the reverse order on Linux)
…ath helpers ride along with Cilium's policy API via pkg/command) and adds a 40 MB size ceiling as the real check
…olver derivation narrowed, profiles 53/ANY, validate decodes documents and checks the name and specs, serve takes DNS defaults (chart dns.profile / dns.resolver), Renovate note - parser: reserved:world-ipv4 / reserved:world-ipv6 (dual-stack, pkg/labels/cidr.go getWorldLabel) are world and the world entity; two fixtures and TestParse_DualStackWorldLabels (a dual-stack world peer parsed as nothing before) - dns: looksLikeDNS takes k8s-app kube-dns / coredns / node-local-dns or the openshift-dns namespace, no longer every kube-system peer on 53; the kubernetes profile and a --dns-resolver without a protocol are ANY (Cilium's dns-matchname.yaml; the forensics §6 design said ANY, the code wrote UDP); derived resolvers keep the observed protocols — 14 goldens unchanged - validate: DecodePolicyDocuments (the YAML decoder finds the document boundaries; "\n---" cut literal blocks); Validate refuses a metadata.name that is not a DNS-1123 subdomain and validates `specs` rule by rule; the envelope carries Specs - serve --dns-profile / --dns-resolver (env CF2CNP_DNS_PROFILE / CF2CNP_DNS_RESOLVER), checked at start; the request's parameters win; its own variables (cobra resets a shared one to the last registered default — measured); chart values dns.profile / dns.resolver → args, and the chart's policy DNS egress follows the profile (networkPolicy.dnsEgress for a custom resolver), 53/ANY - TestEmbeddedCRDVersionMatchesModule; renovate.json says hosted Renovate does not run postUpgradeTasks
…he test binary on the GitHub runner reports no build-info Deps (measured, CI run 34774484089)
…a, the CRD's protocol enum, no nodeSelector on a namespaced policy), the derived DNS resolver is ANY, specs[] render in the rule order - Validate: the API server's ObjectMeta checks (k8s.io/apimachinery/pkg/api/validation: name and namespace DNS-1123, label keys and values, annotations; a namespaced policy without a namespace is checked with kubectl's default), the CRD's upper-case protocol enum on every ports[] of ingress/ingressDeny/egress/egressDeny (Sanitize upper-cases on its copy, the schema sees the file), and the agent's "rule cannot have NodeSelector" for a namespaced policy; `specs` rule by rule. TestValidate_MatchesAdmission (five documents accepted before, refused now; the clusterwide nodeSelector and the missing namespace still pass) - DeriveDNSResolver writes ANY whatever the lookups used: a UDP-only rule denies the TCP retry of a truncated answer under default-deny egress; Cilium's examples write ANY. dropShadowedDNSRule drops the plain twin whatever its protocol. Two goldens regenerated (31-pos-fqdn, 31-pos-dns-visibility: protocol ANY, description ANY/53), the reason in the golden README; kubectl-validate passes on both. TestDerivedDNSResolver_IsANY - render: "specs" shares the rule order with "spec" (specs[] fell to the lexicographic tail). TestRender_SpecsUseRuleOrder - README, forensics §6 and the dns tests say ANY for derived resolvers too
…bel the parser kept (CoreDNS Helm chart: app.kubernetes.io/name=coredns), the chart refuses a policy that would cut the pod off from a named resolver, merge names why a specs-only document is refused - dns: resolverIDLabels looks at k8s-app, app.kubernetes.io/name and app for kube-dns / coredns / node-local-dns; the rule names that label. extractLabels keeps the app.kubernetes.io/* labels and drops k8s-app beside them, so the first fix (k8s-app only) never derived a Helm-chart CoreDNS and fell back to a kube-dns rule selecting none of its pods. Fixture egress-pos-to-helm-coredns.json; TestDeriveDNSResolver_HelmCoreDNSLabels through the parser - chart: `fail` when dns.resolver is set with networkPolicy.enabled and no networkPolicy.dnsEgress (measured: the else branch rendered kube-dns 53/ANY for a pod whose server names another resolver) - merge: a `specs`-only existing document is refused with a message that says so (merge writes spec.* only); TestMergeDocument_SpecsOnlyIsRefusedByName
…ate already passed it: "unknown flag"), Validate refuses a wrong apiVersion or kind and a namespace on a clusterwide policy with the reason - cmd: mergeCmd --l7 → WithL7, as generate; CI runs the template's two calls (generate, then merge --l7) and greps the http rule - Validate: a decoded document must be cilium.io/v2 and a Cilium policy kind (customresource/validator.go refuses the apiVersion; a policy built in memory carries neither); a clusterwide policy with a namespace is refused with "Kubernetes discards it on a cluster-scoped object" (rest/meta.go clears it: the file would apply and mean something else than it says). Three cases added to TestValidate_MatchesAdmission
…and review record; CI writes a job summary: tests by package, the 14 golden captures one by one, every golden against the CRD schema, and cf2cnp validate's count
|
Running-test captures. Beyond the per-demo transcripts linked above, cf2cnp runs on every CI run of the reference lab (a fresh two-cluster Cilium 1.20.1 ClusterMesh on kind, GitHub Actions), and the results are captured, newest first:
cf2cnp-specific: |
…enerated-policy verdicts)
…ack#10, onzack#11, onzack#13 (via onzack#15) and onzack#14, the version bump, his wording The fork's develop is now upstream 2.7.0 plus what it still needs ahead of upstream: the cf2cnp 0.7.0 dependency from the fork's Helm repository (onzack/cf2cnp#3 pending), the reviewed dashboard panels (the open PR), and containerName with its second-release example — which the maintainer removed on his side because the dashboard hard-codes the container name (a1b6f47, "would broke the dashboard"); the proper fix, a dashboard variable for the container, follows as its own PR. The vendored subchart archives and Chart.lock are gone like upstream's (`helm dependency build` fetches them; .gitignore).
|
@R-Studio with hubble-observer 2.7.0 out, this is the other half: the branch is the fork's Why it matters for the chart you just released: hubble-observer 2.7.0's How it is tested:
If you prefer smaller PRs, each feature is its own branch ( |
Closes #2.
@R-Studio
This PR is the fork's
developbranch (https://github.com/ephico2real2/cf2cnp/tree/develop), released as 0.5.0 → 0.7.0. Everything in it was built for a Cilium 1.20.1 ClusterMesh and proven there; each row below links to the recorded demo (transcript, Hubble verdicts, screenshots) that exercises it. If you prefer smaller PRs, every feature is also its own branch (enh/E1…enh/E10) and I will open them one by one on request.1. Features
download_urlbehind a TLS-terminating proxy (the bug in #2)--external-url/CF2CNP_EXTERNAL_URL, or the proxy'sForwarded/X-Forwarded-Proto/X-Forwarded-Hostdownload_urlishttps://…behind an HTTPS gateway instead ofhttp://. Precedence: the flag, then RFC 7239Forwarded, thenX-Forwarded-*, thenr.TLS/r.Host; a garbage proto is ignoredhubble observe -o jsonas is)shop,shop-frontend,shop-blue-frontend); two components of one app no longer produce two objects with one name thatkubectl applywould replaceapp.kubernetes.io/managed-by: cf2cnpplus the selector's labels, sokubectl get cnp -l …finds them?name=<name>io.cilium.k8s.policy.cluster: <cluster>. Since Cilium 1.19 a selector without it matches the local cluster only, so the old output dropped the real cross-cluster caller and admitted a same-labelled local pod--l7,?l7=true, the page's boxl7: one port rule per port that has L7 records, paths anchored and escaped with an optional query string (Envoy matches the whole:path), DNS names kept exactly as observed--dns-visibility,?dnsVisibility=true, the page's boxmatchPattern: "*") to a world policy whose flows carry no names, so the next flows carrydestination_namesand the next generation writestoFQDNsinstead of a CIDR?exclude=<label>=<value>(repeatable; a comma-joined set must match entirely), the page's peer checklistcf2cnp mergecf2cnp merge --existing policy.yaml --input flows.ndjson [-o out.yaml] [--l7]Allow ingress to shop/frontend in cf2cnp-lab27: from pos on TCP/80 (HTTP GET /, GET /checkout); from kiosk on TCP/80— subject and peers as the selector names them, a peer's namespace or cluster when it differs, ports, L7 rules; cut with "and N more" past six peers/kubectl applyhint. No backticks in the script (the page is a Go raw string)Accept: application/jsonorX-Grafana-Actionflows,policiesandyaml;download_url,filename,messageunchanged, and a single flow's cache id is still its UUID, so the hubble-observer dashboard's Download link keeps workingv*taglinux/darwin×amd64/arm64archives and a checksums file, for pipelines that runcf2cnp mergewithout a Go toolchain| Cilium's own policy types (0.7.0) | automatic | The document is
github.com/cilium/cilium/pkg/policy/apiat the pinned Cilium version (v1.20.1): every field of the CiliumNetworkPolicy spec exists, the agent's ownSanitize()runs on every generated policy (on a copy, so the file stays as a user writes it), the CRD of the same version is embedded and checked byte for byte in CI, andcf2cnp versionprints the spec version the build supports. Output is byte-identical to 0.6.3 for the 14 recorded captures except two deliberate changes (below) | demo 35 Part 4 regenerated; docs/CRD-SPEC-FORENSICS.md ||
cf2cnp validate(0.7.0) |cf2cnp validate policy.yaml …| Every document of every file through the agent's checks plus the object's:Sanitize, ObjectMeta (DNS-1123 names, label keys), the CRD's protocol enum,apiVersion/kind, nonodeSelectoron a namespaced policy, a namespace on a clusterwide one,specs; exit = the number refused | review record ||
fromCIDRfor external sources (0.7.0) | automatic | An ingress flow fromreserved:world(orworld-ipv4/world-ipv6on dual-stack) with an address becomesfromCIDR: [<addr>/32](/128IPv6) — what the receiver saw — instead offromEntities: [world]| docs/CRD-SPEC-FORENSICS.md §2.5 || The DNS resolver rule from the flows (0.7.0) |
--dns-profile auto\|kubernetes\|openshift,--dns-resolver <ns>[/<label>=<v>]:<port>[/<proto>],?dnsProfile=,?dnsResolver=,serveand chart defaults (dns.profile,dns.resolver) | The ruletoFQDNsand--dns-visibilityneed names the resolver the workload actually asked (kube-dns byk8s-app, the CoreDNS Helm chart byapp.kubernetes.io/name, NodeLocal DNSCache, OpenShift'sopenshift-dnson 5353 with no label — Cilium's DNS guide), protocolANYas Cilium's examples write it (a truncated UDP answer retries over TCP) | review record ||
merge --l7(0.7.0) |cf2cnp merge … --l7| The flag the policy-PR template already passed | review record |2. Chart (0.4.0 → 0.7.0)
externalURL""--external-urlfor the podextraArgs,extraEnv[]serve --port <containerPort>; the container'sargsare now rendered, so a flag can be given at allcors.allowedOrigins[](any origin, as before)Access-Control-Allow-Origin(withVary: Origin); a foreign origin gets no CORS headerauth.token,auth.existingSecret""/generateand/download— never on the page,/healthor preflight; compared in constant time after hashing, so length does not leak; scheme case-insensitivedns.profile,dns.resolver,networkPolicy.dnsEgress(0.7.0)auto,"",{}dns.profile: openshiftonce); the chart's own policy's DNS egress follows the profile, and refuses to render when a custom resolver is named without a matchingdnsEgressnetworkPolicy.enabled,networkPolicy.fromEndpointsfalse,[]reserved:ingress(a Gateway) andreserved:host(the kubelet's probes) on the container port, plus the listed endpoints; egress to kube-dns only3. Bug fixes
download_urlsaidhttp://behind an HTTPS proxy (r.TLSis nil after TLS termination)metadata.name; in file mode the second file overwrote the first (TestGeneratePolicies_OneFilePerWorkloadfails onmain)app.kubernetes.io/nameproduced one policy name — the secondkubectl applyreplaced the first{}body (no flow) was accepted and produced an empty policyForwardedhost was used unsanitised?id=42would have been blocked once enforcedmergere-serialised the whole file (alphabetical keys, 4-space indentation): a PR diff was the file, not the rule53/UDPtwice in one policyAllow ingress traffic from ns to ns for the shop)/32reserved:world-ipv4/-ipv6) was neither world nor an entity: an egress to it produced no peer at allk8s-app: kube-dns, 53/UDP): on OpenShift and with the CoreDNS Helm chart it selected nothing and cut the pod off from DNS once enforced; UDP alone denied the TCP retry of a truncated answermergehad no--l7although the policy-PR template passed it (unknown flag)4. Tests
internal/testdata/golden/README.md; the Sanitize, admission, dual-stack, DNS-resolver and document-decoding cases),go test ./...green at every tag; CI also checks the embedded CRD byte for byte against the pinned module, validates every golden withkubectl-validate, keeps the Kubernetes client machinery out of the binary (40 MB ceiling), and runs the policy-PR template's two calls. By area: parsing (TestParseFlowsFromBytes_*,_RejectsNonFlows,ParseFlowsFromDirectory_ManyPerFile), the proxy URL (TestDownloadURL_PrecedenceAndHostSanity,_Scheme,TestYolo_JSONUsesBaseURL), naming and merging across flows (TestBuildPolicies_NameIsAFunctionOfTheSelector,_MergesSameWorkload,_KeepsDistinctWorkloads,_DedupesIdenticalRules,TestGeneratePolicies_OneFilePerWorkload), ClusterMesh (TestBuildPolicies_CrossClusterPeerNamesItsCluster,_SameClusterHasNoClusterLabel,_SameLabelsTwoClustersTwoRules,_ClusterLabelWhenClusterNameEmpty), L7 (TestBuildPolicies_L7HTTPRules,_L7DNSRules,_L7RulesArePerPort,TestL7Rules_PathIsEscapedAndAllowsAQuery,_KeepsTheQueryAsObserved), DNS visibility (TestBuildPolicies_DNSVisibilityCompanion,TestDNSVisibility_OneKubeDNSRule), exclude (TestGenerate_ExcludePeers,TestExclude_WholeLabelSetAndSingleLabel), merge (TestMergeDocument_AddsOnlyWhatIsMissing,_KeepsTheDocumentAsItWas,_KeepsOrderCommentsAndStyle,_PortIntEqualsPortString,_RefusesAnotherTarget,_CreatesTheMissingList), hardening (TestCORS_AllowedOrigins,TestAuthToken,TestAuthToken_SchemeCaseAndLengths,TestGenerate_BodyLimit,_BadInputs), the Grafana contract (TestGenerate_GrafanaActionKeepsFlowUUIDAndServesDownload), descriptions (TestDescribe,TestDescribe_CutsLongLists).helm lintclean; rendered with every option on and with none, the CiliumNetworkPolicy and the Secret checked.5. Compatibility
0.7.0 is a drop-in for a 0.6.3 deployment: the same flags and output shape, the new flags optional; regenerating a
toFQDNsor DNS-visibility policy writes the resolver rule as53/ANYinstead of53/UDP(an applied UDP rule stays valid). Two review passes by two reviewers: docs/REVIEW_ENH-003.md.Single-cluster, L4-only output is byte-identical to
mainfor one flow (the cluster label, L7 rules and the DNS rule appear only when the flow carries the data or the option is set).The upstream dashboard's Download link keeps working: a single flow's cache id is still the flow UUID.
Nothing in the chart changes when the new values are left at their defaults.