From ae3484278617c5268581b51f3458a6b150a7070a Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 15 Oct 2026 19:27:44 +0200 Subject: [PATCH 1/7] spec: issue-3628 serve gRPC and HTTP from one endpoint per service Add a design spec that compares the options to serve gRPC, gRPC-Web and HTTP from one port per service (control plane and Artifact CAS), and picks a connection-level split on the current gRPC port, with the legacy HTTP port behind a config flag for a deprecation period. Refs #3628 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 2d702692-c199-4c83-b691-9bb2db0657ac --- docs/specs/issue-3628-single-endpoint.md | 163 +++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 docs/specs/issue-3628-single-endpoint.md diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md new file mode 100644 index 000000000..1b0ff4e5b --- /dev/null +++ b/docs/specs/issue-3628-single-endpoint.md @@ -0,0 +1,163 @@ +--- +owner: migmartri +ticket: https://github.com/chainloop-dev/chainloop/issues/3628 +prd: +--- + + + +# Spec issue-3628: Serve gRPC and HTTP from one endpoint per service + +## Summary + +The control plane and the Artifact CAS will each serve gRPC, gRPC-Web, and HTTP on one port. Today each service has a gRPC port and an HTTP port. An installation thus needs two Services, two ingresses, two hostnames, and two TLS settings per service. The service will accept all connections on its current gRPC port. It will look at each new connection and give it to the gRPC server or to the HTTP server that runs today. The middleware, the authentication, and the native gRPC transport for large CAS streams do not change. The old HTTP port stays on by default for a deprecation period, behind a config flag. The Helm chart moves to one Service and one hostname per service. + +## Problem + +- Each service listens on two ports: gRPC on 9000 and HTTP on 8000 (9001 and 8001 for the CAS in local development). The Helm chart creates two Services, two ingresses, and two hostnames for each service. That is four ingresses and four hostnames for one installation. +- Operators must know which hostname carries which traffic: + - The CLI and the control plane use gRPC to reach the CAS. + - Browsers use HTTP for the OIDC login, the status endpoints, the referrer discovery, and the OpenAPI document. + - The web UI uses gRPC-Web, which the control plane serves on its HTTP port. + - CAS downloads in a browser use HTTP. +- TLS can only be set on the gRPC listener. The HTTP listener is always plain text, so operators must terminate TLS for HTTP somewhere else. +- An earlier attempt (#2642) solved this at the chart level and depended on Gateway API. The team closed it as stale. The problem stays. + +## Goals and Non-Goals + +- Goal: one port and one hostname per service, for all gRPC, gRPC-Web, and HTTP traffic. +- Goal: existing clients and configuration keep working without change during the deprecation period. +- Goal: large CAS uploads and downloads keep their current streaming behavior and throughput. +- Non-goal: move the services to another RPC framework (for example Connect). +- Non-goal: merge the Prometheus metrics port or the profiler port into the single port. These stay internal and separate. +- Non-goal: add Gateway API resources to the Helm chart. +- Non-goal: change the CLI's default endpoints or its configuration format. + +## Requirements + +### R-001: One port serves all protocols +On its single port, each service MUST serve gRPC (HTTP/2), gRPC-Web, and plain HTTP (HTTP/1.1 and HTTP/2). This includes every endpoint that the gRPC port and the HTTP port serve today. +- Done when: these clients work against only the single port: + - the CLI and the web UI + - the OIDC login and the status probes + - the referrer discovery and the OpenAPI document + - CAS browser downloads + +### R-002: Same behavior on both paths +A request on the single port MUST get the same authentication, authorization, validation, logging, tracing, and metrics as the same request on the legacy port. + +### R-003: CAS streaming parity +Through the single port, CAS uploads and downloads MUST stream with bounded memory. Their throughput MUST stay within 10% of the legacy ports for the same artifact. +- Done when: a benchmark uploads and downloads a 1 GB artifact through the single port and through the legacy ports. The implementation PR records the results. + +### R-004: One TLS setting per service +When the operator configures TLS for a service, the single port MUST serve gRPC, gRPC-Web, and HTTP over that TLS configuration. It MUST negotiate HTTP/2 and HTTP/1.1. + +### R-005: Existing configuration keeps working +An installation that does not change its configuration MUST keep working after the upgrade. The single port MUST be the current gRPC address, so existing CLI configurations and the control plane connection to the CAS work as before. + +### R-006: Legacy HTTP port behind a flag +A configuration flag MUST control the legacy HTTP port. During the deprecation period the flag MUST default to on. When the flag is off, the service MUST listen only on the single port, plus the internal metrics and profiler ports. When the legacy port is on, the service SHOULD log a deprecation warning at startup. + +### R-007: Health checks on the single port +The HTTP status endpoint (`/statusz`, including readiness) and the standard gRPC health service MUST be available on the single port. + +### R-008: One Service and one hostname in the Helm chart +The Helm chart MUST create one Service and one external hostname per service. Probes MUST target the single port. The install notes and the CAS download URL MUST use the single hostname. While the legacy flag is on, the chart MAY keep the legacy port on the Service. +- Done when: a default install with nginx ingress serves the CLI, the web UI, and CAS downloads through one hostname per service. + +## Constraints + +- Public, open-source repository: the design must work with common ingress controllers and with no chart-level proxy. +- Backward compatibility for one deprecation period: no client, CLI configuration, or chart value may break on upgrade. +- The CAS spills uploads to disk and sends staged downloads with the kernel copy path. The design must keep both. + +## Proposal + +### What the operator sees + +After the upgrade, nothing changes for the operator. The gRPC port now also answers HTTP and gRPC-Web, and the old HTTP port still works. The operator points one hostname at the gRPC port of each service. When the operator is ready, the operator sets the legacy flag to off, and the HTTP port closes. In a later release the flag defaults to off. A release after that deletes the legacy port. + +### How the single port works + +Each service opens one listener on its current gRPC address. A small classifier reads the start of each new connection, without consuming it: + +- An HTTP/2 connection whose first request has `content-type: application/grpc` goes to the native gRPC server. This server is the same one that runs today, with its middleware, interceptors, message size limits, and streaming transport. +- All other connections go to the HTTP server that runs today. These are HTTP/1.1, HTTP/2 without gRPC, and gRPC-Web. The HTTP server keeps its routes, its middleware, the gRPC-Web wrapper, and the CORS rules of the CAS. + +Both servers keep their own lifecycle. They read from the classifier instead of from their own port. When the operator configures TLS, the listener terminates TLS before the classifier, and it offers HTTP/2 and HTTP/1.1 through ALPN. One certificate then covers all protocols. + +Some proxies send gRPC and plain HTTP requests over one shared upstream connection. The classifier sees only the first request of such a connection. As a safety net, the HTTP server gives any gRPC request that it receives to the gRPC server. The gRPC-Web wrapper already works in this way. The request gets the right result but takes the slower path. The chart configures the supported ingress controllers so that they do not share connections (see Risks). + +CAS streams on the single port use the same native gRPC transport as today, so uploads keep their disk spill and their throughput (R-003). Browser downloads stay on HTTP/1.1 through the ingress, so the kernel copy path stays. + +```mermaid +flowchart LR + CLI[CLI / control plane
gRPC] --> L + UI[Web UI
gRPC-Web] --> L + B[Browser
OIDC, status, downloads] --> L + L["Single port
(optional TLS + ALPN)"] --> C{Classify
new connection} + C -- "HTTP/2 + application/grpc" --> G[Native gRPC server
today's middleware] + C -- "everything else" --> H[HTTP server
routes, gRPC-Web, CORS] + H -. "gRPC request on shared connection" .-> G + LH[Legacy HTTP port
flag, default on] --> H +``` + +### Configuration + +The current gRPC address becomes the single port, and its TLS settings now apply to all protocols. A new flag in the server configuration controls the legacy HTTP listener, which keeps its current address. The Helm chart exposes the same flag as a value for each service. + +### Helm chart + +- One Service per service. It exposes the single port, plus the legacy port while the flag is on. +- One hostname per service. Controllers that can speak HTTP/2 without TLS to the backend use one Ingress. nginx sets the backend protocol per Ingress object. For nginx, the chart renders a second Ingress on the same hostname. It routes the gRPC service path prefixes with the gRPC backend protocol. The body size settings that the CAS needs today move with it. +- These items use the single port and hostname: + - the probes and the install notes + - the CLI `config save` hint + - the control plane connection to the CAS + - the CAS download URL +- The current split values (the second Service and the second ingress) keep working during the deprecation period. The chart documentation marks them as deprecated. + +## Decision Record + +| ID | Decision | Choice | Why (and what we rejected) | Source | +|----|----------|--------|----------------------------|--------| +| D-001 | How to serve both protocols on one port | Split by connection in front of the current gRPC and HTTP servers | Keeps the whole middleware chain and the native gRPC transport for CAS streams, and needs the smallest change. Rejected: one HTTP handler that gives each request to the HTTP adapter of the gRPC server. The gRPC library documents that path as slower and with fewer features, which puts R-003 at risk. Rejected: a move to Connect, because it replaces every middleware and route, which is a large migration for the same result. | drafting | +| D-002 | Which port becomes the single endpoint | The current gRPC port | CLI configurations, the control plane connection to the CAS, and the existing TLS settings keep working. Rejected: the HTTP port, because every gRPC client and address must change, and TLS must be added to it. Rejected: a new port, because every client changes. | drafting | +| D-003 | Deprecation default | Legacy HTTP port on by default, then off by default, then deleted | No upgrade breaks. Operators move when they are ready. Rejected: legacy port opt-in from the first release, because it breaks installations that send HTTP traffic to the old port. | drafting | +| D-004 | Ingress shape | One Service and one hostname. A path split into two Ingress objects on the same host only where the controller needs it | nginx sets the backend protocol per Ingress object, so one object cannot carry both. Rejected: one Ingress only, because it excludes nginx, the most common controller. Rejected: adding Gateway API in this work, because it is a separate feature. | drafting | +| D-005 | Solve in the server or in the chart | In the server | Works with any ingress, any proxy, and outside Kubernetes. Rejected: a chart-level proxy or Gateway API routes (#2642), because they need extra infrastructure and leave two ports in the service. | drafting | +| D-006 | Metrics and profiler ports | Stay separate | They are internal and must not be exposed on the public hostname. | drafting | +| D-007 | Length of the deprecation period | The legacy port defaults to off three months after the single port ships. A release three months after that deletes it. The release notes announce each step. | Gives operators two release windows to move their ingress and probes. Rejected: wait for a major release, because the two-port cost stays for too long. | drafting | +| D-008 | TLS on the single port | When the operator configures TLS, all protocols on the single port use it. An HTTP ingress must use an HTTPS backend. The legacy HTTP port stays plain text during the deprecation period. | One certificate and one setting per service is the goal of this work. Rejected: accept plain text and TLS on the same port, because it adds a second classification step and weakens the TLS setting. | drafting | + +## Open Questions + +None open. D-007 and D-008 record the answers to the questions from drafting. + +## Milestones + +1. **Single port in the servers.** Both services serve all protocols on the gRPC port. The legacy flag exists and defaults to on. The CAS benchmark (R-003) passes. +2. **Helm chart.** One Service and one hostname per service. The split values are deprecated. +3. **Deprecation.** The flag defaults to off. A later release deletes the legacy port. + +## Risks + +| Risk | Mitigation | +|------|------------| +| A proxy shares one upstream connection between gRPC and HTTP requests. A plain HTTP request then lands on a connection that the classifier gave to gRPC. | The chart routes gRPC and HTTP to separate backend protocols on nginx, and keeps HTTP/1.1 for the HTTP path on other controllers. Document the requirement. The HTTP server safety net handles the reverse case. | +| A gRPC client waits for the server before it sends anything, so the classifier cannot read the first request. | The classifier sends the server's HTTP/2 settings frame first when it sees an HTTP/2 preface, then reads the first request. | +| A slow or idle client holds a connection before the classifier decides. | Put a short read deadline on classification only, and clear it when the classifier hands over the connection. Long CAS streams are not affected. | +| CAS throughput drops on the single port. | The benchmark in R-003 runs before release. gRPC streams use the native transport, so the expected result is no change. | From 7a5739e045c7262576aff250af0e4e0a06fdd690 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 15 Oct 2026 19:47:50 +0200 Subject: [PATCH 2/7] spec: issue-3628 require server-level middleware parity tests Refs #3628 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino --- docs/specs/issue-3628-single-endpoint.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 1b0ff4e5b..0f1fa65de 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -57,6 +57,13 @@ On its single port, each service MUST serve gRPC (HTTP/2), gRPC-Web, and plain H ### R-002: Same behavior on both paths A request on the single port MUST get the same authentication, authorization, validation, logging, tracing, and metrics as the same request on the legacy port. +- Done when: server-level tests send the same calls through the single port and through the legacy port, and get the same result. The calls cover: + - no token, a user token, an API token, and a robot account or attestation token + - CAS uploader and downloader tokens on streams and on the download endpoint + - a validation error and a suspended organization + - a gRPC-Web call + + These tests merge before the listener change, so that they first record the current behavior. ### R-003: CAS streaming parity Through the single port, CAS uploads and downloads MUST stream with bounded memory. Their throughput MUST stay within 10% of the legacy ports for the same artifact. @@ -160,4 +167,5 @@ None open. D-007 and D-008 record the answers to the questions from drafting. | A proxy shares one upstream connection between gRPC and HTTP requests. A plain HTTP request then lands on a connection that the classifier gave to gRPC. | The chart routes gRPC and HTTP to separate backend protocols on nginx, and keeps HTTP/1.1 for the HTTP path on other controllers. Document the requirement. The HTTP server safety net handles the reverse case. | | A gRPC client waits for the server before it sends anything, so the classifier cannot read the first request. | The classifier sends the server's HTTP/2 settings frame first when it sees an HTTP/2 preface, then reads the first request. | | A slow or idle client holds a connection before the classifier decides. | Put a short read deadline on classification only, and clear it when the classifier hands over the connection. Long CAS streams are not affected. | +| A middleware behaves differently on the single port, for example an authentication step that does not run. | The server-level tests in R-002 merge first and record the current behavior. The same tests then run against the single port. | | CAS throughput drops on the single port. | The benchmark in R-003 runs before release. gRPC streams use the native transport, so the expected result is no change. | From 19048e854d3113d7e4a3dfff31a1d5b12407e5ca Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 15 Oct 2026 19:52:55 +0200 Subject: [PATCH 3/7] spec: issue-3628 move to the current spec format Use one Decision Record entry per decision, give each Non-Goal its reason, order requirements by importance, and align the parity requirements with the current release instead of the legacy port. Refs #3628 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino --- docs/specs/issue-3628-single-endpoint.md | 116 +++++++++++++++++------ 1 file changed, 86 insertions(+), 30 deletions(-) diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 0f1fa65de..1062a85ce 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -10,9 +10,11 @@ AGENT INSTRUCTIONS. Keep this comment. Read it before you change this file. This spec records a design that humans reviewed and approved. Do not change it as part of other work. Find the status of the parent ticket (the `ticket:` field) before you change anything. If you cannot find the status, or you are not sure which case below applies, do not change the spec. Ask the human first. - Ticket in design, and the spec PR is open: change the spec only through the br:eng-spec skill (`--revise`). Never renumber or reuse an R-xxx or D-xxx ID. -- Ticket in progress: do not rewrite the requirements, the Proposal, or existing Decision Record rows. Record a changed decision as a new Decision Record row. The owner records the differences with the shipped work through br:eng-spec (`--done`). +- Ticket in progress: do not rewrite the requirements, the Proposal, or existing Decision Record entries. Record a changed decision as a new Decision Record entry. The owner records the differences with the shipped work through br:eng-spec (`--done`). - Ticket done or canceled: the spec is frozen. Do not change, delete, or strike through any text. The only allowed edits are typo fixes, broken links, and the "Superseded by" line under the title. For a design change, write a new spec with br:eng-spec that supersedes this one. +When you review this spec, or before you add scope to it, check three things. No requirement contradicts another requirement, a Non-Goal, a Constraint, or a Decision Record entry. The most important requirements come first. The spec solves one problem: when a part can ship or be decided on its own, propose a separate spec instead of extending this one. Report each issue to the human. Do not fix it by yourself. + Do not answer an open question or a review comment on this spec by yourself. The answer is a design decision, and a human makes it. You can propose an answer and discuss it with a human. Change the spec or reply only after the human agrees. When your code goes against this spec, say so in your PR. Do not edit the spec to match the code. @@ -22,7 +24,7 @@ When your code goes against this spec, say so in your PR. Do not edit the spec t ## Summary -The control plane and the Artifact CAS will each serve gRPC, gRPC-Web, and HTTP on one port. Today each service has a gRPC port and an HTTP port. An installation thus needs two Services, two ingresses, two hostnames, and two TLS settings per service. The service will accept all connections on its current gRPC port. It will look at each new connection and give it to the gRPC server or to the HTTP server that runs today. The middleware, the authentication, and the native gRPC transport for large CAS streams do not change. The old HTTP port stays on by default for a deprecation period, behind a config flag. The Helm chart moves to one Service and one hostname per service. +The control plane and the Artifact CAS will each serve gRPC, gRPC-Web, and HTTP on one port. Today each service has a gRPC port and an HTTP port. An installation thus needs two Services, two ingresses, two hostnames, and two TLS settings per service. The service will accept all connections on its current gRPC port. It will look at each new connection and give it to the gRPC server or to the HTTP server that runs today. The middleware, the authentication, and the native gRPC transport for large CAS streams do not change. The legacy HTTP port stays on by default for a deprecation period, behind a config flag. The Helm chart moves to one Service and one hostname per service. ## Problem @@ -40,10 +42,11 @@ The control plane and the Artifact CAS will each serve gRPC, gRPC-Web, and HTTP - Goal: one port and one hostname per service, for all gRPC, gRPC-Web, and HTTP traffic. - Goal: existing clients and configuration keep working without change during the deprecation period. - Goal: large CAS uploads and downloads keep their current streaming behavior and throughput. -- Non-goal: move the services to another RPC framework (for example Connect). -- Non-goal: merge the Prometheus metrics port or the profiler port into the single port. These stay internal and separate. -- Non-goal: add Gateway API resources to the Helm chart. -- Non-goal: change the CLI's default endpoints or its configuration format. +- Non-goal: move the services to another RPC framework (for example Connect). The single port does not need it (see D-001). +- Non-goal: merge the Prometheus metrics port or the profiler port into the single port. These ports are internal and must not be on the public hostname (see D-006). +- Non-goal: add Gateway API resources to the Helm chart. It is a separate chart feature, and the single port works without it (see D-004). +- Non-goal: change the CLI's default endpoints or its configuration format. The single port is the current gRPC address, so the CLI needs no change (see R-005). +- Non-goal: replace the gRPC-Web library of the control plane or change its origin policy. The single port keeps the current gRPC-Web wrapper with no change. Follow-up: a separate ticket covers the library replacement and a security review of its origin policy. ## Requirements @@ -55,34 +58,34 @@ On its single port, each service MUST serve gRPC (HTTP/2), gRPC-Web, and plain H - the referrer discovery and the OpenAPI document - CAS browser downloads -### R-002: Same behavior on both paths -A request on the single port MUST get the same authentication, authorization, validation, logging, tracing, and metrics as the same request on the legacy port. -- Done when: server-level tests send the same calls through the single port and through the legacy port, and get the same result. The calls cover: +### R-005: Existing configuration keeps working +An installation that does not change its configuration MUST keep working after the upgrade. The single port MUST be the current gRPC address, so existing CLI configurations and the control plane connection to the CAS work as before. + +### R-002: Same behavior as today +A request on the single port MUST get the same result as the same request gets today on the separate ports. This covers authentication, authorization, validation, logging, tracing, and metrics. +- Done when: server-level tests send the same calls through the single port and get the same result as today. The calls cover: - no token, a user token, an API token, and a robot account or attestation token - CAS uploader and downloader tokens on streams and on the download endpoint - a validation error and a suspended organization - a gRPC-Web call - These tests merge before the listener change, so that they first record the current behavior. + These tests merge before the listener change, so that they first record the current behavior. After the change, the HTTP calls also run against the legacy HTTP port. ### R-003: CAS streaming parity -Through the single port, CAS uploads and downloads MUST stream with bounded memory. Their throughput MUST stay within 10% of the legacy ports for the same artifact. -- Done when: a benchmark uploads and downloads a 1 GB artifact through the single port and through the legacy ports. The implementation PR records the results. +Through the single port, CAS uploads and downloads MUST stream with bounded memory. Their throughput MUST stay within 10% of the current release for the same artifact. +- Done when: a benchmark uploads and downloads a 1 GB artifact through the single port and through the current release. The implementation PR records the results. ### R-004: One TLS setting per service When the operator configures TLS for a service, the single port MUST serve gRPC, gRPC-Web, and HTTP over that TLS configuration. It MUST negotiate HTTP/2 and HTTP/1.1. -### R-005: Existing configuration keeps working -An installation that does not change its configuration MUST keep working after the upgrade. The single port MUST be the current gRPC address, so existing CLI configurations and the control plane connection to the CAS work as before. - ### R-006: Legacy HTTP port behind a flag -A configuration flag MUST control the legacy HTTP port. During the deprecation period the flag MUST default to on. When the flag is off, the service MUST listen only on the single port, plus the internal metrics and profiler ports. When the legacy port is on, the service SHOULD log a deprecation warning at startup. +A configuration flag MUST control the legacy HTTP port. During the deprecation period the flag MUST default to on. When the flag is off, the service MUST listen only on the single port, plus the internal metrics and profiler ports. When the legacy HTTP port is on, the service SHOULD log a deprecation warning at startup. ### R-007: Health checks on the single port The HTTP status endpoint (`/statusz`, including readiness) and the standard gRPC health service MUST be available on the single port. ### R-008: One Service and one hostname in the Helm chart -The Helm chart MUST create one Service and one external hostname per service. Probes MUST target the single port. The install notes and the CAS download URL MUST use the single hostname. While the legacy flag is on, the chart MAY keep the legacy port on the Service. +The Helm chart MUST create one Service and one external hostname per service. Probes MUST target the single port. The install notes and the CAS download URL MUST use the single hostname. While the legacy flag is on, the chart MAY keep the legacy HTTP port on the Service. - Done when: a default install with nginx ingress serves the CLI, the web UI, and CAS downloads through one hostname per service. ## Constraints @@ -95,7 +98,7 @@ The Helm chart MUST create one Service and one external hostname per service. Pr ### What the operator sees -After the upgrade, nothing changes for the operator. The gRPC port now also answers HTTP and gRPC-Web, and the old HTTP port still works. The operator points one hostname at the gRPC port of each service. When the operator is ready, the operator sets the legacy flag to off, and the HTTP port closes. In a later release the flag defaults to off. A release after that deletes the legacy port. +After the upgrade, nothing changes for the operator. The gRPC port now also answers HTTP and gRPC-Web, and the legacy HTTP port still works. The operator points one hostname at the gRPC port of each service. When the operator is ready, the operator sets the legacy flag to off, and the HTTP port closes. In a later release the flag defaults to off. A release after that deletes the legacy HTTP port. ### How the single port works @@ -128,7 +131,7 @@ The current gRPC address becomes the single port, and its TLS settings now apply ### Helm chart -- One Service per service. It exposes the single port, plus the legacy port while the flag is on. +- One Service per service. It exposes the single port, plus the legacy HTTP port while the flag is on. - One hostname per service. Controllers that can speak HTTP/2 without TLS to the backend use one Ingress. nginx sets the backend protocol per Ingress object. For nginx, the chart renders a second Ingress on the same hostname. It routes the gRPC service path prefixes with the gRPC backend protocol. The body size settings that the CAS needs today move with it. - These items use the single port and hostname: - the probes and the install notes @@ -139,16 +142,69 @@ The current gRPC address becomes the single port, and its TLS settings now apply ## Decision Record -| ID | Decision | Choice | Why (and what we rejected) | Source | -|----|----------|--------|----------------------------|--------| -| D-001 | How to serve both protocols on one port | Split by connection in front of the current gRPC and HTTP servers | Keeps the whole middleware chain and the native gRPC transport for CAS streams, and needs the smallest change. Rejected: one HTTP handler that gives each request to the HTTP adapter of the gRPC server. The gRPC library documents that path as slower and with fewer features, which puts R-003 at risk. Rejected: a move to Connect, because it replaces every middleware and route, which is a large migration for the same result. | drafting | -| D-002 | Which port becomes the single endpoint | The current gRPC port | CLI configurations, the control plane connection to the CAS, and the existing TLS settings keep working. Rejected: the HTTP port, because every gRPC client and address must change, and TLS must be added to it. Rejected: a new port, because every client changes. | drafting | -| D-003 | Deprecation default | Legacy HTTP port on by default, then off by default, then deleted | No upgrade breaks. Operators move when they are ready. Rejected: legacy port opt-in from the first release, because it breaks installations that send HTTP traffic to the old port. | drafting | -| D-004 | Ingress shape | One Service and one hostname. A path split into two Ingress objects on the same host only where the controller needs it | nginx sets the backend protocol per Ingress object, so one object cannot carry both. Rejected: one Ingress only, because it excludes nginx, the most common controller. Rejected: adding Gateway API in this work, because it is a separate feature. | drafting | -| D-005 | Solve in the server or in the chart | In the server | Works with any ingress, any proxy, and outside Kubernetes. Rejected: a chart-level proxy or Gateway API routes (#2642), because they need extra infrastructure and leave two ports in the service. | drafting | -| D-006 | Metrics and profiler ports | Stay separate | They are internal and must not be exposed on the public hostname. | drafting | -| D-007 | Length of the deprecation period | The legacy port defaults to off three months after the single port ships. A release three months after that deletes it. The release notes announce each step. | Gives operators two release windows to move their ingress and probes. Rejected: wait for a major release, because the two-port cost stays for too long. | drafting | -| D-008 | TLS on the single port | When the operator configures TLS, all protocols on the single port use it. An HTTP ingress must use an HTTPS backend. The legacy HTTP port stays plain text during the deprecation period. | One certificate and one setting per service is the goal of this work. Rejected: accept plain text and TLS on the same port, because it adds a second classification step and weakens the TLS setting. | drafting | +### D-001: How to serve both protocols on one port +The service splits by connection in front of the current gRPC and HTTP servers. +- Why: it keeps the whole middleware chain and the native gRPC transport for CAS streams, and it needs the smallest change. +- Alternatives considered: + - One HTTP handler that routes each request: gRPC calls then use the HTTP adapter of the gRPC server. The gRPC library documents that path as slower and with fewer features. That puts R-003 at risk. + - A move to Connect: it replaces every middleware and route, which is a large migration for the same result. +- Source: drafting + +### D-002: Which port becomes the single endpoint +The current gRPC port becomes the single port. +- Why: CLI configurations, the control plane connection to the CAS, and the existing TLS settings keep working. +- Alternatives considered: + - The HTTP port: every gRPC client and address must change, and TLS must be added to it. + - A new port: every client changes. +- Source: drafting + +### D-003: Deprecation default +The legacy HTTP port is on by default, then off by default, then deleted. +- Why: no upgrade breaks, and operators move when they are ready. +- Alternatives considered: + - Legacy HTTP port opt-in from the first release: it breaks installations that send HTTP traffic to that port. +- Source: drafting + +### D-004: Ingress shape +The chart uses one Service and one hostname per service. Only where the controller needs it, the chart splits paths into two Ingress objects on the same host. +- Why: nginx sets the backend protocol per Ingress object, so one object cannot carry both protocols. +- Alternatives considered: + - One Ingress only: it excludes nginx, the most common controller. + - Add Gateway API in this work: it is a separate feature. +- Source: drafting + +### D-005: Solve in the server or in the chart +The services solve it in the server. +- Why: it works with any ingress, any proxy, and outside Kubernetes. +- Alternatives considered: + - A chart-level proxy or Gateway API routes (#2642): they need extra infrastructure and leave two ports in the service. +- Source: drafting + +### D-006: Metrics and profiler ports +The metrics and profiler ports stay separate. +- Why: they are internal and must not be on the public hostname. +- Source: drafting + +### D-007: Length of the deprecation period +The legacy HTTP port defaults to off three months after the single port ships. A release three months after that deletes it. The release notes announce each step. +- Why: operators get two release windows to move their ingress and probes. +- Alternatives considered: + - Wait for a major release: the two-port cost stays for too long. +- Source: drafting + +### D-008: TLS on the single port +When the operator configures TLS, all protocols on the single port use it. An HTTP ingress must use an HTTPS backend. The legacy HTTP port stays plain text during the deprecation period. +- Why: one certificate and one setting per service is the goal of this work. +- Alternatives considered: + - Accept plain text and TLS on the same port: it adds a second classification step and weakens the TLS setting. +- Source: drafting + +### D-009: Helm chart in the same spec +The Helm chart change stays in this spec, as milestone 2. +- Why: the goal is one port and one hostname per service. The chart gives operators the single hostname. +- Alternatives considered: + - A separate spec for the chart: the server change alone does not delete the second hostname. +- Source: drafting ## Open Questions @@ -158,7 +214,7 @@ None open. D-007 and D-008 record the answers to the questions from drafting. 1. **Single port in the servers.** Both services serve all protocols on the gRPC port. The legacy flag exists and defaults to on. The CAS benchmark (R-003) passes. 2. **Helm chart.** One Service and one hostname per service. The split values are deprecated. -3. **Deprecation.** The flag defaults to off. A later release deletes the legacy port. +3. **Deprecation.** The flag defaults to off. A later release deletes the legacy HTTP port. ## Risks From 8b5daf012b87584b8aa6505bb40ae32015cce3af Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 15 Oct 2026 20:06:30 +0200 Subject: [PATCH 4/7] spec: issue-3628 state the connection classifier limits Match every gRPC content type, record that a connection that starts with gRPC carries only gRPC, list the supported proxy setups, and ask reviewers about proxies that share one upstream connection. Refs #3628 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino --- docs/specs/issue-3628-single-endpoint.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 1062a85ce..c6278e836 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -57,6 +57,7 @@ On its single port, each service MUST serve gRPC (HTTP/2), gRPC-Web, and plain H - the OIDC login and the status probes - the referrer discovery and the OpenAPI document - CAS browser downloads +- Done when: the same checks pass through nginx ingress and through Traefik ingress. ### R-005: Existing configuration keeps working An installation that does not change its configuration MUST keep working after the upgrade. The single port MUST be the current gRPC address, so existing CLI configurations and the control plane connection to the CAS work as before. @@ -104,12 +105,14 @@ After the upgrade, nothing changes for the operator. The gRPC port now also answ Each service opens one listener on its current gRPC address. A small classifier reads the start of each new connection, without consuming it: -- An HTTP/2 connection whose first request has `content-type: application/grpc` goes to the native gRPC server. This server is the same one that runs today, with its middleware, interceptors, message size limits, and streaming transport. +- An HTTP/2 connection whose first request has a content type that starts with `application/grpc` goes to the native gRPC server. This includes `application/grpc+proto` and content types with parameters. gRPC-Web (`application/grpc-web`) is not in this family and stays on the HTTP server. This server is the same one that runs today, with its middleware, interceptors, message size limits, and streaming transport. - All other connections go to the HTTP server that runs today. These are HTTP/1.1, HTTP/2 without gRPC, and gRPC-Web. The HTTP server keeps its routes, its middleware, the gRPC-Web wrapper, and the CORS rules of the CAS. Both servers keep their own lifecycle. They read from the classifier instead of from their own port. When the operator configures TLS, the listener terminates TLS before the classifier, and it offers HTTP/2 and HTTP/1.1 through ALPN. One certificate then covers all protocols. -Some proxies send gRPC and plain HTTP requests over one shared upstream connection. The classifier sees only the first request of such a connection. As a safety net, the HTTP server gives any gRPC request that it receives to the gRPC server. The gRPC-Web wrapper already works in this way. The request gets the right result but takes the slower path. The chart configures the supported ingress controllers so that they do not share connections (see Risks). +The classifier decides once per connection, from its first request. A connection that starts with gRPC carries only gRPC: the native gRPC server cannot serve a later plain HTTP request on it. The other direction works. When a gRPC request arrives on a connection that the HTTP server owns, the HTTP server gives it to the gRPC server. The gRPC-Web wrapper already works in this way. The request gets the right result but takes the slower path. + +Thus the single port supports clients and proxies that keep gRPC and plain HTTP on separate HTTP/2 connections. These include direct clients such as the CLI, nginx ingress, and Traefik with HTTP/1.1 for the HTTP path. A proxy that sends both on one shared upstream connection is not supported (see D-010 and Open Questions). CAS streams on the single port use the same native gRPC transport as today, so uploads keep their disk spill and their throughput (R-003). Browser downloads stay on HTTP/1.1 through the ingress, so the kernel copy path stays. @@ -206,9 +209,18 @@ The Helm chart change stays in this spec, as milestone 2. - A separate spec for the chart: the server change alone does not delete the second hostname. - Source: drafting +### D-010: Proxies that share one connection for gRPC and HTTP +The single port routes each connection once. It supports only clients and proxies that keep gRPC and plain HTTP on separate HTTP/2 connections. +- Why: a connection that the native gRPC server owns cannot pass single requests to the HTTP server. Native gRPC transport for CAS streams (R-003) needs the connection split of D-001. +- Alternatives considered: + - Route each request on one HTTP/2 server: it supports every proxy. But all gRPC calls then use the slower HTTP adapter of the gRPC server, which puts R-003 at risk. +- Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927) + ## Open Questions -None open. D-007 and D-008 record the answers to the questions from drafting. +- [ ] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Proposed: not in this spec. Until the legacy HTTP port is gone, those setups can keep using it. Revisit before the release that deletes it. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). + +D-007 and D-008 record the answers to the questions from drafting. ## Milestones @@ -220,7 +232,7 @@ None open. D-007 and D-008 record the answers to the questions from drafting. | Risk | Mitigation | |------|------------| -| A proxy shares one upstream connection between gRPC and HTTP requests. A plain HTTP request then lands on a connection that the classifier gave to gRPC. | The chart routes gRPC and HTTP to separate backend protocols on nginx, and keeps HTTP/1.1 for the HTTP path on other controllers. Document the requirement. The HTTP server safety net handles the reverse case. | +| A proxy shares one upstream connection between gRPC and HTTP requests, for example an Envoy-based proxy or a service mesh. A plain HTTP request then lands on a connection that the classifier gave to gRPC, and fails. | The chart configures nginx and Traefik to use separate upstream connections. The documentation lists the supported setups. The legacy HTTP port stays available during the deprecation period. The open question on mixing proxies gets an answer before the legacy HTTP port is deleted. | | A gRPC client waits for the server before it sends anything, so the classifier cannot read the first request. | The classifier sends the server's HTTP/2 settings frame first when it sees an HTTP/2 preface, then reads the first request. | | A slow or idle client holds a connection before the classifier decides. | Put a short read deadline on classification only, and clear it when the classifier hands over the connection. Long CAS streams are not affected. | | A middleware behaves differently on the single port, for example an authentication step that does not run. | The server-level tests in R-002 merge first and record the current behavior. The same tests then run against the single port. | From 007ed242d4306468743f81703a3b8b7e4fbaa9d8 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Sun, 18 Oct 2026 16:55:45 +0200 Subject: [PATCH 5/7] spec: issue-3628 serve gRPC-Web on the control plane only Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 6b8e2f07-e54d-4ea8-b17d-7c888c48d327 --- docs/specs/issue-3628-single-endpoint.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index c6278e836..2601a1d9a 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -9,22 +9,22 @@ AGENT INSTRUCTIONS. Keep this comment. Read it before you change this file. This spec records a design that humans reviewed and approved. Do not change it as part of other work. Find the status of the parent ticket (the `ticket:` field) before you change anything. If you cannot find the status, or you are not sure which case below applies, do not change the spec. Ask the human first. -- Ticket in design, and the spec PR is open: change the spec only through the br:eng-spec skill (`--revise`). Never renumber or reuse an R-xxx or D-xxx ID. -- Ticket in progress: do not rewrite the requirements, the Proposal, or existing Decision Record entries. Record a changed decision as a new Decision Record entry. The owner records the differences with the shipped work through br:eng-spec (`--done`). +- Ticket in design, and the spec PR is open: change the spec only through the br:eng-spec skill (`--revise`). The spec shows only the current design. Delete a dropped requirement, do not strike it through, and record why in a Decision Record entry that names its ID. Never renumber or reuse an R-xxx or D-xxx ID. +- Ticket in progress: never rewrite or delete existing text. When the build changes a requirement or a decision, supersede it in the implementation PR. Add an entry with the next free ID that starts "Supersedes D-xxx." (or R-xxx). Strike through the heading and text of the old entry, and add "Superseded by D-yyy." under its heading. Name both IDs in the PR body. When the Summary, the Problem, or the main flow becomes wrong, it is a redesign: stop, and ask the human for a new spec. The owner records other differences with the shipped work through br:eng-spec (`--done`). - Ticket done or canceled: the spec is frozen. Do not change, delete, or strike through any text. The only allowed edits are typo fixes, broken links, and the "Superseded by" line under the title. For a design change, write a new spec with br:eng-spec that supersedes this one. When you review this spec, or before you add scope to it, check three things. No requirement contradicts another requirement, a Non-Goal, a Constraint, or a Decision Record entry. The most important requirements come first. The spec solves one problem: when a part can ship or be decided on its own, propose a separate spec instead of extending this one. Report each issue to the human. Do not fix it by yourself. Do not answer an open question or a review comment on this spec by yourself. The answer is a design decision, and a human makes it. You can propose an answer and discuss it with a human. Change the spec or reply only after the human agrees. -When your code goes against this spec, say so in your PR. Do not edit the spec to match the code. +When your code goes against this spec, say so in your PR, and supersede the entry when the ticket is in progress. --> # Spec issue-3628: Serve gRPC and HTTP from one endpoint per service ## Summary -The control plane and the Artifact CAS will each serve gRPC, gRPC-Web, and HTTP on one port. Today each service has a gRPC port and an HTTP port. An installation thus needs two Services, two ingresses, two hostnames, and two TLS settings per service. The service will accept all connections on its current gRPC port. It will look at each new connection and give it to the gRPC server or to the HTTP server that runs today. The middleware, the authentication, and the native gRPC transport for large CAS streams do not change. The legacy HTTP port stays on by default for a deprecation period, behind a config flag. The Helm chart moves to one Service and one hostname per service. +The control plane and the Artifact CAS will each serve gRPC and HTTP on one port. The control plane also serves gRPC-Web on it, as it does today. Today each service has a gRPC port and an HTTP port. An installation thus needs two Services, two ingresses, two hostnames, and two TLS settings per service. The service will accept all connections on its current gRPC port. It will look at each new connection and give it to the gRPC server or to the HTTP server that runs today. The middleware, the authentication, and the native gRPC transport for large CAS streams do not change. The legacy HTTP port stays on by default for a deprecation period, behind a config flag. The Helm chart moves to one Service and one hostname per service. ## Problem @@ -51,7 +51,7 @@ The control plane and the Artifact CAS will each serve gRPC, gRPC-Web, and HTTP ## Requirements ### R-001: One port serves all protocols -On its single port, each service MUST serve gRPC (HTTP/2), gRPC-Web, and plain HTTP (HTTP/1.1 and HTTP/2). This includes every endpoint that the gRPC port and the HTTP port serve today. +On its single port, each service MUST serve gRPC (HTTP/2) and plain HTTP (HTTP/1.1 and HTTP/2). The control plane MUST also serve gRPC-Web. The CAS has no gRPC-Web today and does not get it (see D-011). This includes every endpoint that the gRPC port and the HTTP port serve today. - Done when: these clients work against only the single port: - the CLI and the web UI - the OIDC login and the status probes @@ -68,7 +68,7 @@ A request on the single port MUST get the same result as the same request gets t - no token, a user token, an API token, and a robot account or attestation token - CAS uploader and downloader tokens on streams and on the download endpoint - a validation error and a suspended organization - - a gRPC-Web call + - a gRPC-Web call to the control plane These tests merge before the listener change, so that they first record the current behavior. After the change, the HTTP calls also run against the legacy HTTP port. @@ -106,7 +106,7 @@ After the upgrade, nothing changes for the operator. The gRPC port now also answ Each service opens one listener on its current gRPC address. A small classifier reads the start of each new connection, without consuming it: - An HTTP/2 connection whose first request has a content type that starts with `application/grpc` goes to the native gRPC server. This includes `application/grpc+proto` and content types with parameters. gRPC-Web (`application/grpc-web`) is not in this family and stays on the HTTP server. This server is the same one that runs today, with its middleware, interceptors, message size limits, and streaming transport. -- All other connections go to the HTTP server that runs today. These are HTTP/1.1, HTTP/2 without gRPC, and gRPC-Web. The HTTP server keeps its routes, its middleware, the gRPC-Web wrapper, and the CORS rules of the CAS. +- All other connections go to the HTTP server that runs today. These are HTTP/1.1, HTTP/2 without gRPC, and gRPC-Web. The HTTP server keeps its routes and its middleware. In the control plane it also keeps the gRPC-Web wrapper. In the CAS it keeps the CORS rules for browser downloads. The CAS has no gRPC-Web wrapper, and this spec does not add one. Both servers keep their own lifecycle. They read from the classifier instead of from their own port. When the operator configures TLS, the listener terminates TLS before the classifier, and it offers HTTP/2 and HTTP/1.1 through ALPN. One certificate then covers all protocols. @@ -123,7 +123,7 @@ flowchart LR B[Browser
OIDC, status, downloads] --> L L["Single port
(optional TLS + ALPN)"] --> C{Classify
new connection} C -- "HTTP/2 + application/grpc" --> G[Native gRPC server
today's middleware] - C -- "everything else" --> H[HTTP server
routes, gRPC-Web, CORS] + C -- "everything else" --> H[HTTP server
routes, CORS
gRPC-Web: control plane only] H -. "gRPC request on shared connection" .-> G LH[Legacy HTTP port
flag, default on] --> H ``` @@ -216,6 +216,13 @@ The single port routes each connection once. It supports only clients and proxie - Route each request on one HTTP/2 server: it supports every proxy. But all gRPC calls then use the slower HTTP adapter of the gRPC server, which puts R-003 at risk. - Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927) +### D-011: gRPC-Web only on the control plane +The single port serves gRPC-Web on the control plane only. The CAS serves gRPC and HTTP. +- Why: the CAS has no gRPC-Web today, and no CAS client needs it. A CAS adapter would be a new feature with its own streaming, authentication, and CORS design, and this spec is about one port per service. +- Alternatives considered: + - Add a gRPC-Web adapter to the CAS: it adds scope that can ship on its own. A separate spec can cover it if a client needs it. +- Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4307471522) + ## Open Questions - [ ] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Proposed: not in this spec. Until the legacy HTTP port is gone, those setups can keep using it. Revisit before the release that deletes it. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). From 6e6101e5ce523bad7a1042b72d89a62a27d6f390 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Sun, 18 Oct 2026 20:50:47 +0200 Subject: [PATCH 6/7] spec: issue-3628 make the HTTP port the single endpoint Drops D-002 and adds D-012. The legacy listener is now the gRPC port. Refs #3628 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 89e29529-5cd0-4d75-851b-45db0313c6fa --- docs/specs/issue-3628-single-endpoint.md | 59 +++++++++++++----------- 1 file changed, 31 insertions(+), 28 deletions(-) diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 2601a1d9a..8b413277b 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -24,7 +24,7 @@ When your code goes against this spec, say so in your PR, and supersede the entr ## Summary -The control plane and the Artifact CAS will each serve gRPC and HTTP on one port. The control plane also serves gRPC-Web on it, as it does today. Today each service has a gRPC port and an HTTP port. An installation thus needs two Services, two ingresses, two hostnames, and two TLS settings per service. The service will accept all connections on its current gRPC port. It will look at each new connection and give it to the gRPC server or to the HTTP server that runs today. The middleware, the authentication, and the native gRPC transport for large CAS streams do not change. The legacy HTTP port stays on by default for a deprecation period, behind a config flag. The Helm chart moves to one Service and one hostname per service. +The control plane and the Artifact CAS will each serve gRPC and HTTP on one port. The control plane also serves gRPC-Web on it, as it does today. Today each service has a gRPC port and an HTTP port. An installation thus needs two Services, two ingresses, two hostnames, and two TLS settings per service. The service will accept all connections on its current HTTP port, the one that serves the human-facing hostname today. It will look at each new connection and give it to the gRPC server or to the HTTP server that runs today. The middleware, the authentication, and the native gRPC transport for large CAS streams do not change. The legacy gRPC port stays on by default for a deprecation period, behind a config flag. The Helm chart moves to one Service and one hostname per service. ## Problem @@ -45,7 +45,7 @@ The control plane and the Artifact CAS will each serve gRPC and HTTP on one port - Non-goal: move the services to another RPC framework (for example Connect). The single port does not need it (see D-001). - Non-goal: merge the Prometheus metrics port or the profiler port into the single port. These ports are internal and must not be on the public hostname (see D-006). - Non-goal: add Gateway API resources to the Helm chart. It is a separate chart feature, and the single port works without it (see D-004). -- Non-goal: change the CLI's default endpoints or its configuration format. The single port is the current gRPC address, so the CLI needs no change (see R-005). +- Non-goal: change the CLI's default endpoints or its configuration format. The CLI holds hostnames on the ingress port, so it needs no change (see R-005). - Non-goal: replace the gRPC-Web library of the control plane or change its origin policy. The single port keeps the current gRPC-Web wrapper with no change. Follow-up: a separate ticket covers the library replacement and a security review of its origin policy. ## Requirements @@ -60,7 +60,7 @@ On its single port, each service MUST serve gRPC (HTTP/2) and plain HTTP (HTTP/1 - Done when: the same checks pass through nginx ingress and through Traefik ingress. ### R-005: Existing configuration keeps working -An installation that does not change its configuration MUST keep working after the upgrade. The single port MUST be the current gRPC address, so existing CLI configurations and the control plane connection to the CAS work as before. +An installation that does not change its configuration MUST keep working after the upgrade. The legacy gRPC port keeps serving gRPC at its current address during the deprecation period. Existing CLI configurations and the control plane connection to the CAS hold hostnames on the ingress port, so they keep working when the ingress backend moves to the single port. A client that connects to the gRPC port directly keeps working until the legacy gRPC port closes. ### R-002: Same behavior as today A request on the single port MUST get the same result as the same request gets today on the separate ports. This covers authentication, authorization, validation, logging, tracing, and metrics. @@ -70,7 +70,7 @@ A request on the single port MUST get the same result as the same request gets t - a validation error and a suspended organization - a gRPC-Web call to the control plane - These tests merge before the listener change, so that they first record the current behavior. After the change, the HTTP calls also run against the legacy HTTP port. + These tests merge before the listener change, so that they first record the current behavior. After the change, the gRPC calls also run against the legacy gRPC port. ### R-003: CAS streaming parity Through the single port, CAS uploads and downloads MUST stream with bounded memory. Their throughput MUST stay within 10% of the current release for the same artifact. @@ -79,14 +79,14 @@ Through the single port, CAS uploads and downloads MUST stream with bounded memo ### R-004: One TLS setting per service When the operator configures TLS for a service, the single port MUST serve gRPC, gRPC-Web, and HTTP over that TLS configuration. It MUST negotiate HTTP/2 and HTTP/1.1. -### R-006: Legacy HTTP port behind a flag -A configuration flag MUST control the legacy HTTP port. During the deprecation period the flag MUST default to on. When the flag is off, the service MUST listen only on the single port, plus the internal metrics and profiler ports. When the legacy HTTP port is on, the service SHOULD log a deprecation warning at startup. +### R-006: Legacy gRPC port behind a flag +A configuration flag MUST control the legacy gRPC port. During the deprecation period the flag MUST default to on. When the flag is off, the service MUST listen only on the single port, plus the internal metrics and profiler ports. When the legacy gRPC port is on, the service SHOULD log a deprecation warning at startup. ### R-007: Health checks on the single port The HTTP status endpoint (`/statusz`, including readiness) and the standard gRPC health service MUST be available on the single port. ### R-008: One Service and one hostname in the Helm chart -The Helm chart MUST create one Service and one external hostname per service. Probes MUST target the single port. The install notes and the CAS download URL MUST use the single hostname. While the legacy flag is on, the chart MAY keep the legacy HTTP port on the Service. +The Helm chart MUST create one Service and one external hostname per service. Probes MUST target the single port. The install notes and the CAS download URL MUST use the single hostname. While the legacy flag is on, the chart MAY keep the legacy gRPC port on the Service. - Done when: a default install with nginx ingress serves the CLI, the web UI, and CAS downloads through one hostname per service. ## Constraints @@ -99,11 +99,11 @@ The Helm chart MUST create one Service and one external hostname per service. Pr ### What the operator sees -After the upgrade, nothing changes for the operator. The gRPC port now also answers HTTP and gRPC-Web, and the legacy HTTP port still works. The operator points one hostname at the gRPC port of each service. When the operator is ready, the operator sets the legacy flag to off, and the HTTP port closes. In a later release the flag defaults to off. A release after that deletes the legacy HTTP port. +After the upgrade, nothing changes for the operator. The HTTP port now also answers native gRPC, and the legacy gRPC port still works. The operator points one hostname at the HTTP port of each service. When the operator is ready, the operator sets the legacy flag to off, and the gRPC port closes. In a later release the flag defaults to off. A release after that deletes the legacy gRPC port. ### How the single port works -Each service opens one listener on its current gRPC address. A small classifier reads the start of each new connection, without consuming it: +Each service opens one listener on its current HTTP address. A small classifier reads the start of each new connection, without consuming it: - An HTTP/2 connection whose first request has a content type that starts with `application/grpc` goes to the native gRPC server. This includes `application/grpc+proto` and content types with parameters. gRPC-Web (`application/grpc-web`) is not in this family and stays on the HTTP server. This server is the same one that runs today, with its middleware, interceptors, message size limits, and streaming transport. - All other connections go to the HTTP server that runs today. These are HTTP/1.1, HTTP/2 without gRPC, and gRPC-Web. The HTTP server keeps its routes and its middleware. In the control plane it also keeps the gRPC-Web wrapper. In the CAS it keeps the CORS rules for browser downloads. The CAS has no gRPC-Web wrapper, and this spec does not add one. @@ -125,16 +125,16 @@ flowchart LR C -- "HTTP/2 + application/grpc" --> G[Native gRPC server
today's middleware] C -- "everything else" --> H[HTTP server
routes, CORS
gRPC-Web: control plane only] H -. "gRPC request on shared connection" .-> G - LH[Legacy HTTP port
flag, default on] --> H + LG[Legacy gRPC port
flag, default on] --> G ``` ### Configuration -The current gRPC address becomes the single port, and its TLS settings now apply to all protocols. A new flag in the server configuration controls the legacy HTTP listener, which keeps its current address. The Helm chart exposes the same flag as a value for each service. +The current HTTP address becomes the single port. It gets its own TLS settings, which apply to all protocols. The existing gRPC TLS settings stay with the legacy gRPC listener. A new flag in the server configuration controls the legacy gRPC listener, which keeps its current address. The Helm chart exposes the same flag as a value for each service. ### Helm chart -- One Service per service. It exposes the single port, plus the legacy HTTP port while the flag is on. +- One Service per service. It exposes the single port, plus the legacy gRPC port while the flag is on. - One hostname per service. Controllers that can speak HTTP/2 without TLS to the backend use one Ingress. nginx sets the backend protocol per Ingress object. For nginx, the chart renders a second Ingress on the same hostname. It routes the gRPC service path prefixes with the gRPC backend protocol. The body size settings that the CAS needs today move with it. - These items use the single port and hostname: - the probes and the install notes @@ -153,19 +153,11 @@ The service splits by connection in front of the current gRPC and HTTP servers. - A move to Connect: it replaces every middleware and route, which is a large migration for the same result. - Source: drafting -### D-002: Which port becomes the single endpoint -The current gRPC port becomes the single port. -- Why: CLI configurations, the control plane connection to the CAS, and the existing TLS settings keep working. -- Alternatives considered: - - The HTTP port: every gRPC client and address must change, and TLS must be added to it. - - A new port: every client changes. -- Source: drafting - ### D-003: Deprecation default -The legacy HTTP port is on by default, then off by default, then deleted. +The legacy gRPC port is on by default, then off by default, then deleted. - Why: no upgrade breaks, and operators move when they are ready. - Alternatives considered: - - Legacy HTTP port opt-in from the first release: it breaks installations that send HTTP traffic to that port. + - Legacy gRPC port opt-in from the first release: it breaks clients that connect to that port directly. - Source: drafting ### D-004: Ingress shape @@ -189,14 +181,14 @@ The metrics and profiler ports stay separate. - Source: drafting ### D-007: Length of the deprecation period -The legacy HTTP port defaults to off three months after the single port ships. A release three months after that deletes it. The release notes announce each step. +The legacy gRPC port defaults to off three months after the single port ships. A release three months after that deletes it. The release notes announce each step. - Why: operators get two release windows to move their ingress and probes. - Alternatives considered: - Wait for a major release: the two-port cost stays for too long. - Source: drafting ### D-008: TLS on the single port -When the operator configures TLS, all protocols on the single port use it. An HTTP ingress must use an HTTPS backend. The legacy HTTP port stays plain text during the deprecation period. +When the operator configures TLS, all protocols on the single port use it. An HTTP ingress must use an HTTPS backend. The legacy gRPC port keeps its own TLS settings during the deprecation period. - Why: one certificate and one setting per service is the goal of this work. - Alternatives considered: - Accept plain text and TLS on the same port: it adds a second classification step and weakens the TLS setting. @@ -223,23 +215,34 @@ The single port serves gRPC-Web on the control plane only. The CAS serves gRPC a - Add a gRPC-Web adapter to the CAS: it adds scope that can ship on its own. A separate spec can cover it if a client needs it. - Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4307471522) +### D-012: The current HTTP port is the single endpoint +Drops D-002. The current HTTP port becomes the single port, not the gRPC port. The legacy listener is the gRPC port. +- Why: the human-facing hostname, with the status, download, and login endpoints, is already on the HTTP port. It becomes the one endpoint, with one URL, one certificate, and one TLS setting. +- Why: clients do not break in either direction. CLI configurations and the control plane connection to the CAS hold hostnames on the ingress port. During the deprecation period the legacy gRPC port keeps serving. After it closes, the gRPC ingress changes only its backend port. +- Why: the ingress and chart work is the same size in both variants, so it does not decide the question. +- Cost: TLS settings move to the HTTP port, and the deprecation window applies to the gRPC port. +- Alternatives considered: + - The gRPC port (the dropped D-002): it keeps TLS settings in place, but the human-facing hostname stays on a second port. + - A new port: every client changes. +- Source: [PR comment](https://github.com/chainloop-dev/chainloop/pull/3629#issuecomment-6253559302) + ## Open Questions -- [ ] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Proposed: not in this spec. Until the legacy HTTP port is gone, those setups can keep using it. Revisit before the release that deletes it. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). +- [ ] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Proposed: not in this spec. Until the legacy gRPC port is gone, those setups can route gRPC to it and HTTP to the single port. Revisit before the release that deletes it. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). D-007 and D-008 record the answers to the questions from drafting. ## Milestones -1. **Single port in the servers.** Both services serve all protocols on the gRPC port. The legacy flag exists and defaults to on. The CAS benchmark (R-003) passes. +1. **Single port in the servers.** Both services serve all protocols on the HTTP port. The legacy gRPC flag exists and defaults to on. The CAS benchmark (R-003) passes. 2. **Helm chart.** One Service and one hostname per service. The split values are deprecated. -3. **Deprecation.** The flag defaults to off. A later release deletes the legacy HTTP port. +3. **Deprecation.** The flag defaults to off. A later release deletes the legacy gRPC port. ## Risks | Risk | Mitigation | |------|------------| -| A proxy shares one upstream connection between gRPC and HTTP requests, for example an Envoy-based proxy or a service mesh. A plain HTTP request then lands on a connection that the classifier gave to gRPC, and fails. | The chart configures nginx and Traefik to use separate upstream connections. The documentation lists the supported setups. The legacy HTTP port stays available during the deprecation period. The open question on mixing proxies gets an answer before the legacy HTTP port is deleted. | +| A proxy shares one upstream connection between gRPC and HTTP requests, for example an Envoy-based proxy or a service mesh. A plain HTTP request then lands on a connection that the classifier gave to gRPC, and fails. | The chart configures nginx and Traefik to use separate upstream connections. The documentation lists the supported setups. The legacy gRPC port stays available during the deprecation period. The open question on mixing proxies gets an answer before the legacy gRPC port is deleted. | | A gRPC client waits for the server before it sends anything, so the classifier cannot read the first request. | The classifier sends the server's HTTP/2 settings frame first when it sees an HTTP/2 preface, then reads the first request. | | A slow or idle client holds a connection before the classifier decides. | Put a short read deadline on classification only, and clear it when the classifier hands over the connection. Long CAS streams are not affected. | | A middleware behaves differently on the single port, for example an authentication step that does not run. | The server-level tests in R-002 merge first and record the current behavior. The same tests then run against the single port. | From c9ccbbb0c565954c4d06b9f7ff6937a942e8ee70 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Sun, 18 Oct 2026 21:06:19 +0200 Subject: [PATCH 7/7] spec: issue-3628 name the migration of direct gRPC clients Refs #3628 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 89e29529-5cd0-4d75-851b-45db0313c6fa --- docs/specs/issue-3628-single-endpoint.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 8b413277b..e54b24764 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -60,7 +60,8 @@ On its single port, each service MUST serve gRPC (HTTP/2) and plain HTTP (HTTP/1 - Done when: the same checks pass through nginx ingress and through Traefik ingress. ### R-005: Existing configuration keeps working -An installation that does not change its configuration MUST keep working after the upgrade. The legacy gRPC port keeps serving gRPC at its current address during the deprecation period. Existing CLI configurations and the control plane connection to the CAS hold hostnames on the ingress port, so they keep working when the ingress backend moves to the single port. A client that connects to the gRPC port directly keeps working until the legacy gRPC port closes. +An installation that does not change its configuration MUST keep working during the deprecation period. The legacy gRPC port keeps serving gRPC at its current address during that period. CLI configurations that use ingress hostnames keep working when the ingress backend moves to the single port. The control plane connection to the CAS, and any other client that uses the legacy gRPC address, MUST move to the single HTTP address before the legacy gRPC port is disabled. +- Done when: the release notes name this migration step for the CAS connection and for direct gRPC clients. ### R-002: Same behavior as today A request on the single port MUST get the same result as the same request gets today on the separate ports. This covers authentication, authorization, validation, logging, tracing, and metrics. @@ -99,7 +100,7 @@ The Helm chart MUST create one Service and one external hostname per service. Pr ### What the operator sees -After the upgrade, nothing changes for the operator. The HTTP port now also answers native gRPC, and the legacy gRPC port still works. The operator points one hostname at the HTTP port of each service. When the operator is ready, the operator sets the legacy flag to off, and the gRPC port closes. In a later release the flag defaults to off. A release after that deletes the legacy gRPC port. +After the upgrade, nothing changes for the operator. The HTTP port now also answers native gRPC, and the legacy gRPC port still works. The operator points one hostname at the HTTP port of each service. Before the operator sets the legacy flag to off, the operator moves the control plane connection to the CAS, and any other client that uses the gRPC address directly, to the single HTTP address. Then the operator sets the flag to off, and the gRPC port closes. In a later release the flag defaults to off. A release after that deletes the legacy gRPC port. ### How the single port works @@ -218,7 +219,7 @@ The single port serves gRPC-Web on the control plane only. The CAS serves gRPC a ### D-012: The current HTTP port is the single endpoint Drops D-002. The current HTTP port becomes the single port, not the gRPC port. The legacy listener is the gRPC port. - Why: the human-facing hostname, with the status, download, and login endpoints, is already on the HTTP port. It becomes the one endpoint, with one URL, one certificate, and one TLS setting. -- Why: clients do not break in either direction. CLI configurations and the control plane connection to the CAS hold hostnames on the ingress port. During the deprecation period the legacy gRPC port keeps serving. After it closes, the gRPC ingress changes only its backend port. +- Why: clients that use ingress hostnames do not break in either direction. During the deprecation period the legacy gRPC port keeps serving. After it closes, the gRPC ingress changes only its backend port. Direct clients, such as the control plane connection to the CAS, move to the single HTTP address (see R-005). - Why: the ingress and chart work is the same size in both variants, so it does not decide the question. - Cost: TLS settings move to the HTTP port, and the deprecation window applies to the gRPC port. - Alternatives considered: