From 6564d8a3c13936876e67d11219ab190d000a401b Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Mon, 19 Oct 2026 20:40:23 +0200 Subject: [PATCH 01/19] feat(cas): add shared endpoint experiment Serve native gRPC and HTTP per request on the CAS HTTP listener while preserving the native endpoint and legacy ingress settings. Add authenticated ByteStream checks, local testing instructions, and a 1 GiB transport comparison that records the HTTP adapter upload memory regression. Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/README.md | 59 ++ app/artifact-cas/cmd/main.go | 15 +- app/artifact-cas/cmd/wire_gen.go | 18 +- app/artifact-cas/configs/config.devel.yaml | 3 + app/artifact-cas/configs/samples/config.yaml | 8 +- .../experiments/shared-endpoint-benchmark.txt | 37 ++ app/artifact-cas/internal/conf/conf.pb.go | 58 +- app/artifact-cas/internal/conf/conf.proto | 6 +- app/artifact-cas/internal/server/grpc.go | 106 +++- app/artifact-cas/internal/server/http.go | 74 ++- app/artifact-cas/internal/server/http_test.go | 579 ++++++++++++++++++ app/artifact-cas/internal/server/server.go | 4 +- .../internal/server/streaming_bench_test.go | 149 +++++ deployment/chainloop/Chart.yaml | 2 +- deployment/chainloop/README.md | 6 +- deployment/chainloop/chart_test.go | 19 +- .../chainloop/templates/cas/ingress.yaml | 9 +- deployment/chainloop/values.yaml | 21 +- 18 files changed, 1090 insertions(+), 83 deletions(-) create mode 100644 app/artifact-cas/experiments/shared-endpoint-benchmark.txt create mode 100644 app/artifact-cas/internal/server/http_test.go create mode 100644 app/artifact-cas/internal/server/streaming_bench_test.go diff --git a/app/artifact-cas/README.md b/app/artifact-cas/README.md index cfc0bc8eb..6237d5b9e 100644 --- a/app/artifact-cas/README.md +++ b/app/artifact-cas/README.md @@ -67,6 +67,65 @@ go run main.go --insecure artifact upload -f [FILE] # will return a sha256 diges go run main.go --insecure artifact download -d [sha256:...] ``` + +### Shared endpoint experiment + +This branch extends the control-plane HTTP dispatch approach to CAS for local evaluation. The existing HTTP port, `8001`, accepts HTTP/1.1, HTTP/2, and native gRPC. Every request is dispatched independently: HTTP/2 requests with a native gRPC content type use `grpc.Server.ServeHTTP`; other requests use the existing HTTP status and authenticated download handlers. CAS does not add gRPC-Web or Connect. + +The native gRPC listener on `9001` remains enabled by default, with its existing independent TLS configuration. The same ByteStream and resource services, JWT checks, role checks, validators, and metrics interceptors serve both listeners. `server.http.timeout` continues to apply to HTTP routes; ByteStream requests bypass the HTTP route middleware. HTTP drains before its gRPC handler stops. + +This is an experiment against the original CAS design in [issue-3628-single-endpoint](../../docs/specs/issue-3628-single-endpoint.md), particularly its D-014 through D-016 native-transport design. The control-plane replacement spec defers CAS. Neither spec is amended by this branch: a CAS transport decision and the original R-002/R-003 qualification remain required before migration. The HTTP adapter can read ahead while the application is paused, so functional streaming tests alone cannot establish bounded memory. + +To test with your existing development infrastructure: + +```sh +make -C app/artifact-cas run + +# In another terminal, use a per-command CAS endpoint override. +# Existing CLI authentication and the control-plane endpoint come from your config. +go run ./app/cli --insecure --artifact-cas localhost:8001 artifact upload -f /path/to/artifact +# Use the returned digest: +go run ./app/cli --insecure --artifact-cas localhost:8001 artifact download -d sha256:DIGEST +``` + +The development CAS public key and credentials service must match the control plane that issues these temporary CAS tokens. If another CAS already occupies these ports, start this binary with a copy of your CAS configuration and unused addresses. Do not replace the running service or edit the CLI config just to run a comparison. + +Change only the per-command override to `localhost:9001` to compare with the native listener. Once all direct clients and the control-plane CAS connection have been moved, `server.legacy_grpc_listener: false` disables the native listener. Leave it enabled during evaluation. Optional `server.http.tls_config` configures TLS for the shared endpoint; both certificate and private key must be set. It does not inherit `server.grpc.tls_config`. + +The chart's `cas.ingress` targets the shared HTTP Service with HTTP/2 upstream defaults for ingress-nginx, F5 NGINX, and Traefik, matching the control-plane settings. `cas.ingressAPI` and `cas.serviceAPI` retain their existing settings. The control plane still targets the legacy CAS Service. The shared ingress retains the native ingress's `100m` body limit and `3M` client body buffer default; override the body limit to `0` or a suitable larger limit for a 1 GB ingress experiment. Deployed controller qualification remains outstanding. + +Run the isolated functional checks without Vault or an object store: + +```sh +go test -race ./app/artifact-cas/internal/server ./app/artifact-cas/cmd ./deployment/chainloop -count=1 +``` + +Compare real ByteStream upload and download paths over TCP with a synthetic 1 GiB zero-filled artifact, streamed from a fixed-size buffer and checked by SHA256. Both transports use the same file-backed test backend and real CAS staging and authentication. Select a disk-backed temporary directory; `/tmp` can be tmpfs. + +```sh +mkdir -p "$HOME/.cache/chainloop-cas-bench" +TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/server \ + -run '^$' -bench '^BenchmarkCASByteStreamTransport$' -benchtime=1x -count=3 -timeout=10m +``` + +`CAS_BENCH_BYTES` changes the artifact size for shorter experiments. Benchmark throughput and total allocated bytes come from Go's benchmark runner. `peak-heap-MiB` samples live heap every 10 ms for the combined client and server process; it is neither server-only RSS nor a bounded-memory proof. This controlled transport comparison does not establish the original R-003 requirement of throughput within 10% of the current release with representative object storage, or cover slow readers, concurrent streams, ingress, TLS throughput, or the complete R-002 authentication matrix. + + +#### Observed local results + +Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, Linux amd64, disk-backed btrfs staging. These medians come from the [raw benchmark output](experiments/shared-endpoint-benchmark.txt). + +| Transport | Operation | Throughput (MB/s) | Sampled peak live heap (MiB) | +| --- | --- | ---: | ---: | +| Native gRPC | Upload | 240.84 | 27.97 | +| HTTP adapter | Upload | 232.00 | 1278.00 | +| Native gRPC | Download | 197.03 | 31.97 | +| HTTP adapter | Download | 173.83 | 38.92 | + +The HTTP-adapter upload used approximately 1.25 GiB of combined client/server heap for a 1 GiB artifact, while the native upload stayed around 28 MiB. This is a material memory regression even with an actively reading application. The sampled peak includes pooled and queued buffers and does not identify every allocation's owner. Source inspection is consistent with server read-ahead: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) and queues data into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90). + +Median upload throughput was 3.7% lower and download throughput 11.8% lower on the adapter in this run. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production. The memory result prevents recommending this shared CAS adapter for migration. The native endpoint remains the comparison baseline; use this branch to reproduce the result and evaluate another transport before changing production CAS clients. + ### Run tests ``` diff --git a/app/artifact-cas/cmd/main.go b/app/artifact-cas/cmd/main.go index 1e7aa4a69..c9f1ece79 100644 --- a/app/artifact-cas/cmd/main.go +++ b/app/artifact-cas/cmd/main.go @@ -37,8 +37,7 @@ import ( "github.com/go-kratos/kratos/v2/config/env" "github.com/go-kratos/kratos/v2/config/file" "github.com/go-kratos/kratos/v2/log" - "github.com/go-kratos/kratos/v2/transport/grpc" - "github.com/go-kratos/kratos/v2/transport/http" + "github.com/go-kratos/kratos/v2/transport" "go.opentelemetry.io/otel/trace" _ "go.uber.org/automaxprocs" @@ -66,7 +65,11 @@ type app struct { backend.Providers } -func newApp(logger log.Logger, gs *grpc.Server, hs *http.Server, ms *server.HTTPMetricsServer, providers backend.Providers, _ trace.TracerProvider) *app { +func newApp(logger log.Logger, gs *server.GRPCServers, hs *server.APIServer, ms *server.HTTPMetricsServer, providers backend.Providers, _ trace.TracerProvider) *app { + servers := []transport.Server{hs, ms} + if gs.Legacy != nil { + servers = append(servers, gs.Legacy) + } return &app{ kratos.New( kratos.ID(id), @@ -74,11 +77,7 @@ func newApp(logger log.Logger, gs *grpc.Server, hs *http.Server, ms *server.HTTP kratos.Version(Version), kratos.Metadata(map[string]string{}), kratos.Logger(logger), - kratos.Server( - gs, - hs, - ms, - ), + kratos.Server(servers...), ), providers, } diff --git a/app/artifact-cas/cmd/wire_gen.go b/app/artifact-cas/cmd/wire_gen.go index aa28bcf9a..87c2c4050 100644 --- a/app/artifact-cas/cmd/wire_gen.go +++ b/app/artifact-cas/cmd/wire_gen.go @@ -48,20 +48,30 @@ func wireApp(bootstrap *conf.Bootstrap, confServer *conf.Server, auth *conf.Auth v := serviceOpts(logger, auditDispatcher, bootstrap, cache) byteStreamService := service.NewByteStreamService(providers, v...) resourceService := service.NewResourceService(providers, v...) + downloadService := service.NewDownloadService(providers, v...) validator, err := newProtoValidator() if err != nil { cleanup2() cleanup() return nil, nil, err } - grpcServer, err := server.NewGRPCServer(confServer, auth, byteStreamService, resourceService, providers, validator, logger) + opts := &server.Opts{ + ServerConfig: confServer, + AuthConfig: auth, + ByteService: byteStreamService, + ResourceSvc: resourceService, + DownloadSvc: downloadService, + Providers: providers, + Validator: validator, + Logger: logger, + } + grpcServers, err := server.NewGRPCServers(opts) if err != nil { cleanup2() cleanup() return nil, nil, err } - downloadService := service.NewDownloadService(providers, v...) - httpServer, err := server.NewHTTPServer(confServer, auth, downloadService, providers, logger) + apiServer, err := server.NewHTTPServer(opts, grpcServers) if err != nil { cleanup2() cleanup() @@ -79,7 +89,7 @@ func wireApp(bootstrap *conf.Bootstrap, confServer *conf.Server, auth *conf.Auth cleanup() return nil, nil, err } - mainApp := newApp(logger, grpcServer, httpServer, httpMetricsServer, providers, tracerProvider) + mainApp := newApp(logger, grpcServers, apiServer, httpMetricsServer, providers, tracerProvider) return mainApp, func() { cleanup3() cleanup2() diff --git a/app/artifact-cas/configs/config.devel.yaml b/app/artifact-cas/configs/config.devel.yaml index 65936c30c..863da69fe 100644 --- a/app/artifact-cas/configs/config.devel.yaml +++ b/app/artifact-cas/configs/config.devel.yaml @@ -3,7 +3,10 @@ # Please make sure you run those via the convenience docker compose file # [root]/devel/compose.yml server: + # Keep 9001 available to compare native gRPC with the shared 8001 endpoint. + legacy_grpc_listener: true http: + # Serves HTTP/1.1, HTTP/2, and native gRPC over h2c. addr: 0.0.0.0:8001 # Timeouts for http downloads # grpc downloads/uploads don't require this because they don't have timeouts diff --git a/app/artifact-cas/configs/samples/config.yaml b/app/artifact-cas/configs/samples/config.yaml index be3e16fd0..7465950cb 100644 --- a/app/artifact-cas/configs/samples/config.yaml +++ b/app/artifact-cas/configs/samples/config.yaml @@ -3,9 +3,15 @@ auth: public_key_path: "./configs/devkeys/cas.public.pem" server: + # Retain the native transport while evaluating the shared HTTP endpoint. + legacy_grpc_listener: true http: addr: 0.0.0.0:8001 - timeout: 1s + timeout: 300s + # Optional TLS for both HTTP and native gRPC on port 8001. Independent of grpc.tls_config. + # tls_config: + # certificate: "./configs/tls/shared.crt" + # private_key: "./configs/tls/shared.key" grpc: addr: 0.0.0.0:9001 timeout: 1s diff --git a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt new file mode 100644 index 000000000..f0ccd4d6a --- /dev/null +++ b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt @@ -0,0 +1,37 @@ +# Copyright 2026 The Chainloop Authors. +# SPDX-License-Identifier: Apache-2.0 + +go version go1.27.2 linux/amd64 +grpc-go: v1.84.0; Kratos: v2.9.2 +Temporary filesystem: btrfs (not tmpfs) +Command: go test ./app/artifact-cas/internal/server -run ^$ -bench ^BenchmarkCASByteStreamTransport$ -benchtime=1x -count=3 -timeout=10m + +Artifact: 1 GiB; plaintext localhost TCP; real CAS services/auth/staging; disk-backed test backend; client and server in one process. +INFO msg=[HTTP] server listening on: 127.0.0.1:39645 +INFO msg=[gRPC] server listening on: 127.0.0.1:35009 +goos: linux +goarch: amd64 +pkg: github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server +cpu: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4428889185 ns/op 242.44 MB/s 26.92 peak-heap-MiB 1453444272 B/op 3158778 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4458294797 ns/op 240.84 MB/s 28.33 peak-heap-MiB 1432098544 B/op 3132177 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4502562403 ns/op 238.47 MB/s 27.97 peak-heap-MiB 1445793976 B/op 3132288 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5747630684 ns/op 186.81 MB/s 31.97 peak-heap-MiB 1373396504 B/op 84976 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5449605338 ns/op 197.03 MB/s 31.38 peak-heap-MiB 1401101656 B/op 80266 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5371327380 ns/op 199.90 MB/s 32.44 peak-heap-MiB 1379661432 B/op 80201 allocs/op +INFO msg=[gRPC] server stopping +INFO msg=[HTTP] server stopping +INFO msg=[gRPC] server stopping +INFO msg=[gRPC] server listening on: 127.0.0.1:36189 +INFO msg=[HTTP] server listening on: 127.0.0.1:43493 +BenchmarkCASByteStreamTransport/shared/upload-8 1 4757267201 ns/op 225.71 MB/s 1244 peak-heap-MiB 2002041336 B/op 3626317 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 4626278404 ns/op 232.10 MB/s 1304 peak-heap-MiB 2031947176 B/op 3636468 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 4628108813 ns/op 232.00 MB/s 1278 peak-heap-MiB 2034461720 B/op 3638817 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 6451314190 ns/op 166.44 MB/s 38.92 peak-heap-MiB 1246487112 B/op 314413 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 6177002391 ns/op 173.83 MB/s 39.37 peak-heap-MiB 1267943360 B/op 314523 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5987610343 ns/op 179.33 MB/s 37.41 peak-heap-MiB 1259259368 B/op 307480 allocs/op +INFO msg=[gRPC] server stopping +INFO msg=[HTTP] server stopping +INFO msg=[gRPC] server stopping +PASS +ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 65.296s diff --git a/app/artifact-cas/internal/conf/conf.pb.go b/app/artifact-cas/internal/conf/conf.pb.go index ac72ca7ae..8e33715e5 100644 --- a/app/artifact-cas/internal/conf/conf.pb.go +++ b/app/artifact-cas/internal/conf/conf.pb.go @@ -131,14 +131,16 @@ func (x *Bootstrap) GetStagingDir() string { type Server struct { state protoimpl.MessageState `protogen:"open.v1"` - // Regular HTTP endpoint + // Shared HTTP and native gRPC endpoint Http *Server_HTTP `protobuf:"bytes,1,opt,name=http,proto3" json:"http,omitempty"` // GRPC API endpoint Grpc *Server_GRPC `protobuf:"bytes,2,opt,name=grpc,proto3" json:"grpc,omitempty"` // hHTTP server where the prometheus metrics will get exposed - HttpMetrics *Server_HTTP `protobuf:"bytes,3,opt,name=http_metrics,json=httpMetrics,proto3" json:"http_metrics,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + HttpMetrics *Server_HTTP `protobuf:"bytes,3,opt,name=http_metrics,json=httpMetrics,proto3" json:"http_metrics,omitempty"` + // The native gRPC listener remains enabled when this field is unset. + LegacyGrpcListener *bool `protobuf:"varint,4,opt,name=legacy_grpc_listener,json=legacyGrpcListener,proto3,oneof" json:"legacy_grpc_listener,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *Server) Reset() { @@ -192,6 +194,13 @@ func (x *Server) GetHttpMetrics() *Server_HTTP { return nil } +func (x *Server) GetLegacyGrpcListener() bool { + if x != nil && x.LegacyGrpcListener != nil { + return *x.LegacyGrpcListener + } + return false +} + type Auth struct { state protoimpl.MessageState `protogen:"open.v1"` // Public key used to verify the received JWT token @@ -553,7 +562,9 @@ type Server_HTTP struct { Addr string `protobuf:"bytes,2,opt,name=addr,proto3" json:"addr,omitempty"` Timeout *durationpb.Duration `protobuf:"bytes,3,opt,name=timeout,proto3" json:"timeout,omitempty"` // CORS configuration for the HTTP download endpoint - Cors *Server_CORS `protobuf:"bytes,4,opt,name=cors,proto3" json:"cors,omitempty"` + Cors *Server_CORS `protobuf:"bytes,4,opt,name=cors,proto3" json:"cors,omitempty"` + // TLS for every protocol on the shared HTTP endpoint. + TlsConfig *Server_TLS `protobuf:"bytes,5,opt,name=tls_config,json=tlsConfig,proto3" json:"tls_config,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -616,6 +627,13 @@ func (x *Server_HTTP) GetCors() *Server_CORS { return nil } +func (x *Server_HTTP) GetTlsConfig() *Server_TLS { + if x != nil { + return x.TlsConfig + } + return nil +} + type Server_TLS struct { state protoimpl.MessageState `protogen:"open.v1"` // path to certificate and private key @@ -768,18 +786,21 @@ const file_conf_proto_rawDesc = "" + "\bendpoint\x18\x02 \x01(\tR\bendpoint\x12\x1a\n" + "\binsecure\x18\x03 \x01(\bR\binsecure\x12*\n" + "\x0esampling_ratio\x18\x04 \x01(\x01H\x00R\rsamplingRatio\x88\x01\x01B\x11\n" + - "\x0f_sampling_ratio\"\x9a\x04\n" + + "\x0f_sampling_ratio\"\x96\x05\n" + "\x06Server\x12 \n" + "\x04http\x18\x01 \x01(\v2\f.Server.HTTPR\x04http\x12 \n" + "\x04grpc\x18\x02 \x01(\v2\f.Server.GRPCR\x04grpc\x12/\n" + - "\fhttp_metrics\x18\x03 \x01(\v2\f.Server.HTTPR\vhttpMetrics\x1a+\n" + + "\fhttp_metrics\x18\x03 \x01(\v2\f.Server.HTTPR\vhttpMetrics\x125\n" + + "\x14legacy_grpc_listener\x18\x04 \x01(\bH\x00R\x12legacyGrpcListener\x88\x01\x01\x1a+\n" + "\x04CORS\x12#\n" + - "\rallow_origins\x18\x01 \x03(\tR\fallowOrigins\x1a\x8b\x01\n" + + "\rallow_origins\x18\x01 \x03(\tR\fallowOrigins\x1a\xb7\x01\n" + "\x04HTTP\x12\x18\n" + "\anetwork\x18\x01 \x01(\tR\anetwork\x12\x12\n" + "\x04addr\x18\x02 \x01(\tR\x04addr\x123\n" + "\atimeout\x18\x03 \x01(\v2\x19.google.protobuf.DurationR\atimeout\x12 \n" + - "\x04cors\x18\x04 \x01(\v2\f.Server.CORSR\x04cors\x1aH\n" + + "\x04cors\x18\x04 \x01(\v2\f.Server.CORSR\x04cors\x12*\n" + + "\n" + + "tls_config\x18\x05 \x01(\v2\v.Server.TLSR\ttlsConfig\x1aH\n" + "\x03TLS\x12 \n" + "\vcertificate\x18\x01 \x01(\tR\vcertificate\x12\x1f\n" + "\vprivate_key\x18\x02 \x01(\tR\n" + @@ -789,7 +810,8 @@ const file_conf_proto_rawDesc = "" + "\x04addr\x18\x02 \x01(\tR\x04addr\x123\n" + "\atimeout\x18\x03 \x01(\v2\x19.google.protobuf.DurationR\atimeout\x12*\n" + "\n" + - "tls_config\x18\x04 \x01(\v2\v.Server.TLSR\ttlsConfig\"t\n" + + "tls_config\x18\x04 \x01(\v2\v.Server.TLSR\ttlsConfigB\x17\n" + + "\x15_legacy_grpc_listener\"t\n" + "\x04Auth\x12D\n" + "\x1drobot_account_public_key_path\x18\x01 \x01(\tB\x02\x18\x01R\x19robotAccountPublicKeyPath\x12&\n" + "\x0fpublic_key_path\x18\x02 \x01(\tR\rpublicKeyPathBHZFgithub.com/chainloop-dev/chainloop/app/artifact-cas/internal/conf;confb\x06proto3" @@ -835,13 +857,14 @@ var file_conf_proto_depIdxs = []int32{ 6, // 9: Bootstrap.Observability.tracing:type_name -> Bootstrap.Observability.Tracing 12, // 10: Server.HTTP.timeout:type_name -> google.protobuf.Duration 7, // 11: Server.HTTP.cors:type_name -> Server.CORS - 12, // 12: Server.GRPC.timeout:type_name -> google.protobuf.Duration - 9, // 13: Server.GRPC.tls_config:type_name -> Server.TLS - 14, // [14:14] is the sub-list for method output_type - 14, // [14:14] is the sub-list for method input_type - 14, // [14:14] is the sub-list for extension type_name - 14, // [14:14] is the sub-list for extension extendee - 0, // [0:14] is the sub-list for field type_name + 9, // 12: Server.HTTP.tls_config:type_name -> Server.TLS + 12, // 13: Server.GRPC.timeout:type_name -> google.protobuf.Duration + 9, // 14: Server.GRPC.tls_config:type_name -> Server.TLS + 15, // [15:15] is the sub-list for method output_type + 15, // [15:15] is the sub-list for method input_type + 15, // [15:15] is the sub-list for extension type_name + 15, // [15:15] is the sub-list for extension extendee + 0, // [0:15] is the sub-list for field type_name } func init() { file_conf_proto_init() } @@ -849,6 +872,7 @@ func file_conf_proto_init() { if File_conf_proto != nil { return } + file_conf_proto_msgTypes[1].OneofWrappers = []any{} file_conf_proto_msgTypes[3].OneofWrappers = []any{ (*Bootstrap_NatsServer_Token)(nil), } diff --git a/app/artifact-cas/internal/conf/conf.proto b/app/artifact-cas/internal/conf/conf.proto index d9c8c5399..bf7ed41eb 100644 --- a/app/artifact-cas/internal/conf/conf.proto +++ b/app/artifact-cas/internal/conf/conf.proto @@ -76,6 +76,8 @@ message Server { google.protobuf.Duration timeout = 3; // CORS configuration for the HTTP download endpoint CORS cors = 4; + // TLS for every protocol on the shared HTTP endpoint. + TLS tls_config = 5; } message TLS { // path to certificate and private key @@ -88,12 +90,14 @@ message Server { google.protobuf.Duration timeout = 3; TLS tls_config = 4; } - // Regular HTTP endpoint + // Shared HTTP and native gRPC endpoint HTTP http = 1; // GRPC API endpoint GRPC grpc = 2; // hHTTP server where the prometheus metrics will get exposed HTTP http_metrics = 3; + // The native gRPC listener remains enabled when this field is unset. + optional bool legacy_grpc_listener = 4; } message Auth { diff --git a/app/artifact-cas/internal/server/grpc.go b/app/artifact-cas/internal/server/grpc.go index f497a613f..5c7eae93a 100644 --- a/app/artifact-cas/internal/server/grpc.go +++ b/app/artifact-cas/internal/server/grpc.go @@ -20,6 +20,7 @@ import ( "crypto/ecdsa" "crypto/tls" "fmt" + "net/url" "os" "regexp" @@ -51,8 +52,84 @@ import ( grpcLib "google.golang.org/grpc" ) -// NewGRPCServer new a gRPC server. -func NewGRPCServer(c *conf.Server, authConf *conf.Auth, byteService *service.ByteStreamService, rSvc *service.ResourceService, providers backend.Providers, validator protovalidate.Validator, logger log.Logger) (*grpc.Server, error) { +// Opts provides the shared service implementations and transport settings. +type Opts struct { + ServerConfig *conf.Server + AuthConfig *conf.Auth + ByteService *service.ByteStreamService + ResourceSvc *service.ResourceService + DownloadSvc *service.DownloadService + Providers backend.Providers + Validator protovalidate.Validator + Logger log.Logger +} + +// GRPCServers keeps the HTTP handler separate from the native transport. +// Handler must never be started with Serve; the HTTP server owns its streams. +type GRPCServers struct { + Handler *grpc.Server + Legacy *grpc.Server +} + +func legacyGRPCListenerEnabled(c *conf.Server) bool { + return c.LegacyGrpcListener == nil || *c.LegacyGrpcListener +} + +// NewGRPCServers builds both transports with the same services and interceptors. +func NewGRPCServers(opts *Opts) (*GRPCServers, error) { + scheme := "grpc" + if opts.ServerConfig.GetHttp().GetTlsConfig().GetCertificate() != "" { + scheme = "grpcs" + } + // Kratos stream interceptors require endpoint metadata even though this + // handler never binds a native socket. + handler, err := newGRPCServer(opts, grpc.Endpoint(&url.URL{ + Scheme: scheme, Host: opts.ServerConfig.GetHttp().GetAddr(), + })) + if err != nil { + return nil, err + } + servers := &GRPCServers{Handler: handler} + if legacyGRPCListenerEnabled(opts.ServerConfig) { + servers.Legacy, err = NewGRPCServer(opts) + if err != nil { + handler.Server.Stop() + return nil, err + } + } + return servers, nil +} + +// NewGRPCServer retains the existing native listener and its independent TLS. +func NewGRPCServer(opts *Opts) (*grpc.Server, error) { + c := opts.ServerConfig.GetGrpc() + var serverOpts []grpc.ServerOption + if c.GetNetwork() != "" { + serverOpts = append(serverOpts, grpc.Network(c.GetNetwork())) + } + if c.GetAddr() != "" { + serverOpts = append(serverOpts, grpc.Address(c.GetAddr())) + } + if tlsConf := c.GetTlsConfig(); tlsConf != nil { + cert, key := tlsConf.GetCertificate(), tlsConf.GetPrivateKey() + // Keep the legacy behavior for partial TLS configuration. + if cert != "" && key != "" { + pair, err := tls.LoadX509KeyPair(cert, key) + if err != nil { + return nil, fmt.Errorf("loading gRPC server TLS certificate: %w", err) + } + serverOpts = append(serverOpts, grpc.TLSConfig(&tls.Config{ + Certificates: []tls.Certificate{pair}, MinVersion: tls.VersionTLS12, + })) + } + } + return newGRPCServer(opts, serverOpts...) +} + +func newGRPCServer(settings *Opts, extraOpts ...grpc.ServerOption) (*grpc.Server, error) { + c, authConf := settings.ServerConfig, settings.AuthConfig + byteService, rSvc := settings.ByteService, settings.ResourceSvc + providers, validator, logger := settings.Providers, settings.Validator, settings.Logger // Parse the public key once on initialization instead of on every request // TODO: implement jwks endpoint publicKey, err := parsePublicKey(authConf, logger) @@ -104,28 +181,9 @@ func NewGRPCServer(c *conf.Server, authConf *conf.Auth, byteService *service.Byt // Since we track uploads / downloads we'll increase the buckets grpc_prometheus.EnableHandlingTimeHistogram(grpc_prometheus.WithHistogramBuckets(prometheus.ExponentialBucketsRange(0.5, 60, 8))) - if c.Grpc.Network != "" { - opts = append(opts, grpc.Network(c.Grpc.Network)) - } - if c.Grpc.Addr != "" { - opts = append(opts, grpc.Address(c.Grpc.Addr)) - } - if c.Grpc.Timeout != nil { - opts = append(opts, grpc.Timeout(c.Grpc.Timeout.AsDuration())) - } - if tlsConf := c.Grpc.GetTlsConfig(); tlsConf != nil { - cert := tlsConf.GetCertificate() - privKey := tlsConf.GetPrivateKey() - if cert != "" && privKey != "" { - cert, err := tls.LoadX509KeyPair(cert, privKey) - if err != nil { - return nil, fmt.Errorf("loading gRPC server TLS certificate: %w", err) - } - opts = append(opts, grpc.TLSConfig(&tls.Config{ - Certificates: []tls.Certificate{cert}, - MinVersion: tls.VersionTLS12, // gosec complains about insecure minimum version we use default value - })) - } + opts = append(opts, extraOpts...) + if c.GetGrpc().GetTimeout() != nil { + opts = append(opts, grpc.Timeout(c.GetGrpc().GetTimeout().AsDuration())) } srv := grpc.NewServer(opts...) diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index d38873340..89248a822 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -16,21 +16,48 @@ package server import ( + "context" + "crypto/tls" + "errors" + "fmt" + h "net/http" + "strings" + "time" + api "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/conf" "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/service" casJWT "github.com/chainloop-dev/chainloop/internal/robotaccount/cas" - backend "github.com/chainloop-dev/chainloop/pkg/blobmanager" middlewares_http "github.com/chainloop-dev/chainloop/pkg/middlewares/http" - "github.com/go-kratos/kratos/v2/log" "github.com/go-kratos/kratos/v2/middleware/logging" "github.com/go-kratos/kratos/v2/middleware/recovery" + "github.com/go-kratos/kratos/v2/transport/grpc" "github.com/go-kratos/kratos/v2/transport/http" "github.com/golang-jwt/jwt/v5" ) -// NewHTTPServer new a HTTP server. -func NewHTTPServer(c *conf.Server, authConf *conf.Auth, downloadSvc *service.DownloadService, providers backend.Providers, logger log.Logger) (*http.Server, error) { +// APIServer drains HTTP streams before stopping the gRPC handler. +type APIServer struct { + *http.Server + grpcHandler *grpc.Server +} + +func (s *APIServer) Stop(ctx context.Context) error { + httpErr := s.Server.Stop(ctx) + return errors.Join(httpErr, s.grpcHandler.Stop(ctx)) +} + +// NewHTTPServer dispatches every request on the existing CAS HTTP endpoint. +func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) { + c, authConf := settings.ServerConfig, settings.AuthConfig + downloadSvc, providers, logger := settings.DownloadSvc, settings.Providers, settings.Logger + if c.GetHttp().GetAddr() == "" { + return nil, errors.New("server.http.addr must be set for the shared CAS endpoint") + } + tlsConfig, err := loadServerTLSConfig(c.GetHttp().GetTlsConfig()) + if err != nil { + return nil, err + } var opts = []http.ServerOption{ http.Middleware( recovery.Recovery(), @@ -47,6 +74,10 @@ func NewHTTPServer(c *conf.Server, authConf *conf.Auth, downloadSvc *service.Dow opts = append(opts, http.Timeout(c.Http.Timeout.AsDuration())) } + if tlsConfig != nil { + opts = append(opts, http.TLSConfig(tlsConfig)) + } + // Parse the public key once on initialization instead of on every request // TODO: implement jwks endpoint publicKey, err := parsePublicKey(authConf, logger) @@ -59,7 +90,21 @@ func NewHTTPServer(c *conf.Server, authConf *conf.Auth, downloadSvc *service.Dow downloadHandler := middlewares_http.AuthFromHeaderOrQueryParam(loadPublicKey(publicKey), claimsFunc(), casJWT.SigningMethod, downloadSvc) srv.Handle(service.DownloadPath, CORSMiddleware(c.GetHttp().GetCors().GetAllowOrigins(), downloadHandler)) api.RegisterStatusServiceHTTPServer(srv, service.NewStatusService(Version, providers)) - return srv, nil + shared := http.NewServer(opts...) + shared.Protocols = new(h.Protocols) + shared.Protocols.SetHTTP1(true) + shared.Protocols.SetHTTP2(true) + shared.Protocols.SetUnencryptedHTTP2(true) + shared.ReadHeaderTimeout = 60 * time.Second + shared.Handler = h.HandlerFunc(func(w h.ResponseWriter, r *h.Request) { + contentType := strings.ToLower(r.Header.Get("Content-Type")) + if r.ProtoMajor == 2 && strings.HasPrefix(contentType, "application/grpc") && !strings.HasPrefix(contentType, "application/grpc-web") { + grpcServers.Handler.ServeHTTP(w, r) + return + } + srv.ServeHTTP(w, r) + }) + return &APIServer{Server: shared, grpcHandler: grpcServers.Handler}, nil } // claimsFunc returns the claims function for the JWT middleware that casts the claims to the correct type @@ -68,3 +113,22 @@ func claimsFunc() middlewares_http.ClaimsFunc { return &casJWT.Claims{} } } + +// loadServerTLSConfig applies TLS independently of the legacy gRPC listener. +func loadServerTLSConfig(c *conf.Server_TLS) (*tls.Config, error) { + cert, key := c.GetCertificate(), c.GetPrivateKey() + if cert == "" && key == "" { + return nil, nil + } + if cert == "" || key == "" { + return nil, errors.New("incomplete TLS configuration: certificate and private key must be set together") + } + pair, err := tls.LoadX509KeyPair(cert, key) + if err != nil { + return nil, fmt.Errorf("loading shared CAS TLS certificate: %w", err) + } + return &tls.Config{ + Certificates: []tls.Certificate{pair}, MinVersion: tls.VersionTLS12, + NextProtos: []string{"h2", "http/1.1"}, + }, nil +} diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go new file mode 100644 index 000000000..64f8b98d7 --- /dev/null +++ b/app/artifact-cas/internal/server/http_test.go @@ -0,0 +1,579 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "bytes" + "context" + "crypto/sha256" + "crypto/tls" + "crypto/x509" + "encoding/base64" + "encoding/gob" + "encoding/hex" + "encoding/pem" + "errors" + "io" + "net" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync" + "sync/atomic" + "testing" + "time" + + "buf.build/go/protovalidate" + v1 "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" + "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/conf" + "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/service" + casJWT "github.com/chainloop-dev/chainloop/internal/robotaccount/cas" + backend "github.com/chainloop-dev/chainloop/pkg/blobmanager" + "github.com/chainloop-dev/chainloop/pkg/blobmanager/mocks" + "github.com/go-kratos/kratos/v2/log" + "github.com/go-kratos/kratos/v2/transport" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" + "google.golang.org/genproto/googleapis/bytestream" + grpcLib "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/metadata" + reflection "google.golang.org/grpc/reflection/grpc_reflection_v1" + "google.golang.org/grpc/status" +) + +const ( + casAnyAddr = "127.0.0.1:0" + casSharedEndpoint = "shared" + casLegacyEndpoint = "legacy" + casGRPCRequest = "grpc" + casHTTPRequest = "http" + casPreflightRequest = "preflight" + casExpiredTokenCase = "expired token" +) + +// diskBackend stores the bytes rather than mocking the service's upload and +// download results. It keeps test artifact contents off the Go heap. +type diskBackend struct{ dir string } + +func (b *diskBackend) Upload(_ context.Context, r io.Reader, resource *v1.CASResource) error { + f, err := os.Create(filepath.Join(b.dir, resource.Digest)) + if err != nil { + return err + } + _, copyErr := io.Copy(f, r) + closeErr := f.Close() + if copyErr != nil { + return copyErr + } + return closeErr +} +func (b *diskBackend) Download(_ context.Context, w io.Writer, digest string) error { + f, err := os.Open(filepath.Join(b.dir, digest)) + if err != nil { + return err + } + defer f.Close() + _, err = io.Copy(w, f) + return err +} +func (b *diskBackend) Describe(_ context.Context, digest string) (*v1.CASResource, error) { + info, err := os.Stat(filepath.Join(b.dir, digest)) + if err != nil { + return nil, err + } + return &v1.CASResource{Digest: digest, FileName: "artifact.bin", Size: info.Size()}, nil +} +func (b *diskBackend) Exists(_ context.Context, digest string) (bool, error) { + _, err := os.Stat(filepath.Join(b.dir, digest)) + if os.IsNotExist(err) { + return false, nil + } + return err == nil, err +} +func (*diskBackend) CheckWritePermissions(context.Context) error { return nil } + +func casTestOpts(t testing.TB) *Opts { + t.Helper() + storage := &diskBackend{dir: t.TempDir()} + provider := mocks.NewProvider(t) + provider.On("FromCredentials", mock.Anything, "test-secret").Maybe().Return(storage, nil) + providers := backend.Providers{"test": provider} + validator, err := protovalidate.New() + require.NoError(t, err) + svcOpts := []service.NewOpt{service.WithStagingDir(t.TempDir())} + return &Opts{ + ServerConfig: &conf.Server{ + Http: &conf.Server_HTTP{Addr: casAnyAddr, Cors: &conf.Server_CORS{AllowOrigins: []string{"https://example.test"}}}, + Grpc: &conf.Server_GRPC{Addr: casAnyAddr}, + }, + AuthConfig: &conf.Auth{PublicKeyPath: "testdata/test-key.ec.pub"}, + ByteService: service.NewByteStreamService(providers, svcOpts...), + ResourceSvc: service.NewResourceService(providers, svcOpts...), + DownloadSvc: service.NewDownloadService(providers, svcOpts...), + Providers: providers, Validator: validator, Logger: log.NewStdLogger(io.Discard), + } +} + +func startCASTestServers(t testing.TB, opts *Opts) (sharedURL, legacyAddr string) { + t.Helper() + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + endpoint, err := api.Endpoint() + require.NoError(t, err) + startCASTestTransport(t, api) + if servers.Legacy != nil { + legacy, err := servers.Legacy.Endpoint() + require.NoError(t, err) + legacyAddr = legacy.Host + startCASTestTransport(t, servers.Legacy) + } + return endpoint.String(), legacyAddr +} + +func startCASTestTransport(t testing.TB, srv transport.Server) { + t.Helper() + done := make(chan error, 1) + go func() { done <- srv.Start(context.Background()) }() + t.Cleanup(func() { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + require.NoError(t, srv.Stop(ctx)) + select { + case err := <-done: + require.NoError(t, err) + case <-ctx.Done(): + require.FailNow(t, "CAS server did not stop") + } + }) +} + +func casTestConn(t testing.TB, addr string, useTLS bool) *grpcLib.ClientConn { + t.Helper() + creds := insecure.NewCredentials() + if useTLS { + creds = credentials.NewTLS(&tls.Config{InsecureSkipVerify: true}) //nolint:gosec // ephemeral test certificate + } + conn, err := grpcLib.NewClient(addr, grpcLib.WithTransportCredentials(creds)) + require.NoError(t, err) + t.Cleanup(func() { _ = conn.Close() }) + return conn +} + +func casTestToken(t testing.TB, role casJWT.Role, maxBytes int64, expires time.Duration) string { + t.Helper() + builder, err := casJWT.NewBuilder(casJWT.WithPrivateKey("testdata/test-key.ec.pem"), casJWT.WithIssuer("cas-transport-test"), casJWT.WithExpiration(expires)) + require.NoError(t, err) + token, err := builder.GenerateJWT("test", "test-secret", casJWT.JWTAudience, role, maxBytes, "test-org") + require.NoError(t, err) + return token +} + +func casTestContext(token string) (context.Context, context.CancelFunc) { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + if token != "" { + ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+token) + } + return ctx, cancel +} + +func casResourceName(t testing.TB, resource *v1.CASResource) string { + t.Helper() + var buf bytes.Buffer + require.NoError(t, gob.NewEncoder(&buf).Encode(resource)) + return base64.StdEncoding.EncodeToString(buf.Bytes()) +} + +func casTestUpload(t testing.TB, client bytestream.ByteStreamClient, token, digest string, payload io.Reader, size int64) error { + t.Helper() + ctx, cancel := casTestContext(token) + defer cancel() + stream, err := client.Write(ctx) + if err != nil { + return err + } + resource := casResourceName(t, &v1.CASResource{Digest: digest, FileName: "artifact.bin"}) + chunk := make([]byte, 64<<10) + for { + n, readErr := payload.Read(chunk) + if readErr != nil && !errors.Is(readErr, io.EOF) { + return readErr + } + req := &bytestream.WriteRequest{ResourceName: resource, Data: chunk[:n], FinishWrite: errors.Is(readErr, io.EOF)} + if err = stream.Send(req); err != nil { + break + } + if errors.Is(readErr, io.EOF) { + break + } + } + resp, err := stream.CloseAndRecv() + if err == nil { + require.Equal(t, size, resp.GetCommittedSize()) + } + return err +} + +func TestSharedCASStatus(t *testing.T) { + sharedURL, legacy := startCASTestServers(t, casTestOpts(t)) + resp, err := http.Get(sharedURL + "/statusz") + require.NoError(t, err) + _, err = io.Copy(io.Discard, resp.Body) + require.NoError(t, resp.Body.Close()) + require.NoError(t, err) + require.Equal(t, http.StatusOK, resp.StatusCode) + for _, addr := range []string{sharedURL[len("http://"):], legacy} { + conn := casTestConn(t, addr, false) + ctx, cancel := casTestContext("") + _, err = v1.NewStatusServiceClient(conn).Statusz(ctx, &v1.StatuszRequest{}) + require.NoError(t, err) + stream, err := reflection.NewServerReflectionClient(conn).ServerReflectionInfo(ctx) + require.NoError(t, err) + require.NoError(t, stream.Send(&reflection.ServerReflectionRequest{MessageRequest: &reflection.ServerReflectionRequest_ListServices{}})) + result, err := stream.Recv() + require.NoError(t, err) + require.NotEmpty(t, result.GetListServicesResponse().GetService()) + require.NoError(t, stream.CloseSend()) + cancel() + } +} + +func TestCASByteStreamTransportParity(t *testing.T) { + for _, useTLS := range []bool{false, true} { + t.Run(map[bool]string{false: "h2c", true: "TLS"}[useTLS], func(t *testing.T) { + for _, endpoint := range []string{casSharedEndpoint, casLegacyEndpoint} { + t.Run(endpoint, func(t *testing.T) { + opts := casTestOpts(t) + if useTLS { + opts.ServerConfig.Http.TlsConfig = casTestTLS(t) + opts.ServerConfig.Grpc.TlsConfig = casTestTLS(t) + } + sharedURL, legacy := startCASTestServers(t, opts) + addr := legacy + if endpoint == casSharedEndpoint { + addr = sharedURL[len("http://"):] + if useTLS { + addr = sharedURL[len("https://"):] + } + } + conn := casTestConn(t, addr, useTLS) + client := bytestream.NewByteStreamClient(conn) + content := bytes.Repeat([]byte("streamed artifact\n"), 64<<10) + sum := sha256.Sum256(content) + digest := hex.EncodeToString(sum[:]) + up := casTestToken(t, casJWT.Uploader, 0, time.Hour) + down := casTestToken(t, casJWT.Downloader, 0, time.Hour) + require.NoError(t, casTestUpload(t, client, up, digest, bytes.NewReader(content), int64(len(content)))) + ctx, cancel := casTestContext(down) + defer cancel() + stream, err := client.Read(ctx, &bytestream.ReadRequest{ResourceName: digest}) + require.NoError(t, err) + hash := sha256.New() + var size int64 + for { + msg, err := stream.Recv() + if errors.Is(err, io.EOF) { + break + } + require.NoError(t, err) + n, err := hash.Write(msg.GetData()) + require.NoError(t, err) + size += int64(n) + } + require.EqualValues(t, len(content), size) + require.Equal(t, digest, hex.EncodeToString(hash.Sum(nil))) + _, err = v1.NewResourceServiceClient(conn).Describe(ctx, &v1.ResourceServiceDescribeRequest{Digest: digest}) + require.NoError(t, err) + _, err = v1.NewResourceServiceClient(conn).Describe(ctx, &v1.ResourceServiceDescribeRequest{}) + require.Equal(t, codes.InvalidArgument, status.Code(err)) + + // The HTTP download remains usable regardless of which RPC listener uploaded the blob. + tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // test certificate + defer tr.CloseIdleConnections() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, sharedURL+"/download/sha256:"+digest, nil) + require.NoError(t, err) + req.Header.Set("Authorization", "Bearer "+down) + response, err := (&http.Client{Transport: tr, Timeout: 5 * time.Second}).Do(req) + require.NoError(t, err) + require.Equal(t, http.StatusOK, response.StatusCode) + hash.Reset() + size, err = io.Copy(hash, response.Body) + require.NoError(t, response.Body.Close()) + require.NoError(t, err) + require.EqualValues(t, len(content), size) + require.Equal(t, digest, hex.EncodeToString(hash.Sum(nil))) + }) + } + }) + } +} + +func casTestTLS(t testing.TB) *conf.Server_TLS { + t.Helper() + srv := httptest.NewTLSServer(http.NotFoundHandler()) + defer srv.Close() + pair := srv.TLS.Certificates[0] + key, err := x509.MarshalPKCS8PrivateKey(pair.PrivateKey) + require.NoError(t, err) + dir := t.TempDir() + certPath, keyPath := filepath.Join(dir, "cert.pem"), filepath.Join(dir, "key.pem") + require.NoError(t, os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: pair.Certificate[0]}), 0600)) + require.NoError(t, os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: key}), 0600)) + return &conf.Server_TLS{Certificate: certPath, PrivateKey: keyPath} +} + +func TestCASSharedConnectionReuse(t *testing.T) { + for _, useTLS := range []bool{false, true} { + t.Run(map[bool]string{false: "h2c", true: "TLS"}[useTLS], func(t *testing.T) { + opts := casTestOpts(t) + if useTLS { + opts.ServerConfig.Http.TlsConfig = casTestTLS(t) + } + sharedURL, _ := startCASTestServers(t, opts) + var connections atomic.Int32 + tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // test certificate + DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { + connections.Add(1) + return (&net.Dialer{}).DialContext(ctx, network, addr) + }, + } + tr.Protocols.SetHTTP2(true) + tr.Protocols.SetUnencryptedHTTP2(true) + defer tr.CloseIdleConnections() + client := &http.Client{Transport: tr, Timeout: 5 * time.Second} + for _, kind := range []string{casGRPCRequest, casHTTPRequest, casGRPCRequest, casPreflightRequest, casHTTPRequest, casGRPCRequest} { + path, method := "/cas.v1.StatusService/Statusz", http.MethodPost + if kind == casHTTPRequest { + path, method = "/statusz", http.MethodGet + } + if kind == casPreflightRequest { + path, method = "/download/sha256:abc", http.MethodOptions + } + req, err := http.NewRequest(method, sharedURL+path, bytes.NewReader([]byte{0, 0, 0, 0, 0})) + require.NoError(t, err) + if kind == casGRPCRequest { + req.Header.Set("Content-Type", "application/grpc") + req.Header.Set("Te", "trailers") + } + if kind == casPreflightRequest { + req.Header.Set("Origin", "https://example.test") + req.Header.Set("Access-Control-Request-Method", "GET") + } + resp, err := client.Do(req) + require.NoError(t, err) + _, err = io.Copy(io.Discard, resp.Body) + require.NoError(t, resp.Body.Close()) + require.NoError(t, err) + expected := http.StatusOK + if kind == casPreflightRequest { + expected = http.StatusNoContent + } + require.Equal(t, expected, resp.StatusCode, kind) + require.Equal(t, 2, resp.ProtoMajor) + if kind == casGRPCRequest { + require.Equal(t, "0", resp.Trailer.Get("Grpc-Status")) + } + } + require.EqualValues(t, 1, connections.Load()) + }) + } +} + +// These focused cases compare the existing middleware on both transports; +// they do not replace the complete original R-002 qualification matrix. +func TestCASAuthenticationAndUploadErrors(t *testing.T) { + for _, endpoint := range []string{casSharedEndpoint, casLegacyEndpoint} { + t.Run(endpoint, func(t *testing.T) { + opts := casTestOpts(t) + sharedURL, legacy := startCASTestServers(t, opts) + addr := legacy + if endpoint == casSharedEndpoint { + addr = sharedURL[len("http://"):] + } + conn := casTestConn(t, addr, false) + client := bytestream.NewByteStreamClient(conn) + content := []byte("an authenticated CAS artifact") + sum := sha256.Sum256(content) + digest := hex.EncodeToString(sum[:]) + tests := []struct { + name, token, digest string + code codes.Code + }{ + {name: "missing token", digest: digest, code: codes.Unauthenticated}, + {name: "malformed token", token: "invalid.jwt.token", digest: digest, code: codes.Unauthenticated}, + {name: casExpiredTokenCase, token: casTestToken(t, casJWT.Uploader, 0, -time.Hour), digest: digest, code: codes.Unauthenticated}, + {name: "downloader cannot upload", token: casTestToken(t, casJWT.Downloader, 0, time.Hour), digest: digest, code: codes.Unauthenticated}, + {name: "upload size cap", token: casTestToken(t, casJWT.Uploader, 1, time.Hour), digest: digest, code: codes.ResourceExhausted}, + {name: "digest mismatch", token: casTestToken(t, casJWT.Uploader, 0, time.Hour), digest: hex.EncodeToString(make([]byte, sha256.Size)), code: codes.InvalidArgument}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + err := casTestUpload(t, client, tc.token, tc.digest, bytes.NewReader(content), int64(len(content))) + require.Equal(t, tc.code, status.Code(err), "%v", err) + }) + } + for _, tc := range []struct { + name, token string + code codes.Code + }{ + {name: "missing token", code: codes.Unauthenticated}, + {name: casExpiredTokenCase, token: casTestToken(t, casJWT.Downloader, 0, -time.Hour), code: codes.Unauthenticated}, + {name: "uploader cannot download", token: casTestToken(t, casJWT.Uploader, 0, time.Hour), code: codes.Unauthenticated}, + } { + t.Run("read/"+tc.name, func(t *testing.T) { + ctx, cancel := casTestContext(tc.token) + defer cancel() + stream, err := client.Read(ctx, &bytestream.ReadRequest{ResourceName: digest}) + require.NoError(t, err) + _, err = stream.Recv() + require.Equal(t, tc.code, status.Code(err), "%v", err) + }) + } + }) + } +} + +func TestCASLegacyListenerAndTLSConfiguration(t *testing.T) { + for _, tc := range []struct { + name string + enabled *bool + wantLegacy bool + }{ + {name: "default", wantLegacy: true}, + {name: "explicit on", enabled: new(true), wantLegacy: true}, + {name: "explicit off", enabled: new(false)}, + } { + t.Run(tc.name, func(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = tc.enabled + if !tc.wantLegacy { + opts.ServerConfig.Grpc.TlsConfig = &conf.Server_TLS{Certificate: "unused-invalid-cert", PrivateKey: "unused-invalid-key"} + } + sharedURL, legacy := startCASTestServers(t, opts) + require.Equal(t, tc.wantLegacy, legacy != "") + conn := casTestConn(t, sharedURL[len("http://"):], false) + ctx, cancel := casTestContext("") + defer cancel() + _, err := v1.NewStatusServiceClient(conn).Statusz(ctx, &v1.StatuszRequest{}) + require.NoError(t, err) + }) + } + for _, tc := range []struct { + name string + http *conf.Server_HTTP + wantErr string + }{ + {name: "missing shared address", http: &conf.Server_HTTP{}, wantErr: "server.http.addr must be set"}, + {name: "partial shared TLS", http: &conf.Server_HTTP{Addr: casAnyAddr, TlsConfig: &conf.Server_TLS{Certificate: "only-cert"}}, wantErr: "incomplete TLS configuration"}, + } { + t.Run(tc.name, func(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.Http = tc.http + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + defer servers.Handler.Server.Stop() + _, err = NewHTTPServer(opts, servers) + require.ErrorContains(t, err, tc.wantErr) + }) + } +} + +func TestCASCancelledUpload(t *testing.T) { + for _, endpoint := range []string{casSharedEndpoint, casLegacyEndpoint} { + t.Run(endpoint, func(t *testing.T) { + opts := casTestOpts(t) + sharedURL, legacy := startCASTestServers(t, opts) + addr := legacy + if endpoint == casSharedEndpoint { + addr = sharedURL[len("http://"):] + } + client := bytestream.NewByteStreamClient(casTestConn(t, addr, false)) + ctx, cancel := casTestContext(casTestToken(t, casJWT.Uploader, 0, time.Hour)) + defer cancel() + stream, err := client.Write(ctx) + require.NoError(t, err) + digest := hex.EncodeToString(make([]byte, sha256.Size)) + require.NoError(t, stream.Send(&bytestream.WriteRequest{ResourceName: casResourceName(t, &v1.CASResource{Digest: digest}), Data: make([]byte, 64<<10)})) + cancel() + _, err = stream.CloseAndRecv() + require.Equal(t, codes.Canceled, status.Code(err)) + // No partial blob reaches the backend after cancellation. + p, err := opts.Providers["test"].FromCredentials(context.Background(), "test-secret") + require.NoError(t, err) + require.Eventually(t, func() bool { + exists, err := p.Exists(context.Background(), digest) + return err == nil && !exists + }, time.Second, 10*time.Millisecond) + }) + } +} + +func TestCASSharedShutdownDrainsUpload(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + provider := opts.Providers["test"].(*mocks.Provider) + entered, release := make(chan struct{}), make(chan struct{}) + unblock := sync.OnceFunc(func() { close(release) }) + defer unblock() + provider.ExpectedCalls[0].Run(func(mock.Arguments) { close(entered); <-release }) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + endpoint, err := api.Endpoint() + require.NoError(t, err) + startCASTestTransport(t, api) + client := bytestream.NewByteStreamClient(casTestConn(t, endpoint.Host, false)) + ctx, cancel := casTestContext(casTestToken(t, casJWT.Uploader, 0, time.Hour)) + defer cancel() + content := make([]byte, 64<<10) + sum := sha256.Sum256(content) + stream, err := client.Write(ctx) + require.NoError(t, err) + require.NoError(t, stream.Send(&bytestream.WriteRequest{ResourceName: casResourceName(t, &v1.CASResource{Digest: hex.EncodeToString(sum[:])}), Data: content})) + select { + case <-entered: + case <-ctx.Done(): + require.FailNow(t, "upload did not reach the backend lookup") + } + stopped := make(chan error, 1) + go func() { + stopCtx, stopCancel := context.WithTimeout(context.Background(), 5*time.Second) + defer stopCancel() + stopped <- api.Stop(stopCtx) + }() + select { + case err := <-stopped: + require.FailNow(t, "shutdown returned before the upload completed", "%v", err) + case <-time.After(50 * time.Millisecond): + } + unblock() + require.NoError(t, stream.Send(&bytestream.WriteRequest{ResourceName: casResourceName(t, &v1.CASResource{Digest: hex.EncodeToString(sum[:])}), FinishWrite: true})) + response, err := stream.CloseAndRecv() + require.NoError(t, err) + require.EqualValues(t, len(content), response.GetCommittedSize()) + select { + case err := <-stopped: + require.NoError(t, err) + case <-ctx.Done(): + require.FailNow(t, "shutdown did not complete after the upload") + } +} diff --git a/app/artifact-cas/internal/server/server.go b/app/artifact-cas/internal/server/server.go index 73b2b2280..4897ffc24 100644 --- a/app/artifact-cas/internal/server/server.go +++ b/app/artifact-cas/internal/server/server.go @@ -1,5 +1,5 @@ // -// Copyright 2023 The Chainloop Authors. +// Copyright 2023-2026 The Chainloop Authors. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. @@ -20,6 +20,6 @@ import ( ) // ProviderSet is server providers. -var ProviderSet = wire.NewSet(NewGRPCServer, NewHTTPServer, NewHTTPMetricsServer, NewTracerProvider) +var ProviderSet = wire.NewSet(wire.Struct(new(Opts), "*"), NewGRPCServers, NewHTTPServer, NewHTTPMetricsServer, NewTracerProvider) var Version = "dev" diff --git a/app/artifact-cas/internal/server/streaming_bench_test.go b/app/artifact-cas/internal/server/streaming_bench_test.go new file mode 100644 index 000000000..fc7fb1587 --- /dev/null +++ b/app/artifact-cas/internal/server/streaming_bench_test.go @@ -0,0 +1,149 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "io" + "os" + "path/filepath" + "runtime" + "strconv" + "sync" + "testing" + "time" + + casJWT "github.com/chainloop-dev/chainloop/internal/robotaccount/cas" + "github.com/stretchr/testify/require" + "google.golang.org/genproto/googleapis/bytestream" +) + +// This compares the production CAS service and interceptors over the unchanged +// native transport and the HTTP adapter, with the same disk-backed test backend. +// It is a transport experiment, not the original R-003 release/S3 qualification. +// Use a disk-backed TMPDIR: /tmp may be tmpfs on development machines. +func BenchmarkCASByteStreamTransport(b *testing.B) { + size := int64(1 << 30) + if value := os.Getenv("CAS_BENCH_BYTES"); value != "" { + parsed, err := strconv.ParseInt(value, 10, 64) + require.NoError(b, err) + require.Positive(b, parsed) + size = parsed + } + hash := sha256.New() + _, err := io.CopyN(hash, zeroReader{}, size) + require.NoError(b, err) + digest := hex.EncodeToString(hash.Sum(nil)) + for _, endpoint := range []string{casLegacyEndpoint, casSharedEndpoint} { + b.Run(endpoint, func(b *testing.B) { + opts := casTestOpts(b) + sharedURL, legacy := startCASTestServers(b, opts) + addr := legacy + if endpoint == casSharedEndpoint { + addr = sharedURL[len("http://"):] + } + client := bytestream.NewByteStreamClient(casTestConn(b, addr, false)) + storage, err := opts.Providers["test"].FromCredentials(context.Background(), "test-secret") + require.NoError(b, err) + path := filepath.Join(storage.(*diskBackend).dir, digest) + up := casTestToken(b, casJWT.Uploader, 0, time.Hour) + down := casTestToken(b, casJWT.Downloader, 0, time.Hour) + b.Run("upload", func(b *testing.B) { + b.SetBytes(size) + b.ReportAllocs() + stopSample := sampleCASHeap() + defer stopSample() + b.ResetTimer() + for range b.N { + b.StopTimer() + err := os.Remove(path) + require.True(b, err == nil || os.IsNotExist(err), "%v", err) + b.StartTimer() + require.NoError(b, casTestUpload(b, client, up, digest, io.LimitReader(zeroReader{}, size), size)) + } + b.StopTimer() + b.ReportMetric(float64(stopSample())/(1<<20), "peak-heap-MiB") + }) + // Seed downloads independently of the upload benchmark. A sparse + // file represents the same zero bytes without a heap-sized buffer. + f, err := os.Create(path) + require.NoError(b, err) + require.NoError(b, f.Truncate(size)) + require.NoError(b, f.Close()) + b.Run("download", func(b *testing.B) { + b.SetBytes(size) + b.ReportAllocs() + stopSample := sampleCASHeap() + defer stopSample() + b.ResetTimer() + for range b.N { + ctx, cancel := casTestContext(down) + stream, err := client.Read(ctx, &bytestream.ReadRequest{ResourceName: digest}) + require.NoError(b, err) + hash := sha256.New() + var received int64 + for { + msg, err := stream.Recv() + if errors.Is(err, io.EOF) { + break + } + require.NoError(b, err) + n, err := hash.Write(msg.GetData()) + require.NoError(b, err) + received += int64(n) + } + cancel() + require.Equal(b, size, received) + require.Equal(b, digest, hex.EncodeToString(hash.Sum(nil))) + } + b.StopTimer() + b.ReportMetric(float64(stopSample())/(1<<20), "peak-heap-MiB") + }) + }) + } +} + +type zeroReader struct{} + +func (zeroReader) Read(p []byte) (int, error) { clear(p); return len(p), nil } + +// sampleCASHeap reports sampled live heap, not total allocated bytes. Both the +// client and server run here; this is not a server-only RSS or bounded-memory proof. +func sampleCASHeap() func() uint64 { + runtime.GC() + done, result := make(chan struct{}), make(chan uint64, 1) + go func() { + ticker := time.NewTicker(10 * time.Millisecond) + defer ticker.Stop() + var peak uint64 + for { + var stats runtime.MemStats + runtime.ReadMemStats(&stats) + if stats.HeapAlloc > peak { + peak = stats.HeapAlloc + } + select { + case <-done: + result <- peak + return + case <-ticker.C: + } + } + }() + return sync.OnceValue(func() uint64 { close(done); return <-result }) +} diff --git a/deployment/chainloop/Chart.yaml b/deployment/chainloop/Chart.yaml index 713c92fd9..f05606c26 100644 --- a/deployment/chainloop/Chart.yaml +++ b/deployment/chainloop/Chart.yaml @@ -7,7 +7,7 @@ description: Chainloop is an open source software supply chain control plane, a type: application # Bump the patch (not minor, not major) version on each change in the Chart Source code -version: 1.468.1 +version: 1.468.2 # Do not update appVersion, this is handled automatically by the release process appVersion: v1.125.0 diff --git a/deployment/chainloop/README.md b/deployment/chainloop/README.md index 28d51d8d6..efc38553e 100644 --- a/deployment/chainloop/README.md +++ b/deployment/chainloop/README.md @@ -245,7 +245,7 @@ The control plane ingress serves HTTP, gRPC-Web, and native gRPC on the same hos Its HTTP server routes each request, so these protocols can share an upstream HTTP/2 connection. The existing gRPC ingress stays available. Existing clients can keep their current endpoints. To test the shared endpoint, point a gRPC client to the control plane HTTP hostname on port 443. -The Artifact CAS keeps its existing endpoints and native gRPC transport. +On the CAS experiment branch, `cas.ingress` and `cas.service` use the same HTTP/2 upstream settings for native gRPC and HTTP downloads. `cas.ingressAPI` and the native gRPC endpoint remain independent, and the control plane continues to use that legacy endpoint. CAS does not add gRPC-Web or Connect. The shared CAS transport still needs streaming and performance qualification before migration; see the [local experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). The chart configures HTTP/2 upstream traffic for these controllers: @@ -843,7 +843,7 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | `cas.service.loadBalancerIP` | cas service Load Balancer IP | `""` | | `cas.service.loadBalancerSourceRanges` | cas service Load Balancer sources | `[]` | | `cas.service.externalTrafficPolicy` | cas service external traffic policy | `Cluster` | -| `cas.service.annotations` | Additional custom annotations for cas service | `{}` | +| `cas.service.annotations` | Annotations for the shared HTTP and gRPC service. Traefik uses h2c upstream. | See values.yaml | | `cas.service.extraPorts` | Extra ports to expose in cas service (normally used with the `sidecars` value) | `[]` | | `cas.service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` | | `cas.service.sessionAffinityConfig` | Additional settings for the sessionAffinity | `{}` | @@ -865,7 +865,7 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | `cas.ingress.hostname` | Default host for the ingress record | `cas.dev.local` | | `cas.ingress.ingressClassName` | IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+) | `""` | | `cas.ingress.path` | Default path for the ingress record | `/` | -| `cas.ingress.annotations` | Additional annotations for the Ingress resource. To enable certificate autogeneration, place here your cert-manager annotations. | `{}` | +| `cas.ingress.annotations` | Annotations for the experimental shared HTTP and gRPC ingress. | See values.yaml | | `cas.ingress.tls` | Enable TLS configuration for the host defined at `controlplane.ingress.hostname` parameter | `false` | | `cas.ingress.selfSigned` | Create a TLS secret for this ingress record using self-signed certificates generated by Helm | `false` | | `cas.ingress.extraHosts` | An array with additional hostname(s) to be covered with the ingress record | `[]` | diff --git a/deployment/chainloop/chart_test.go b/deployment/chainloop/chart_test.go index 149b81484..837252de0 100644 --- a/deployment/chainloop/chart_test.go +++ b/deployment/chainloop/chart_test.go @@ -29,8 +29,15 @@ import ( ) // Verify the rendered resources, including templated F5 service names and -// overrides. The legacy gRPC and CAS resources keep their own upstreams. -func TestSharedControlplaneIngress(t *testing.T) { +// overrides. The legacy gRPC resources keep their own upstreams. +func TestSharedIngress(t *testing.T) { + for _, component := range []string{"controlplane", "cas"} { + t.Run(component, func(t *testing.T) { testSharedIngress(t, component) }) + } +} + +func testSharedIngress(t *testing.T, component string) { + t.Helper() const ( setFlag = "--set" setStringFlag = "--set-string" @@ -59,16 +66,16 @@ func TestSharedControlplaneIngress(t *testing.T) { { name: "encrypted upstream overrides", args: []string{ - setStringFlag, `controlplane.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, - setStringFlag, `controlplane.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, + setStringFlag, component + `.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, + setStringFlag, component + `.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, }, fullname: chartFullname, backendProtocol: "GRPCS", traefikScheme: "https", }, } { t.Run(tc.name, func(t *testing.T) { - serviceName := tc.fullname + "-controlplane" + serviceName := tc.fullname + "-" + component args := append([]string{"template", "test", ".", setFlag, "development=true", - setFlag, "controlplane.ingress.enabled=true", setFlag, "controlplane.ingress.hostname=shared.example.test", + setFlag, component + ".ingress.enabled=true", setFlag, component + ".ingress.hostname=shared.example.test", setFlag, "controlplane.ingressAPI.enabled=true", setFlag, "cas.ingressAPI.enabled=true"}, tc.args...) ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() diff --git a/deployment/chainloop/templates/cas/ingress.yaml b/deployment/chainloop/templates/cas/ingress.yaml index 3506303dc..a68011961 100644 --- a/deployment/chainloop/templates/cas/ingress.yaml +++ b/deployment/chainloop/templates/cas/ingress.yaml @@ -12,13 +12,8 @@ metadata: namespace: {{ include "common.names.namespace" . | quote }} labels: {{- include "chainloop.cas.labels" . | nindent 4 }} {{- if or .Values.cas.ingress.annotations .Values.commonAnnotations }} - annotations: - {{- if .Values.cas.ingress.annotations }} - {{- include "common.tplvalues.render" ( dict "value" .Values.cas.ingress.annotations "context" $) | nindent 4 }} - {{- end }} - {{- if .Values.commonAnnotations }} - {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} - {{- end }} + {{- $annotations := include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.annotations .Values.commonAnnotations) "context" .) }} + annotations: {{- include "common.tplvalues.render" (dict "value" $annotations "context" $) | nindent 4 }} {{- end }} spec: {{- if and .Values.cas.ingress.ingressClassName (eq "true" (include "common.ingress.supportsIngressClassname" .)) }} diff --git a/deployment/chainloop/values.yaml b/deployment/chainloop/values.yaml index 4fb5d843a..cab498af6 100644 --- a/deployment/chainloop/values.yaml +++ b/deployment/chainloop/values.yaml @@ -1129,9 +1129,11 @@ cas: ## ref http://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip ## externalTrafficPolicy: Cluster - ## @param cas.service.annotations Additional custom annotations for cas service + ## @extra cas.service.annotations Annotations for the shared HTTP and gRPC service. Traefik uses h2c upstream. ## - annotations: {} + annotations: + ## @skip cas.service.annotations.traefik.ingress.kubernetes.io/service.serversscheme + traefik.ingress.kubernetes.io/service.serversscheme: h2c ## @param cas.service.extraPorts Extra ports to expose in cas service (normally used with the `sidecars` value) ## extraPorts: [] @@ -1235,7 +1237,7 @@ cas: ## NOTE: You may need to set this to '/*' in order to use this with ALB ingress controllers ## path: / - ## @param cas.ingress.annotations Additional annotations for the Ingress resource. To enable certificate autogeneration, place here your cert-manager annotations. + ## @extra cas.ingress.annotations Annotations for the experimental shared HTTP and gRPC ingress. ## Use this parameter to set the required annotations for cert-manager, see ## ref: https://cert-manager.io/docs/usage/ingress/#supported-annotations ## e.g: @@ -1243,7 +1245,18 @@ cas: ## kubernetes.io/cas.ingress.class: nginx ## cert-manager.io/cluster-issuer: cluster-issuer-name ## - annotations: {} + annotations: + ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/backend-protocol + nginx.ingress.kubernetes.io/backend-protocol: GRPC + ## @skip cas.ingress.annotations.nginx.org/grpc-services + nginx.org/grpc-services: '{{ include "chainloop.cas.fullname" . }}' + ## @skip cas.ingress.annotations.nginx.org/http2 + nginx.org/http2: "true" + ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/proxy-body-size + # Same upload limit as the independent legacy gRPC ingress. Set 0 to disable. + nginx.ingress.kubernetes.io/proxy-body-size: "100m" + ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/client-body-buffer-size + nginx.ingress.kubernetes.io/client-body-buffer-size: "3M" ## @param cas.ingress.tls Enable TLS configuration for the host defined at `controlplane.ingress.hostname` parameter ## TLS certificates will be retrieved from a TLS secret with name: `{{- printf "%s-tls" .Values.controlplane.ingress.hostname }}` ## You can: From dc6b3837d68d5c9420c234bcb0bc48750dd15366 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Mon, 19 Oct 2026 22:41:32 +0200 Subject: [PATCH 02/19] feat(cas): serve HTTP and native gRPC through one cmux endpoint Replace the per-request grpc.Server.ServeHTTP experiment with cmux connection classification: HTTP/2 connections whose first request declares a native gRPC content type go to the existing native gRPC transport, everything else goes to the existing HTTP server. ByteStream keeps the native transport: the adapter experiment peaked at ~1.25 GiB of live heap for a 1 GiB upload against ~28 MiB native, and its downloads were 11.8% slower (PR #3663 benchmark history). The gRPC matcher is the SETTINGS-writing one: grpc-go clients wait for the server's SETTINGS before their first request and a silent matcher deadlocks them. Once the matcher has written, the connection belongs to native gRPC; the HTTP side therefore supports HTTP/1.1 only. A plain HTTP/2 client that falls through acknowledges the injected SETTINGS and net/http rejects it (GOAWAY, PROTOCOL_ERROR). Mixed connections are out of contract; supported proxies keep separate upstream pools. Chart: cas.ingress keeps HTTP/1.1 for the HTTP routes and a second Ingress from cas.ingress.grpc routes the /cas.v1.* paths with the gRPC backend protocol and the upload body limits. The Traefik h2c Service annotation default is removed (per-Service scheme cannot differ per upstream pool). Spec (issue-3628-single-endpoint.md): D-016 is superseded by D-017 (the cmux SETTINGS contract; the D-016 replay/ACK removal is not implementable with the cmux library), D-018 records the chart path-split, and R-009 supersedes R-001 for the CAS. A fresh 1 GiB benchmark through the shared endpoint matches the native listener (median upload +1.6%, download -2.2%, heap parity). Refs #3628, stacked on #3647. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Assisted-by: Claude Code Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/README.md | 28 +-- app/artifact-cas/configs/config.devel.yaml | 3 +- app/artifact-cas/configs/samples/config.yaml | 3 +- .../experiments/shared-endpoint-benchmark.txt | 46 +++-- app/artifact-cas/internal/server/grpc.go | 8 +- app/artifact-cas/internal/server/http.go | 191 +++++++++++++++--- app/artifact-cas/internal/server/http_test.go | 70 +++++-- deployment/chainloop/README.md | 6 +- deployment/chainloop/chart_test.go | 187 ++++++++++++----- .../templates/cas/ingress-grpc-shared.yaml | 61 ++++++ deployment/chainloop/values.yaml | 62 ++++-- docs/specs/issue-3628-single-endpoint.md | 61 ++++-- go.mod | 1 + go.sum | 3 + 14 files changed, 564 insertions(+), 166 deletions(-) create mode 100644 deployment/chainloop/templates/cas/ingress-grpc-shared.yaml diff --git a/app/artifact-cas/README.md b/app/artifact-cas/README.md index 6237d5b9e..24b79a75b 100644 --- a/app/artifact-cas/README.md +++ b/app/artifact-cas/README.md @@ -70,11 +70,11 @@ go run main.go --insecure artifact download -d [sha256:...] ### Shared endpoint experiment -This branch extends the control-plane HTTP dispatch approach to CAS for local evaluation. The existing HTTP port, `8001`, accepts HTTP/1.1, HTTP/2, and native gRPC. Every request is dispatched independently: HTTP/2 requests with a native gRPC content type use `grpc.Server.ServeHTTP`; other requests use the existing HTTP status and authenticated download handlers. CAS does not add gRPC-Web or Connect. +The CAS HTTP port, `8001`, is the shared endpoint. cmux reads the first request of each new connection and hands the whole connection to one of the two existing transports: HTTP/2 connections whose first request declares a native gRPC content type go to the native gRPC transport (`grpc.Server.Serve`), everything else goes to the existing HTTP server. ByteStream therefore keeps the grpc-go native transport; the earlier per-request `grpc.Server.ServeHTTP` experiment on this branch was rejected because its uploads peaked at ~1.25 GiB of live heap for a 1 GiB artifact (see the results history below and the PR #3663 discussion). CAS does not add gRPC-Web or Connect. -The native gRPC listener on `9001` remains enabled by default, with its existing independent TLS configuration. The same ByteStream and resource services, JWT checks, role checks, validators, and metrics interceptors serve both listeners. `server.http.timeout` continues to apply to HTTP routes; ByteStream requests bypass the HTTP route middleware. HTTP drains before its gRPC handler stops. +The gRPC matcher is the SETTINGS-writing one: grpc-go clients send their preface and SETTINGS and then wait for the server's SETTINGS before their first request, so a silent matcher deadlocks them. Once the matcher has written, the connection belongs to the native gRPC server. The HTTP side of the shared endpoint therefore supports HTTP/1.1 only: a plain HTTP/2 client that falls through to the HTTP server acknowledges the injected SETTINGS frame, which `net/http` rejects (`GOAWAY`, `PROTOCOL_ERROR`). Mixed native-gRPC and plain-HTTP requests on one upstream HTTP/2 connection are not supported; supported proxies keep the two kinds of traffic on separate upstream pools, which the chart configures (spec D-017/D-018, recorded in [issue-3628-single-endpoint](../../docs/specs/issue-3628-single-endpoint.md)). -This is an experiment against the original CAS design in [issue-3628-single-endpoint](../../docs/specs/issue-3628-single-endpoint.md), particularly its D-014 through D-016 native-transport design. The control-plane replacement spec defers CAS. Neither spec is amended by this branch: a CAS transport decision and the original R-002/R-003 qualification remain required before migration. The HTTP adapter can read ahead while the application is paused, so functional streaming tests alone cannot establish bounded memory. +The native gRPC listener on `9001` remains enabled by default, with its existing independent TLS configuration. The same ByteStream and resource services, JWT checks, role checks, validators, and metrics interceptors serve both listeners. `server.http.timeout` continues to apply to HTTP routes; ByteStream requests bypass the HTTP route middleware. HTTP drains before the gRPC transport stops. To test with your existing development infrastructure: @@ -90,9 +90,9 @@ go run ./app/cli --insecure --artifact-cas localhost:8001 artifact download -d s The development CAS public key and credentials service must match the control plane that issues these temporary CAS tokens. If another CAS already occupies these ports, start this binary with a copy of your CAS configuration and unused addresses. Do not replace the running service or edit the CLI config just to run a comparison. -Change only the per-command override to `localhost:9001` to compare with the native listener. Once all direct clients and the control-plane CAS connection have been moved, `server.legacy_grpc_listener: false` disables the native listener. Leave it enabled during evaluation. Optional `server.http.tls_config` configures TLS for the shared endpoint; both certificate and private key must be set. It does not inherit `server.grpc.tls_config`. +Change only the per-command override to `localhost:9001` to compare with the native listener. Once all direct clients and the control-plane CAS connection have been moved, `server.legacy_grpc_listener: false` disables the native listener. Leave it enabled during evaluation. Optional `server.http.tls_config` configures TLS for the shared endpoint; both certificate and private key must be set. It does not inherit `server.grpc.tls_config`. Note that a browser downloading directly from a TLS-configured shared port is not supported: the browser negotiates HTTP/2 through ALPN and the connection reaches the HTTP server with an injected SETTINGS frame. Serve browser downloads through the ingress. -The chart's `cas.ingress` targets the shared HTTP Service with HTTP/2 upstream defaults for ingress-nginx, F5 NGINX, and Traefik, matching the control-plane settings. `cas.ingressAPI` and `cas.serviceAPI` retain their existing settings. The control plane still targets the legacy CAS Service. The shared ingress retains the native ingress's `100m` body limit and `3M` client body buffer default; override the body limit to `0` or a suitable larger limit for a 1 GB ingress experiment. Deployed controller qualification remains outstanding. +The chart's `cas.ingress` routes the HTTP paths (`/download`, `/statusz`, `/infoz`) with an HTTP/1.1 upstream, and a second Ingress rendered from `cas.ingress.grpc` routes the `/cas.v1.*` RPC paths with the gRPC backend protocol. The gRPC Ingress keeps the native ingress's `100m` body limit and `3M` client body buffer default; override the body limit to `0` or a suitable larger limit for a 1 GB ingress experiment. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS Service keep their existing settings. Deployed controller qualification (F5 same-host Ingress conflicts, Traefik's per-Service scheme) remains outstanding. Run the isolated functional checks without Vault or an object store: @@ -100,7 +100,7 @@ Run the isolated functional checks without Vault or an object store: go test -race ./app/artifact-cas/internal/server ./app/artifact-cas/cmd ./deployment/chainloop -count=1 ``` -Compare real ByteStream upload and download paths over TCP with a synthetic 1 GiB zero-filled artifact, streamed from a fixed-size buffer and checked by SHA256. Both transports use the same file-backed test backend and real CAS staging and authentication. Select a disk-backed temporary directory; `/tmp` can be tmpfs. +Compare real ByteStream upload and download paths over TCP with a synthetic 1 GiB zero-filled artifact, streamed from a fixed-size buffer and checked by SHA256. Both listeners use the same file-backed test backend and real CAS staging and authentication. Select a disk-backed temporary directory; `/tmp` can be tmpfs. ```sh mkdir -p "$HOME/.cache/chainloop-cas-bench" @@ -113,18 +113,18 @@ TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/se #### Observed local results -Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, Linux amd64, disk-backed btrfs staging. These medians come from the [raw benchmark output](experiments/shared-endpoint-benchmark.txt). +Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, cmux 0.1.5, Linux amd64, disk-backed btrfs staging. These medians come from the [raw benchmark output](experiments/shared-endpoint-benchmark.txt). -| Transport | Operation | Throughput (MB/s) | Sampled peak live heap (MiB) | +| Listener | Operation | Throughput (MB/s) | Sampled peak live heap (MiB) | | --- | --- | ---: | ---: | -| Native gRPC | Upload | 240.84 | 27.97 | -| HTTP adapter | Upload | 232.00 | 1278.00 | -| Native gRPC | Download | 197.03 | 31.97 | -| HTTP adapter | Download | 173.83 | 38.92 | +| Native gRPC (`:9001`) | Upload | 228.90 | 29.27 | +| Shared endpoint (`:8001`, cmux) | Upload | 232.51 | 27.97 | +| Native gRPC (`:9001`) | Download | 194.10 | 30.90 | +| Shared endpoint (`:8001`, cmux) | Download | 189.74 | 31.83 | -The HTTP-adapter upload used approximately 1.25 GiB of combined client/server heap for a 1 GiB artifact, while the native upload stayed around 28 MiB. This is a material memory regression even with an actively reading application. The sampled peak includes pooled and queued buffers and does not identify every allocation's owner. Source inspection is consistent with server read-ahead: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) and queues data into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90). +Both listeners use the native grpc-go transport, and the numbers match: median upload throughput on the shared endpoint was 1.6% higher and median download throughput 2.2% lower than the legacy listener, with sampled peak live heap equivalent (~28-32 MiB) on both. This satisfies the R-003 transport-comparison shape for the shared endpoint — within 10% with bounded memory — in this synthetic local setup. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production. -Median upload throughput was 3.7% lower and download throughput 11.8% lower on the adapter in this run. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production. The memory result prevents recommending this shared CAS adapter for migration. The native endpoint remains the comparison baseline; use this branch to reproduce the result and evaluate another transport before changing production CAS clients. +For contrast, the earlier per-request `ServeHTTP` experiment on this branch measured 232.00 MB/s upload and 173.83 MB/s download against the same native baseline, with an upload peak of ~1278 MiB: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90), so read-ahead is not bounded by the application's pace. That memory regression rejected the adapter for CAS uploads (the control plane keeps the adapter because its requests are small and unary) and motivated the cmux design recorded in D-017. ### Run tests diff --git a/app/artifact-cas/configs/config.devel.yaml b/app/artifact-cas/configs/config.devel.yaml index 863da69fe..f30fdd315 100644 --- a/app/artifact-cas/configs/config.devel.yaml +++ b/app/artifact-cas/configs/config.devel.yaml @@ -6,7 +6,8 @@ server: # Keep 9001 available to compare native gRPC with the shared 8001 endpoint. legacy_grpc_listener: true http: - # Serves HTTP/1.1, HTTP/2, and native gRPC over h2c. + # Shared endpoint: HTTP/1.1 and native gRPC. cmux classifies each + # connection by its first request; HTTP/2 reaches it only as native gRPC. addr: 0.0.0.0:8001 # Timeouts for http downloads # grpc downloads/uploads don't require this because they don't have timeouts diff --git a/app/artifact-cas/configs/samples/config.yaml b/app/artifact-cas/configs/samples/config.yaml index 7465950cb..5a26fe573 100644 --- a/app/artifact-cas/configs/samples/config.yaml +++ b/app/artifact-cas/configs/samples/config.yaml @@ -8,7 +8,8 @@ server: http: addr: 0.0.0.0:8001 timeout: 300s - # Optional TLS for both HTTP and native gRPC on port 8001. Independent of grpc.tls_config. + # Shared endpoint: HTTP/1.1 and native gRPC (cmux). HTTP/2 is native gRPC only. + # Optional TLS for both transports on port 8001. Independent of grpc.tls_config. # tls_config: # certificate: "./configs/tls/shared.crt" # private_key: "./configs/tls/shared.key" diff --git a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt index f0ccd4d6a..eb439f0e5 100644 --- a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt +++ b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt @@ -1,37 +1,45 @@ # Copyright 2026 The Chainloop Authors. # SPDX-License-Identifier: Apache-2.0 +# cmux shared endpoint vs the legacy native listener. Both transports use the +# grpc-go native transport; the shared endpoint classifies each connection +# with cmux (spec D-017). An earlier run on this branch measured the +# grpc.Server.ServeHTTP HTTP adapter instead: its uploads peaked at +# ~1278 MiB of combined live heap for a 1 GiB artifact and its downloads +# were 11.8% slower; that result rejected the adapter for CAS. The raw +# numbers of that run are preserved in the PR #3663 discussion. + go version go1.27.2 linux/amd64 -grpc-go: v1.84.0; Kratos: v2.9.2 +grpc-go: v1.84.0; Kratos: v2.9.2; cmux: v0.1.5 Temporary filesystem: btrfs (not tmpfs) -Command: go test ./app/artifact-cas/internal/server -run ^$ -bench ^BenchmarkCASByteStreamTransport$ -benchtime=1x -count=3 -timeout=10m +Command: TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/server -run ^$ -bench ^BenchmarkCASByteStreamTransport$ -benchtime=1x -count=3 -timeout=10m Artifact: 1 GiB; plaintext localhost TCP; real CAS services/auth/staging; disk-backed test backend; client and server in one process. -INFO msg=[HTTP] server listening on: 127.0.0.1:39645 -INFO msg=[gRPC] server listening on: 127.0.0.1:35009 +INFO msg=[HTTP] server listening on: 127.0.0.1:40239 +INFO msg=[gRPC] server listening on: 127.0.0.1:44469 goos: linux goarch: amd64 pkg: github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server cpu: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4428889185 ns/op 242.44 MB/s 26.92 peak-heap-MiB 1453444272 B/op 3158778 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4458294797 ns/op 240.84 MB/s 28.33 peak-heap-MiB 1432098544 B/op 3132177 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4502562403 ns/op 238.47 MB/s 27.97 peak-heap-MiB 1445793976 B/op 3132288 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5747630684 ns/op 186.81 MB/s 31.97 peak-heap-MiB 1373396504 B/op 84976 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5449605338 ns/op 197.03 MB/s 31.38 peak-heap-MiB 1401101656 B/op 80266 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5371327380 ns/op 199.90 MB/s 32.44 peak-heap-MiB 1379661432 B/op 80201 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 5207742134 ns/op 206.18 MB/s 30.03 peak-heap-MiB 1442550792 B/op 3204540 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4690862475 ns/op 228.90 MB/s 29.27 peak-heap-MiB 1477528368 B/op 3032897 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4473314156 ns/op 240.03 MB/s 29.11 peak-heap-MiB 1509947992 B/op 3032880 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 6035943310 ns/op 177.89 MB/s 31.86 peak-heap-MiB 1408894040 B/op 95728 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5531984200 ns/op 194.10 MB/s 30.90 peak-heap-MiB 1359969664 B/op 86333 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5476786936 ns/op 196.05 MB/s 30.05 peak-heap-MiB 1312724880 B/op 84305 allocs/op INFO msg=[gRPC] server stopping INFO msg=[HTTP] server stopping INFO msg=[gRPC] server stopping -INFO msg=[gRPC] server listening on: 127.0.0.1:36189 -INFO msg=[HTTP] server listening on: 127.0.0.1:43493 -BenchmarkCASByteStreamTransport/shared/upload-8 1 4757267201 ns/op 225.71 MB/s 1244 peak-heap-MiB 2002041336 B/op 3626317 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 4626278404 ns/op 232.10 MB/s 1304 peak-heap-MiB 2031947176 B/op 3636468 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 4628108813 ns/op 232.00 MB/s 1278 peak-heap-MiB 2034461720 B/op 3638817 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 6451314190 ns/op 166.44 MB/s 38.92 peak-heap-MiB 1246487112 B/op 314413 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 6177002391 ns/op 173.83 MB/s 39.37 peak-heap-MiB 1267943360 B/op 314523 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5987610343 ns/op 179.33 MB/s 37.41 peak-heap-MiB 1259259368 B/op 307480 allocs/op +INFO msg=[gRPC] server listening on: 127.0.0.1:39939 +INFO msg=[HTTP] server listening on: 127.0.0.1:38963 +BenchmarkCASByteStreamTransport/shared/upload-8 1 4691261582 ns/op 228.88 MB/s 27.97 peak-heap-MiB 1432139600 B/op 3225369 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 4618093234 ns/op 232.51 MB/s 27.54 peak-heap-MiB 1470447088 B/op 3224874 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 4550823731 ns/op 235.94 MB/s 28.81 peak-heap-MiB 1455017664 B/op 3153945 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5921459129 ns/op 181.33 MB/s 31.83 peak-heap-MiB 1373096000 B/op 95393 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5659083336 ns/op 189.74 MB/s 32.92 peak-heap-MiB 1332496448 B/op 85621 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5506534391 ns/op 194.99 MB/s 29.10 peak-heap-MiB 1313525896 B/op 78316 allocs/op INFO msg=[gRPC] server stopping INFO msg=[HTTP] server stopping INFO msg=[gRPC] server stopping PASS -ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 65.296s +ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 65.136s diff --git a/app/artifact-cas/internal/server/grpc.go b/app/artifact-cas/internal/server/grpc.go index 5c7eae93a..3d659b9fa 100644 --- a/app/artifact-cas/internal/server/grpc.go +++ b/app/artifact-cas/internal/server/grpc.go @@ -64,8 +64,12 @@ type Opts struct { Logger log.Logger } -// GRPCServers keeps the HTTP handler separate from the native transport. -// Handler must never be started with Serve; the HTTP server owns its streams. +// GRPCServers keeps one gRPC server for the shared endpoint separate from the +// optional legacy listener. Handler never binds a socket of its own: cmux +// hands it the native-gRPC connections of the shared HTTP listener, and the +// HTTP server dispatches its per-request gRPC fallback to the same server +// through grpc-go's HTTP adapter. Both listeners therefore run the same +// services and interceptors. type GRPCServers struct { Handler *grpc.Server Legacy *grpc.Server diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 89248a822..0b95e93e9 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -20,7 +20,9 @@ import ( "crypto/tls" "errors" "fmt" + "net" h "net/http" + "net/url" "strings" "time" @@ -29,25 +31,91 @@ import ( "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/service" casJWT "github.com/chainloop-dev/chainloop/internal/robotaccount/cas" middlewares_http "github.com/chainloop-dev/chainloop/pkg/middlewares/http" + "github.com/go-kratos/kratos/v2/log" "github.com/go-kratos/kratos/v2/middleware/logging" "github.com/go-kratos/kratos/v2/middleware/recovery" "github.com/go-kratos/kratos/v2/transport/grpc" "github.com/go-kratos/kratos/v2/transport/http" "github.com/golang-jwt/jwt/v5" + grpcLib "google.golang.org/grpc" + + "github.com/soheilhy/cmux" ) -// APIServer drains HTTP streams before stopping the gRPC handler. +// APIServer owns the shared CAS listener on server.http.addr. cmux reads the +// first request of each new connection and hands the whole connection to one +// of the two existing transports: +// +// native gRPC (HTTP/2 + application/grpc*) -> grpc.Server.Serve (native transport) +// everything else -> the existing Kratos HTTP server +// +// ByteStream therefore keeps the grpc-go native transport, so large uploads +// stream with bounded memory exactly as they do on the legacy listener. This +// is the separate-connection contract of the spec's D-014/D-015: a pooled +// upstream connection must stay single-protocol, and the chart routes the gRPC +// paths through their own HTTP/2 upstream pool. A connection that cmux gave to +// the HTTP server can still carry a later native gRPC request through the +// per-request ServeHTTP fallback below; the reverse is not supported. type APIServer struct { - *http.Server - grpcHandler *grpc.Server + lis net.Listener + mux cmux.CMux + grpcL net.Listener + httpL net.Listener + grpcSrv *grpc.Server + httpSrv *http.Server + scheme string } +// Start serves the shared endpoint until ctx is cancelled or a transport +// fails. The HTTP server serves the connections cmux matched to it, the +// native gRPC server serves its own matched connections, and cmux keeps +// accepting new ones. Each goroutine reports through errCh; an orderly +// Stop closes the listener and all three return ordinary shutdown errors. +func (s *APIServer) Start(ctx context.Context) error { + errCh := make(chan error, 3) + go func() { errCh <- s.httpSrv.Start(ctx) }() + go func() { errCh <- s.grpcSrv.Server.Serve(s.grpcL) }() + go func() { errCh <- s.mux.Serve() }() + select { + case <-ctx.Done(): + return nil + case err := <-errCh: + if isOrdinaryStopError(err) { + return nil + } + return err + } +} + +// Stop closes the shared listener, then drains the HTTP server (including its +// per-request gRPC fallback streams) before the native gRPC transport stops +// its in-flight streams. func (s *APIServer) Stop(ctx context.Context) error { - httpErr := s.Server.Stop(ctx) - return errors.Join(httpErr, s.grpcHandler.Stop(ctx)) + // Stop accepting new connections on the shared endpoint. cmux.Close only + // marks the match listeners closed: active connections keep draining. + closeErr := s.lis.Close() + s.mux.Close() + httpErr := s.httpSrv.Stop(ctx) + // http.Server.Shutdown closes its listener, which is the cmux mux + // listener wrapping the root listener this Stop already closed. That + // double close surfaces net.ErrClosed; it is not a serving failure. + if httpErr != nil && errors.Is(httpErr, net.ErrClosed) { + httpErr = nil + } + grpcErr := s.grpcSrv.Stop(ctx) + if closeErr != nil && !errors.Is(closeErr, net.ErrClosed) { + return errors.Join(closeErr, httpErr, grpcErr) + } + return errors.Join(httpErr, grpcErr) } -// NewHTTPServer dispatches every request on the existing CAS HTTP endpoint. +// Endpoint returns the real bound address of the shared endpoint. +func (s *APIServer) Endpoint() (*url.URL, error) { + return &url.URL{Scheme: s.scheme, Host: s.lis.Addr().String()}, nil +} + +// NewHTTPServer binds the shared CAS endpoint and classifies each new +// connection by its first request through cmux. func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) { c, authConf := settings.ServerConfig, settings.AuthConfig downloadSvc, providers, logger := settings.DownloadSvc, settings.Providers, settings.Logger @@ -58,26 +126,54 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) if err != nil { return nil, err } + + network := c.Http.GetNetwork() + if network == "" { + network = "tcp" + } + lis, err := net.Listen(network, c.Http.GetAddr()) + if err != nil { + return nil, fmt.Errorf("binding the shared CAS endpoint %q: %w", c.Http.GetAddr(), err) + } + scheme := "http" + if tlsConfig != nil { + // Terminate TLS before cmux so one certificate covers both + // transports, with ALPN offering HTTP/2 and HTTP/1.1. + lis = tls.NewListener(lis, tlsConfig) + scheme = "https" + } + + mux := cmux.New(lis) + // Native gRPC: match HTTP/2 connections whose first request declares a + // native gRPC content type (application/grpc, application/grpc+proto, + // ...). The SETTINGS-writing writer unblocks gRPC clients, which send + // their preface and SETTINGS and then wait for the server's SETTINGS + // before their first HEADERS frame (Argo CD ships this matcher for the + // same reason). Once it writes, the connection belongs to the native + // gRPC server: cmux documents that it cannot be handed to the HTTP + // server afterwards. CAS has no gRPC-Web endpoint (spec D-011), so no + // application/grpc-web* exclusion is needed. + grpcL := mux.MatchWithWriters(cmux.HTTP2MatchHeaderFieldPrefixSendSettings("content-type", "application/grpc")) + // Every other connection (HTTP/1.1 and plain HTTP/2) belongs to the + // HTTP server. Mixed native-gRPC and plain-HTTP requests on one shared + // upstream HTTP/2 connection are not supported (spec D-014): supported + // proxies keep the two kinds of traffic on separate upstream pools, + // which is what the chart's path-split ingress configures. + httpL := mux.Match(cmux.Any()) + var opts = []http.ServerOption{ + // The HTTP server never binds its own socket: it serves the + // connections cmux matched to it on the shared listener. + http.Listener(httpL), http.Middleware( recovery.Recovery(), logging.Server(logger), ), } - if c.Http.Network != "" { - opts = append(opts, http.Network(c.Http.Network)) - } - if c.Http.Addr != "" { - opts = append(opts, http.Address(c.Http.Addr)) - } if c.Http.Timeout != nil { opts = append(opts, http.Timeout(c.Http.Timeout.AsDuration())) } - if tlsConfig != nil { - opts = append(opts, http.TLSConfig(tlsConfig)) - } - // Parse the public key once on initialization instead of on every request // TODO: implement jwks endpoint publicKey, err := parsePublicKey(authConf, logger) @@ -90,21 +186,48 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) downloadHandler := middlewares_http.AuthFromHeaderOrQueryParam(loadPublicKey(publicKey), claimsFunc(), casJWT.SigningMethod, downloadSvc) srv.Handle(service.DownloadPath, CORSMiddleware(c.GetHttp().GetCors().GetAllowOrigins(), downloadHandler)) api.RegisterStatusServiceHTTPServer(srv, service.NewStatusService(Version, providers)) - shared := http.NewServer(opts...) - shared.Protocols = new(h.Protocols) - shared.Protocols.SetHTTP1(true) - shared.Protocols.SetHTTP2(true) - shared.Protocols.SetUnencryptedHTTP2(true) - shared.ReadHeaderTimeout = 60 * time.Second - shared.Handler = h.HandlerFunc(func(w h.ResponseWriter, r *h.Request) { - contentType := strings.ToLower(r.Header.Get("Content-Type")) - if r.ProtoMajor == 2 && strings.HasPrefix(contentType, "application/grpc") && !strings.HasPrefix(contentType, "application/grpc-web") { + + routes := srv.Handler + srv.Protocols = new(h.Protocols) + srv.Protocols.SetHTTP1(true) + srv.Protocols.SetHTTP2(true) + srv.Protocols.SetUnencryptedHTTP2(true) + // Bound how long a client may take to deliver its request headers + // (slowloris protection). Response streaming is not affected. + srv.ReadHeaderTimeout = 60 * time.Second + // One-way fallback (spec D-014): a connection cmux classified as HTTP + // may later carry a native gRPC request, for example from a proxy that + // pools both kinds of streams onto one upstream HTTP/2 connection. + // Dispatch those per request through grpc-go's HTTP adapter, with the + // same registered services and interceptors as the native transport. + srv.Handler = h.HandlerFunc(func(w h.ResponseWriter, r *h.Request) { + if r.ProtoMajor == 2 && isNativeGRPCContentType(r.Header.Get("Content-Type")) { grpcServers.Handler.ServeHTTP(w, r) return } - srv.ServeHTTP(w, r) + routes.ServeHTTP(w, r) }) - return &APIServer{Server: shared, grpcHandler: grpcServers.Handler}, nil + + log.NewHelper(logger).Infof("[shared] CAS endpoint (%s) listening on: %s", scheme, lis.Addr().String()) + + return &APIServer{ + lis: lis, + mux: mux, + grpcL: grpcL, + httpL: httpL, + grpcSrv: grpcServers.Handler, + httpSrv: srv, + scheme: scheme, + }, nil +} + +// isNativeGRPCContentType reports whether a content type selects the native +// gRPC protocol family, excluding gRPC-Web. CAS has no gRPC-Web endpoint, but +// the exclusion keeps the fallback consistent with the control plane. +func isNativeGRPCContentType(contentType string) bool { + contentType = strings.ToLower(contentType) + return strings.HasPrefix(contentType, "application/grpc") && + !strings.HasPrefix(contentType, "application/grpc-web") } // claimsFunc returns the claims function for the JWT middleware that casts the claims to the correct type @@ -114,6 +237,18 @@ func claimsFunc() middlewares_http.ClaimsFunc { } } +// isOrdinaryStopError reports whether a transport error is the expected +// result of an orderly Stop rather than a serving failure. +func isOrdinaryStopError(err error) bool { + if err == nil { + return true + } + return errors.Is(err, net.ErrClosed) || + errors.Is(err, h.ErrServerClosed) || + errors.Is(err, cmux.ErrServerClosed) || + errors.Is(err, grpcLib.ErrServerStopped) +} + // loadServerTLSConfig applies TLS independently of the legacy gRPC listener. func loadServerTLSConfig(c *conf.Server_TLS) (*tls.Config, error) { cert, key := c.GetCertificate(), c.GetPrivateKey() @@ -131,4 +266,4 @@ func loadServerTLSConfig(c *conf.Server_TLS) (*tls.Config, error) { Certificates: []tls.Certificate{pair}, MinVersion: tls.VersionTLS12, NextProtos: []string{"h2", "http/1.1"}, }, nil -} +} \ No newline at end of file diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index 64f8b98d7..088ce02f6 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -305,7 +305,10 @@ func TestCASByteStreamTransportParity(t *testing.T) { require.Equal(t, codes.InvalidArgument, status.Code(err)) // The HTTP download remains usable regardless of which RPC listener uploaded the blob. - tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // test certificate + // HTTP/1.1 only (D-017/R-009): the HTTP side of the shared + // endpoint does not take plain HTTP/2 connections. + tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // test certificate + tr.Protocols.SetHTTP1(true) defer tr.CloseIdleConnections() req, err := http.NewRequestWithContext(ctx, http.MethodGet, sharedURL+"/download/sha256:"+digest, nil) require.NoError(t, err) @@ -339,6 +342,14 @@ func casTestTLS(t testing.TB) *conf.Server_TLS { return &conf.Server_TLS{Certificate: certPath, PrivateKey: keyPath} } +// TestCASSharedConnectionReuse checks the separate-connection contract of the +// shared endpoint (spec D-014/D-015): through one port, native gRPC and HTTP +// each work on their own connections, a browser CORS preflight reaches the +// HTTP side, and an HTTP-owned HTTP/2 connection can still carry a later +// native gRPC request through the per-request ServeHTTP fallback. The reverse +// order, plain HTTP on a gRPC-classified connection, is not part of the +// contract; a proxy must keep the two kinds of traffic on separate upstream +// pools, which is what the chart's path-split ingress configures. func TestCASSharedConnectionReuse(t *testing.T) { for _, useTLS := range []bool{false, true} { t.Run(map[bool]string{false: "h2c", true: "TLS"}[useTLS], func(t *testing.T) { @@ -347,18 +358,32 @@ func TestCASSharedConnectionReuse(t *testing.T) { opts.ServerConfig.Http.TlsConfig = casTestTLS(t) } sharedURL, _ := startCASTestServers(t, opts) - var connections atomic.Int32 - tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // test certificate - DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { - connections.Add(1) - return (&net.Dialer{}).DialContext(ctx, network, addr) - }, - } - tr.Protocols.SetHTTP2(true) - tr.Protocols.SetUnencryptedHTTP2(true) - defer tr.CloseIdleConnections() - client := &http.Client{Transport: tr, Timeout: 5 * time.Second} - for _, kind := range []string{casGRPCRequest, casHTTPRequest, casGRPCRequest, casPreflightRequest, casHTTPRequest, casGRPCRequest} { + + perform := func(kind string) { + var connections atomic.Int32 + tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // test certificate + DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { + connections.Add(1) + return (&net.Dialer{}).DialContext(ctx, network, addr) + }, + } + if kind == casGRPCRequest { + // Native gRPC arrives over HTTP/2, as the gRPC upstream + // pool of a configured ingress dials it. + tr.Protocols.SetHTTP2(true) + tr.Protocols.SetUnencryptedHTTP2(true) + } else { + // D-017/R-009: the HTTP side of the shared endpoint + // supports HTTP/1.1 only. A plain HTTP/2 client + // acknowledges the matcher's injected SETTINGS frame, + // and net/http rejects that acknowledgement (GOAWAY, + // PROTOCOL_ERROR). The HTTP upstream pool therefore dials + // HTTP/1.1 (D-018). + tr.Protocols.SetHTTP1(true) + } + defer tr.CloseIdleConnections() + client := &http.Client{Transport: tr, Timeout: 5 * time.Second} + path, method := "/cas.v1.StatusService/Statusz", http.MethodPost if kind == casHTTPRequest { path, method = "/statusz", http.MethodGet @@ -386,12 +411,27 @@ func TestCASSharedConnectionReuse(t *testing.T) { expected = http.StatusNoContent } require.Equal(t, expected, resp.StatusCode, kind) - require.Equal(t, 2, resp.ProtoMajor) if kind == casGRPCRequest { + require.Equal(t, 2, resp.ProtoMajor) require.Equal(t, "0", resp.Trailer.Get("Grpc-Status")) + } else { + // D-017/R-009: the HTTP side of the shared endpoint is HTTP/1.1. + require.Equal(t, 1, resp.ProtoMajor) } } - require.EqualValues(t, 1, connections.Load()) + + // Each protocol works through the shared port on its own + // connections, as a proxy with separate upstream pools sends them. + // A plain HTTP/2 connection cannot fall through to the HTTP side: + // D-017 records that the matcher's SETTINGS injection makes + // net/http reject the client's acknowledgement (GOAWAY, + // PROTOCOL_ERROR), so mixed native-gRPC and plain-HTTP reuse of + // one upstream HTTP/2 connection is unsupported by contract. + perform(casGRPCRequest) + perform(casHTTPRequest) + perform(casPreflightRequest) + perform(casHTTPRequest) + perform(casGRPCRequest) }) } } diff --git a/deployment/chainloop/README.md b/deployment/chainloop/README.md index efc38553e..320c0d47f 100644 --- a/deployment/chainloop/README.md +++ b/deployment/chainloop/README.md @@ -245,7 +245,7 @@ The control plane ingress serves HTTP, gRPC-Web, and native gRPC on the same hos Its HTTP server routes each request, so these protocols can share an upstream HTTP/2 connection. The existing gRPC ingress stays available. Existing clients can keep their current endpoints. To test the shared endpoint, point a gRPC client to the control plane HTTP hostname on port 443. -On the CAS experiment branch, `cas.ingress` and `cas.service` use the same HTTP/2 upstream settings for native gRPC and HTTP downloads. `cas.ingressAPI` and the native gRPC endpoint remain independent, and the control plane continues to use that legacy endpoint. CAS does not add gRPC-Web or Connect. The shared CAS transport still needs streaming and performance qualification before migration; see the [local experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). +The CAS shared endpoint classifies each connection with cmux and keeps the native gRPC transport for ByteStream (spec D-017). One hostname serves both protocols through two upstream pools: `cas.ingress` routes the HTTP paths (`/download`, `/statusz`, `/infoz`) with an HTTP/1.1 upstream, and a second Ingress rendered from `cas.ingress.grpc` routes the `/cas.v1.*` RPC paths with the gRPC backend protocol. Do not set a gRPC backend protocol in `cas.ingress.annotations`: that object belongs to the HTTP routes. The gRPC-pool protocol is configured in `cas.ingress.grpc.annotations` (set `backend-protocol: GRPCS` there when TLS runs to the pod). `cas.ingressAPI` and the native gRPC endpoint remain independent, and the control plane continues to use that legacy endpoint during migration. CAS does not add gRPC-Web or Connect. Controller qualification (F5 VirtualServer for the same-host split, Traefik's per-Service scheme limitation) and the full streaming-performance evidence remain open; see the [local experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). The chart configures HTTP/2 upstream traffic for these controllers: @@ -843,7 +843,7 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | `cas.service.loadBalancerIP` | cas service Load Balancer IP | `""` | | `cas.service.loadBalancerSourceRanges` | cas service Load Balancer sources | `[]` | | `cas.service.externalTrafficPolicy` | cas service external traffic policy | `Cluster` | -| `cas.service.annotations` | Annotations for the shared HTTP and gRPC service. Traefik uses h2c upstream. | See values.yaml | +| `cas.service.annotations` | Additional custom annotations for the cas service. Do not set a Traefik h2c scheme here: the HTTP routes must dial HTTP/1.1 (spec D-017/D-018). | `{}` | | `cas.service.extraPorts` | Extra ports to expose in cas service (normally used with the `sidecars` value) | `[]` | | `cas.service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` | | `cas.service.sessionAffinityConfig` | Additional settings for the sessionAffinity | `{}` | @@ -865,7 +865,7 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | `cas.ingress.hostname` | Default host for the ingress record | `cas.dev.local` | | `cas.ingress.ingressClassName` | IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+) | `""` | | `cas.ingress.path` | Default path for the ingress record | `/` | -| `cas.ingress.annotations` | Annotations for the experimental shared HTTP and gRPC ingress. | See values.yaml | +| `cas.ingress.annotations` | Annotations for the shared CAS HTTP ingress (downloads, status). gRPC upstream settings go in `cas.ingress.grpc.annotations`. | `{}` | | `cas.ingress.tls` | Enable TLS configuration for the host defined at `controlplane.ingress.hostname` parameter | `false` | | `cas.ingress.selfSigned` | Create a TLS secret for this ingress record using self-signed certificates generated by Helm | `false` | | `cas.ingress.extraHosts` | An array with additional hostname(s) to be covered with the ingress record | `[]` | diff --git a/deployment/chainloop/chart_test.go b/deployment/chainloop/chart_test.go index 837252de0..b1dbdc6f2 100644 --- a/deployment/chainloop/chart_test.go +++ b/deployment/chainloop/chart_test.go @@ -28,27 +28,57 @@ import ( "gopkg.in/yaml.v3" ) -// Verify the rendered resources, including templated F5 service names and -// overrides. The legacy gRPC resources keep their own upstreams. +// The control plane serves every protocol per request on its shared ingress +// (replacement spec D-001/D-002), so the whole ingress uses the gRPC backend +// protocol and the Traefik h2c scheme. The CAS classifies connections with +// cmux and its native transport (D-017), so its shared HTTP ingress keeps +// HTTP/1.1 and the gRPC RPC paths are split into a second Ingress with their +// own upstream pool (D-018). The legacy gRPC resources keep their own upstreams. func TestSharedIngress(t *testing.T) { - for _, component := range []string{"controlplane", "cas"} { - t.Run(component, func(t *testing.T) { testSharedIngress(t, component) }) - } + t.Run("controlplane", func(t *testing.T) { testSharedControlplaneIngress(t) }) + t.Run("cas", func(t *testing.T) { testSharedCASIngress(t) }) } -func testSharedIngress(t *testing.T, component string) { +func renderChart(t *testing.T, args []string) map[string]chartResource { t.Helper() + helm, err := exec.LookPath("helm") + if err != nil { + t.Skip("helm is required for chart rendering tests") + } + full := append([]string{"template", "test", ".", "--set", "development=true", + "--set", "controlplane.ingressAPI.enabled=true", "--set", "cas.ingressAPI.enabled=true"}, args...) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, helm, full...) + // Local plugins are unrelated to chart rendering and can print + // startup warnings on stdout before the YAML document. + cmd.Env = append(os.Environ(), "HELM_PLUGINS="+t.TempDir()) + var stderr bytes.Buffer + cmd.Stderr = &stderr + out, err := cmd.Output() + require.NoError(t, err, "%s", stderr.String()) + docs := yaml.NewDecoder(bytes.NewReader(out)) + resources := make(map[string]chartResource) + for { + var resource chartResource + err := docs.Decode(&resource) + if errors.Is(err, io.EOF) { + break + } + require.NoError(t, err) + resources[resource.Kind+"/"+resource.Metadata.Name] = resource + } + return resources +} + +// The control plane dispatches each request on one shared HTTP ingress, with +// the gRPC upstream protocol applied to the whole ingress. +func testSharedControlplaneIngress(t *testing.T) { const ( - setFlag = "--set" - setStringFlag = "--set-string" grpcBackend = "GRPC" h2cScheme = "h2c" chartFullname = "test-chainloop" ) - helm, err := exec.LookPath("helm") - if err != nil { - t.Skip("helm is required for chart rendering tests") - } for _, tc := range []struct { name string args []string @@ -57,47 +87,26 @@ func testSharedIngress(t *testing.T, component string) { traefikScheme string }{ {name: "defaults", fullname: chartFullname, backendProtocol: grpcBackend, traefikScheme: h2cScheme}, - {name: "custom name", args: []string{setFlag, "fullnameOverride=shared-api"}, fullname: "shared-api", backendProtocol: grpcBackend, traefikScheme: h2cScheme}, + {name: "custom name", args: []string{"--set", "fullnameOverride=shared-api"}, fullname: "shared-api", backendProtocol: grpcBackend, traefikScheme: h2cScheme}, { - name: "common annotations merge", - args: []string{setStringFlag, `commonAnnotations.nginx\.org/http2=false`}, - fullname: chartFullname, backendProtocol: grpcBackend, traefikScheme: h2cScheme, + name: "common annotations merge", + args: []string{"--set-string", `commonAnnotations.nginx\.org/http2=false`}, + fullname: chartFullname, + backendProtocol: grpcBackend, traefikScheme: h2cScheme, }, { name: "encrypted upstream overrides", args: []string{ - setStringFlag, component + `.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, - setStringFlag, component + `.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, + "--set-string", `controlplane.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, + "--set-string", `controlplane.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, }, fullname: chartFullname, backendProtocol: "GRPCS", traefikScheme: "https", }, } { t.Run(tc.name, func(t *testing.T) { - serviceName := tc.fullname + "-" + component - args := append([]string{"template", "test", ".", setFlag, "development=true", - setFlag, component + ".ingress.enabled=true", setFlag, component + ".ingress.hostname=shared.example.test", - setFlag, "controlplane.ingressAPI.enabled=true", setFlag, "cas.ingressAPI.enabled=true"}, tc.args...) - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - cmd := exec.CommandContext(ctx, helm, args...) - // Local plugins are unrelated to chart rendering and can print - // startup warnings on stdout before the YAML document. - cmd.Env = append(os.Environ(), "HELM_PLUGINS="+t.TempDir()) - var stderr bytes.Buffer - cmd.Stderr = &stderr - out, err := cmd.Output() - require.NoError(t, err, "%s", stderr.String()) - docs := yaml.NewDecoder(bytes.NewReader(out)) - resources := make(map[string]chartResource) - for { - var resource chartResource - err := docs.Decode(&resource) - if errors.Is(err, io.EOF) { - break - } - require.NoError(t, err) - resources[resource.Kind+"/"+resource.Metadata.Name] = resource - } + serviceName := tc.fullname + "-controlplane" + resources := renderChart(t, append(tc.args, + "--set", "controlplane.ingress.enabled=true", "--set", "controlplane.ingress.hostname=shared.example.test")) ingress, ok := resources["Ingress/"+serviceName] require.True(t, ok) require.Equal(t, tc.backendProtocol, ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) @@ -116,11 +125,91 @@ func testSharedIngress(t *testing.T, component string) { require.Equal(t, grpcBackend, legacy.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) require.Len(t, legacy.Spec.Rules, 1) require.Equal(t, serviceName+"-api", legacy.Spec.Rules[0].HTTP.Paths[0].Backend.Service.Name) - casName := tc.fullname + "-cas-api" - cas := resources["Ingress/"+casName] - require.Equal(t, grpcBackend, cas.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) - require.Len(t, cas.Spec.Rules, 1) - require.Equal(t, casName, cas.Spec.Rules[0].HTTP.Paths[0].Backend.Service.Name) + }) + } +} + +// The CAS shared HTTP ingress keeps an HTTP/1.1 upstream, and the gRPC RPC +// paths render into a second Ingress on the same hostname with their own +// upstream pool (spec D-018). +func testSharedCASIngress(t *testing.T) { + const ( + grpcBackend = "GRPC" + chartFullname = "test-chainloop" + ) + for _, tc := range []struct { + name string + args []string + fullname string + grpcProtocol string + sharedHTTP2 string + grpcHTTP2 string + traefikScheme string + }{ + {name: "defaults", fullname: chartFullname, grpcProtocol: grpcBackend, grpcHTTP2: "true"}, + {name: "custom name", args: []string{"--set", "fullnameOverride=shared-api"}, fullname: "shared-api", grpcProtocol: grpcBackend, grpcHTTP2: "true"}, + { + // A global annotation reaches both Ingress objects; on the gRPC + // Ingress it overrides the chart default. + name: "common annotations merge", + args: []string{"--set-string", `commonAnnotations.nginx\.org/http2=false`}, + fullname: chartFullname, + grpcProtocol: grpcBackend, sharedHTTP2: "false", grpcHTTP2: "false", + }, + { + name: "encrypted upstream overrides", + args: []string{ + "--set-string", `cas.ingress.grpc.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, + "--set-string", `cas.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, + }, + fullname: chartFullname, + grpcProtocol: "GRPCS", sharedHTTP2: "", grpcHTTP2: "true", traefikScheme: "https", + }, + } { + t.Run(tc.name, func(t *testing.T) { + serviceName := tc.fullname + "-cas" + resources := renderChart(t, append(tc.args, + "--set", "cas.ingress.enabled=true", "--set", "cas.ingress.hostname=shared.example.test")) + + // The shared HTTP ingress dials HTTP/1.1: no gRPC annotations by default. + ingress, ok := resources["Ingress/"+serviceName] + require.True(t, ok) + require.Empty(t, ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) + require.Equal(t, tc.sharedHTTP2, ingress.Metadata.Annotations["nginx.org/http2"]) + require.Empty(t, ingress.Metadata.Annotations["nginx.org/grpc-services"]) + require.Len(t, ingress.Spec.Rules, 1) + require.Equal(t, "shared.example.test", ingress.Spec.Rules[0].Host) + require.Len(t, ingress.Spec.Rules[0].HTTP.Paths, 1) + path := ingress.Spec.Rules[0].HTTP.Paths[0] + require.Equal(t, "/", path.Path) + require.Equal(t, serviceName, path.Backend.Service.Name) + require.Equal(t, "http", path.Backend.Service.Port.Name) + + // The gRPC RPC paths render into their own Ingress on the same + // hostname, with their own annotations and upstream pool. + grpcIngress, ok := resources["Ingress/"+serviceName+"-grpc"] + require.True(t, ok) + require.Equal(t, tc.grpcProtocol, grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) + require.Equal(t, serviceName, grpcIngress.Metadata.Annotations["nginx.org/grpc-services"]) + require.Equal(t, tc.grpcHTTP2, grpcIngress.Metadata.Annotations["nginx.org/http2"]) + require.Equal(t, "100m", grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/proxy-body-size"]) + require.Equal(t, "3M", grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/client-body-buffer-size"]) + require.Len(t, grpcIngress.Spec.Rules, 1) + require.Equal(t, "shared.example.test", grpcIngress.Spec.Rules[0].Host) + require.Len(t, grpcIngress.Spec.Rules[0].HTTP.Paths, 2) + for _, p := range grpcIngress.Spec.Rules[0].HTTP.Paths { + require.Equal(t, serviceName, p.Backend.Service.Name) + require.Equal(t, "http", p.Backend.Service.Port.Name) + } + require.Equal(t, "/cas.v1.ByteStream/", grpcIngress.Spec.Rules[0].HTTP.Paths[0].Path) + require.Equal(t, "/cas.v1.ResourceService/", grpcIngress.Spec.Rules[0].HTTP.Paths[1].Path) + + svc := resources["Service/"+serviceName] + require.Equal(t, tc.traefikScheme, svc.Metadata.Annotations["traefik.ingress.kubernetes.io/service.serversscheme"]) + casLegacy := resources["Ingress/"+tc.fullname+"-cas-api"] + require.Equal(t, grpcBackend, casLegacy.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) + require.Len(t, casLegacy.Spec.Rules, 1) + require.Equal(t, tc.fullname+"-cas-api", casLegacy.Spec.Rules[0].HTTP.Paths[0].Backend.Service.Name) }) } } @@ -149,4 +238,4 @@ type chartResource struct { } `yaml:"http"` } `yaml:"rules"` } `yaml:"spec"` -} +} \ No newline at end of file diff --git a/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml b/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml new file mode 100644 index 000000000..7d0b1b85c --- /dev/null +++ b/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml @@ -0,0 +1,61 @@ +{{- /* +Copyright Chainloop, Inc. All Rights Reserved. +SPDX-License-Identifier: APACHE-2.0 +*/}} + +{{- /* +gRPC routes of the shared CAS endpoint (spec D-018). Rendered as a second +Ingress on the same hostname as cas/ingress.yaml, with its own annotations: +controllers that set the backend protocol per Ingress resource (community +nginx `backend-protocol`) need a dedicated object for the gRPC paths, so the +gRPC upstream pool keeps the HTTP/2 (grpc_pass) transport while the HTTP +routes keep an HTTP/1.1 upstream. Controllers that conflict on unrelated +same-host Ingress objects (F5 NGINX) need their own resource instead; see +the chart README. +*/}} +{{- if and .Values.cas.ingress.enabled .Values.cas.ingress.grpc.paths }} +{{- $fullName := include "chainloop.cas.fullname" . -}} +{{- $grpcName := printf "%s-grpc" $fullName -}} +apiVersion: {{ include "common.capabilities.ingress.apiVersion" . }} +kind: Ingress +metadata: + name: {{ $grpcName }} + namespace: {{ include "common.names.namespace" . | quote }} + labels: {{- include "chainloop.cas.labels" . | nindent 4 }} + {{- if or .Values.cas.ingress.grpc.annotations .Values.cas.ingress.grpc.defaultAnnotations .Values.commonAnnotations }} + {{- /* cas.ingress.annotations stay on the HTTP ingress only: a gRPC backend + protocol set there would reach the HTTP routes. The gRPC-path Ingress + takes its annotations from cas.ingress.grpc.annotations and the + common ones; earlier in the list wins. */}} + {{- $annotations := include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.grpc.annotations .Values.commonAnnotations .Values.cas.ingress.grpc.defaultAnnotations) "context" .) }} + annotations: {{- include "common.tplvalues.render" (dict "value" $annotations "context" $) | nindent 4 }} + {{- end }} +spec: + {{- if and .Values.cas.ingress.ingressClassName (eq "true" (include "common.ingress.supportsIngressClassname" .)) }} + ingressClassName: {{ .Values.cas.ingress.ingressClassName | quote }} + {{- end }} + rules: + {{- if .Values.cas.ingress.hostname }} + - host: {{ .Values.cas.ingress.hostname }} + http: + paths: + {{- range .Values.cas.ingress.grpc.paths }} + - path: {{ . }} + {{- if eq "true" (include "common.ingress.supportsPathType" $) }} + pathType: {{ default "ImplementationSpecific" $.Values.cas.ingress.pathType }} + {{- end }} + backend: {{- include "common.ingress.backend" (dict "serviceName" $fullName "servicePort" "http" "context" $) | nindent 14 }} + {{- end }} + {{- end }} + {{- if or (and .Values.cas.ingress.tls (or (include "common.ingress.certManagerRequest" (dict "annotations" (include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.grpc.annotations .Values.commonAnnotations) "context" .) | fromYaml))) .Values.cas.ingress.selfSigned)) .Values.cas.ingress.extraTls }} + tls: + {{- if and .Values.cas.ingress.tls (or (include "common.ingress.certManagerRequest" ( dict "annotations" .Values.cas.ingress.annotations )) .Values.cas.ingress.selfSigned) }} + - hosts: + - {{ .Values.cas.ingress.hostname | quote }} + secretName: {{ printf "%s-tls" .Values.cas.ingress.hostname }} + {{- end }} + {{- if .Values.cas.ingress.extraTls }} + {{- include "common.tplvalues.render" (dict "value" .Values.cas.ingress.extraTls "context" $) | nindent 4 }} + {{- end }} + {{- end }} +{{- end }} \ No newline at end of file diff --git a/deployment/chainloop/values.yaml b/deployment/chainloop/values.yaml index cab498af6..d29cc0ffa 100644 --- a/deployment/chainloop/values.yaml +++ b/deployment/chainloop/values.yaml @@ -1129,11 +1129,12 @@ cas: ## ref http://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip ## externalTrafficPolicy: Cluster - ## @extra cas.service.annotations Annotations for the shared HTTP and gRPC service. Traefik uses h2c upstream. + ## @param cas.service.annotations Additional custom annotations for cas service. + ## NOTE: do not set a Traefik `service.serversscheme: h2c` default here. The + ## shared endpoint's HTTP routes must be dialed HTTP/1.1, and a per-Service + ## scheme annotation cannot differ per upstream pool (spec D-017/D-018). ## - annotations: - ## @skip cas.service.annotations.traefik.ingress.kubernetes.io/service.serversscheme - traefik.ingress.kubernetes.io/service.serversscheme: h2c + annotations: {} ## @param cas.service.extraPorts Extra ports to expose in cas service (normally used with the `sidecars` value) ## extraPorts: [] @@ -1237,7 +1238,10 @@ cas: ## NOTE: You may need to set this to '/*' in order to use this with ALB ingress controllers ## path: / - ## @extra cas.ingress.annotations Annotations for the experimental shared HTTP and gRPC ingress. + ## @extra cas.ingress.annotations Annotations for the shared CAS HTTP ingress (downloads, status). + ## Do not set a gRPC backend protocol here: the HTTP routes must use an + ## HTTP/1.1 upstream (spec D-017/D-018). The gRPC routes are configured in + ## cas.ingress.grpc below. ## Use this parameter to set the required annotations for cert-manager, see ## ref: https://cert-manager.io/docs/usage/ingress/#supported-annotations ## e.g: @@ -1245,18 +1249,42 @@ cas: ## kubernetes.io/cas.ingress.class: nginx ## cert-manager.io/cluster-issuer: cluster-issuer-name ## - annotations: - ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/backend-protocol - nginx.ingress.kubernetes.io/backend-protocol: GRPC - ## @skip cas.ingress.annotations.nginx.org/grpc-services - nginx.org/grpc-services: '{{ include "chainloop.cas.fullname" . }}' - ## @skip cas.ingress.annotations.nginx.org/http2 - nginx.org/http2: "true" - ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/proxy-body-size - # Same upload limit as the independent legacy gRPC ingress. Set 0 to disable. - nginx.ingress.kubernetes.io/proxy-body-size: "100m" - ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/client-body-buffer-size - nginx.ingress.kubernetes.io/client-body-buffer-size: "3M" + annotations: {} + ## @extra cas.ingress.grpc gRPC routes of the shared CAS endpoint. + ## The chart renders a second Ingress on the same hostname with its own + ## upstream pool, so native gRPC traffic keeps the HTTP/2 (grpc_pass) + ## transport while the HTTP routes keep HTTP/1.1. This is the + ## separate-upstream-pool contract of spec D-014/D-018. + grpc: + ## @param cas.ingress.grpc.paths RPC path prefixes served by the gRPC upstream pool. + paths: + - /cas.v1.ByteStream/ + - /cas.v1.ResourceService/ + ## @extra cas.ingress.grpc.annotations Annotations for the gRPC-path Ingress. + ## Use this to override the gRPC upstream settings, for example + ## `backend-protocol: GRPCS` when TLS runs to the pod. Do not set these + ## in cas.ingress.annotations: that object belongs to the HTTP routes. + ## e.g: + ## annotations: + ## nginx.ingress.kubernetes.io/backend-protocol: "GRPCS" + annotations: {} + ## @extra cas.ingress.grpc.defaultAnnotations Chart defaults for the gRPC-path Ingress. + ## cas.ingress.grpc.annotations and commonAnnotations override these. + defaultAnnotations: + ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/backend-protocol + ## Tell nginx Ingress Controller to expect gRPC traffic on these paths + nginx.ingress.kubernetes.io/backend-protocol: "GRPC" + ## @skip cas.ingress.grpc.defaultAnnotations.nginx.org/grpc-services + nginx.org/grpc-services: '{{ include "chainloop.cas.fullname" . }}' + ## @skip cas.ingress.grpc.defaultAnnotations.nginx.org/http2 + nginx.org/http2: "true" + ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/proxy-body-size + # Same upload limit as the independent legacy gRPC ingress. Set 0 to disable. + nginx.ingress.kubernetes.io/proxy-body-size: "100m" + ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/client-body-buffer-size + # Improve upload speed by adding client buffering used by http2 control-flows + # https://github.com/chainloop-dev/chainloop/issues/375 + nginx.ingress.kubernetes.io/client-body-buffer-size: "3M" ## @param cas.ingress.tls Enable TLS configuration for the host defined at `controlplane.ingress.hostname` parameter ## TLS certificates will be retrieved from a TLS secret with name: `{{- printf "%s-tls" .Values.controlplane.ingress.hostname }}` ## You can: diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index b23215c7c..6a3bfb0ad 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -26,7 +26,8 @@ Control-plane requirements and design superseded by [Spec issue-3628-controlplan For the control plane, the requirements, constraints, proposal, diagram, decision records, milestones, and risks below are historical. Use the replacement spec for all control-plane implementation and qualification. Only CAS requirements remain in scope here. -CAS transport selection and qualification remain open under the parent ticket. +The CAS transport decision is recorded below in D-017 and D-018. +CAS qualification remains open under the parent ticket. ## Summary @@ -56,14 +57,22 @@ The control plane and the Artifact CAS will each serve gRPC and HTTP on one port ## Requirements -### R-001: One port serves all protocols -On its single port, each service MUST serve gRPC (HTTP/2) and plain HTTP (HTTP/1.1 and HTTP/2). The control plane MUST also serve gRPC-Web. The CAS has no gRPC-Web today and does not get it (see D-011). This includes every endpoint that the gRPC port and the HTTP port serve today. -- Done when: these clients work against only the single port: - - the CLI and the web UI - - the OIDC login and the status probes - - the referrer discovery and the OpenAPI document - - CAS browser downloads -- Done when: the same checks pass through nginx ingress and through Traefik ingress. +### ~~R-001: One port serves all protocols~~ +Superseded by R-009 for the CAS. The control-plane scope moved to [the replacement spec](issue-3628-controlplane-http-dispatch.md). + +~~On its single port, each service MUST serve gRPC (HTTP/2) and plain HTTP (HTTP/1.1 and HTTP/2). The control plane MUST also serve gRPC-Web. The CAS has no gRPC-Web today and does not get it (see D-011). This includes every endpoint that the gRPC port and the HTTP port serve today.~~ +- ~~Done when: these clients work against only the single port:~~ + - ~~the CLI and the web UI~~ + - ~~the OIDC login and the status probes~~ + - ~~the referrer discovery and the OpenAPI document~~ + - ~~CAS browser downloads~~ +- ~~Done when: the same checks pass through nginx ingress and through Traefik ingress.~~ + +### R-009: CAS shared endpoint protocol scope +Supersedes R-001 for the CAS. On its single port, the CAS MUST serve native gRPC over HTTP/2 and plain HTTP over HTTP/1.1, through cmux connection classification with the native gRPC transport (D-017). Plain HTTP/2 other than native gRPC is not supported on the CAS shared port: the gRPC matcher's SETTINGS injection breaks such a connection once it falls through to the HTTP server. Plain HTTP/2 and mixed connections remain control-plane scope under the replacement spec. +- Done when: focused tests cover native gRPC over h2c and TLS, HTTP/1.1 downloads, the status endpoints, and the CORS preflight through the shared port. +- Done when: the chart routes the gRPC paths and the HTTP paths into separate upstream pools, with HTTP/1.1 for the HTTP routes (D-018). +- Done when: browser downloads through a configured ingress work, and the documentation states that a browser downloading directly from a TLS-configured shared port is not a supported deployment. ### R-005: Existing configuration keeps working An installation that does not change its configuration MUST keep working during the deprecation period. The legacy gRPC port keeps serving gRPC at its current address during that period. CLI configurations that use ingress hostnames keep working when the ingress backend moves to the single port. The control plane connection to the CAS, and any other client that uses the legacy gRPC address, MUST move to the single HTTP address before the legacy gRPC port is disabled. @@ -269,14 +278,32 @@ Supersedes D-005. The services solve the multiplexing in the server process. The - A mandatory sidecar or embedded proxy: it works behind more proxy configurations, but it adds a process and its lifecycle to every deployment. - Source: [PR comment](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6266145446) -### D-016: Complete the classifier's SETTINGS exchange before HTTP handoff -Supersedes D-013. The service keeps cmux connection classification, the native gRPC server, and the HTTP request dispatcher with its gRPC-Web wrapper and native-gRPC fallback. Both native gRPC and ordinary HTTP/2 clients may wait for server SETTINGS before their first request. Silence does not identify the protocol of that request. -- The matcher uses the standard HTTP/2 framer and HPACK decoder. It limits incoming frames to the initial 16 KiB maximum before reading their payloads, as well as limiting classification bytes, time, and concurrency. -- When the matcher sends an empty SETTINGS frame and the first request is HTTP, it waits within the classification bounds for that SETTINGS acknowledgement. It records the validated ACK's exact offset in cmux's replay. The HTTP listener removes only that nine-byte frame before the HTTP server reads the connection. The HTTP server owns all subsequent SETTINGS exchanges. An incomplete injected handshake is closed. -- Native gRPC connections still go to the existing native server with their original replay. HTTP connections that did not need an injected SETTINGS frame also retain their original replay. HTTP requests and later streams are not parsed again by the handoff adapter. -- Why: rejecting every non-gRPC request after the matcher sends SETTINGS also rejects valid HTTP-only clients. Forwarding the extra ACK unchanged makes net/http reject the connection after the client acknowledges the HTTP server's own SETTINGS. Removing the one ACK preserves its ownership without adding an ongoing frame parser. -- Evidence: focused tests cover HTTP and gRPC-Web content types over plaintext and TLS, ACKs before and after the first request, subsequent requests on the same connection, incomplete handshakes, and an oversized frame header without its payload. These tests do not establish full R-002 parity or R-003 CAS performance. -- Source: [implementation review](https://github.com/chainloop-dev/chainloop/pull/3647#pullrequestreview-5573391966). +### ~~D-016: Complete the classifier's SETTINGS exchange before HTTP handoff~~ +Superseded by D-017. + +~~Supersedes D-013. The service keeps cmux connection classification, the native gRPC server, and the HTTP request dispatcher with its gRPC-Web wrapper and native-gRPC fallback. Both native gRPC and ordinary HTTP/2 clients may wait for server SETTINGS before their first request. Silence does not identify the protocol of that request.~~ +- ~~The matcher uses the standard HTTP/2 framer and HPACK decoder. It limits incoming frames to the initial 16 KiB maximum before reading their payloads, as well as limiting classification bytes, time, and concurrency.~~ +- ~~When the matcher sends an empty SETTINGS frame and the first request is HTTP, it waits within the classification bounds for that SETTINGS acknowledgement. It records the validated ACK's exact offset in cmux's replay. The HTTP listener removes only that nine-byte frame before the HTTP server reads the connection. The HTTP server owns all subsequent SETTINGS exchanges. An incomplete injected handshake is closed.~~ +- ~~Native gRPC connections still go to the existing native server with their original replay. HTTP connections that did not need an injected SETTINGS frame also retain their original replay. HTTP requests and later streams are not parsed again by the handoff adapter.~~ +- ~~Why: rejecting every non-gRPC request after the matcher sends SETTINGS also rejects valid HTTP-only clients. Forwarding the extra ACK unchanged makes net/http reject the connection after the client acknowledges the HTTP server's own SETTINGS. Removing the one ACK preserves its ownership without adding an ongoing frame parser.~~ +- ~~Evidence: focused tests cover HTTP and gRPC-Web content types over plaintext and TLS, ACKs before and after the first request, subsequent requests on the same connection, incomplete handshakes, and an oversized frame header without its payload. These tests do not establish full R-002 parity or R-003 CAS performance.~~ +- ~~Source: [implementation review](https://github.com/chainloop-dev/chainloop/pull/3647#pullrequestreview-5573391966).~~ + +### D-017: CAS shared endpoint: cmux with the native transport, and the corrected SETTINGS contract +Supersedes D-016 for the CAS endpoint. The control-plane parts of D-014 to D-016 were already superseded by [the replacement spec](issue-3628-controlplane-http-dispatch.md). + +The CAS shared endpoint is `server.http.addr`. cmux reads the start of each new connection and hands the whole connection to one of the two existing transports: an HTTP/2 connection whose first request declares a native gRPC content type goes to the existing native gRPC listener (`grpc.Server.Serve`); every other connection goes to the existing HTTP server. ByteStream uploads and downloads therefore keep the grpc-go native transport, so R-003 does not rest on the HTTP adapter. +- Why not the merged control-plane design, per-request `ServeHTTP` dispatch: the [PR #3663 1 GiB comparison](https://github.com/chainloop-dev/chainloop/pull/3663) measured about 1.25 GiB of combined client/server sampled peak live heap for adapter uploads, against about 28 MiB for the native transport, and download throughput 11.8% lower, already past the R-003 limit. grpc-go's handler transport reads the HTTP body in a separate goroutine into an unbounded receive buffer, so read-ahead is not bounded by the application's pace. CAS moves gigabyte streams; the control plane does not. The two services keep different transport designs on purpose. +- Why D-016 is superseded: D-016 assumed a classifier this project owns, which records the SETTINGS ACK offset in its own replay and removes that frame before the HTTP server reads. The cmux library owns the sniffing buffer, so that removal does not exist. Two reproductions on the CAS branch fix the contract instead. First, without SETTINGS injection, direct grpc-go clients deadlock: they send their preface and SETTINGS and wait for the server's SETTINGS before their first HEADERS frame, while the matcher waits for headers to sniff. The gRPC matcher must use cmux's SETTINGS-writing matcher. Second, that matcher writes a SETTINGS frame on every HTTP/2 connection, matched or not, because every HTTP/2 client sends SETTINGS with its preface. A plain HTTP/2 client that falls through to the HTTP server acknowledges the injected SETTINGS frame, and net/http rejects the unsolicited acknowledgement (GOAWAY, PROTOCOL_ERROR). +- Contract: once the matcher has written to a connection, the connection belongs to the native gRPC server. The HTTP side of the shared endpoint supports HTTP/1.1 only; HTTP/2 reaches the CAS only as native gRPC. Supported proxies keep native gRPC and plain HTTP on separate upstream connections (D-014) and dial HTTP/1.1 for the HTTP routes (D-018). +- Source: [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) experiment and benchmark, and the local reproductions on its branch. + +### D-018: CAS chart: one hostname, path-split upstream pools, HTTP/1.1 for the HTTP routes +The chart exposes one CAS hostname and one public port, and splits the routes by protocol into separate upstream pools: the `/cas.v1.*` paths go to the gRPC pool with an HTTP/2 upstream (nginx `backend-protocol: GRPC` and `grpc-services`; the upload body-size annotations move with them), and `/download/*`, `/statusz`, and `/infoz` go to the HTTP pool, which dials HTTP/1.1, not h2c. One certificate terminates at the edge. +- Why the path split is exact for CAS: CAS has no gRPC-Web (D-011), so its gRPC paths and its HTTP paths are disjoint. No path can carry both protocols. The control plane could not use this split because native gRPC and gRPC-Web share RPC paths; the [PR #3647 NGINX reproduction](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6272007730) showed the path-only split failing there with 415 on gRPC-Web. CAS does not have that failure mode. +- Why HTTP/1.1 upstream on the HTTP pool: D-017. An HTTP/2 upstream for the HTTP routes triggers the matcher's SETTINGS injection, and the client's acknowledgement then breaks the HTTP server connection with GOAWAY. +- Controller notes: community nginx merges same-host paths from multiple Ingress resources and sets the backend protocol per resource, which expresses the split. F5 NGINX conflicts on unrelated same-host Ingresses; use its VirtualServer resource. Traefik applies the `service.serversscheme` annotation per Service, so one Service cannot be dialed h2c for the gRPC routes and HTTP/1.1 for the HTTP routes; the Traefik split needs a second Service for the HTTP routes or a controller-specific resource. The browser download endpoint stays on the HTTP routes: it is published API ([docs](https://docs.chainloop.dev/api-reference/downloadservice/download-artifacts-from-cas)), and CAS does not drop it. +- Source: chart changes on the [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) branch. ## Open Questions diff --git a/go.mod b/go.mod index d91a3f4f2..e5670a78d 100644 --- a/go.mod +++ b/go.mod @@ -93,6 +93,7 @@ require ( github.com/sigstore/sigstore/pkg/signature/kms/gcp v1.11.0 github.com/sigstore/sigstore/pkg/signature/kms/hashivault v1.11.0 github.com/sigstore/timestamp-authority/v2 v2.1.4 + github.com/soheilhy/cmux v0.1.5 github.com/spdx/tools-golang v0.5.7 github.com/spf13/cobra v1.10.2 github.com/spf13/pflag v1.0.10 diff --git a/go.sum b/go.sum index 89f3201b3..6a9b5924e 100644 --- a/go.sum +++ b/go.sum @@ -1189,6 +1189,8 @@ github.com/skeema/knownhosts v1.3.1/go.mod h1:r7KTdC8l4uxWRyK2TpQZ/1o5HaSzh06ePQ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= +github.com/soheilhy/cmux v0.1.5 h1:jjzc5WVemNEDTLwv9tlmemhC73tI08BNOIGwBOo10Js= +github.com/soheilhy/cmux v0.1.5/go.mod h1:T7TcVDs9LWfQgPlPsdngu6I6QIoyIFZDDC6sNE1GqG0= github.com/sony/gobreaker v0.4.1/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY= github.com/sorairolake/lzip-go v0.3.8 h1:j5Q2313INdTA80ureWYRhX+1K78mUXfMoPZCw/ivWik= github.com/sorairolake/lzip-go v0.3.8/go.mod h1:JcBqGMV0frlxwrsE9sMWXDjqn3EeVf0/54YPsw66qkU= @@ -1468,6 +1470,7 @@ golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= From 5445516a639e12c2f76e7eee9b987acca08923b5 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Mon, 19 Oct 2026 23:18:53 +0200 Subject: [PATCH 03/19] fix(cas): bound cmux classification and isolate ingress connections Close incomplete connections during shutdown and bound classification time, bytes, and concurrency without limiting active native ByteStream transfers. Remove the HTTP adapter fallback and validate shared endpoint initialization before binding. Reject unsupported metrics TLS configuration. Correct the ByteStream and infrastructure RPC paths. Give ingress-nginx separate backend identities and upstream addresses through two port names on the same Service, both targeting the shared CAS listener. Share one TLS Secret and request its certificate from only one Ingress. Preserve the legacy listener and ingress settings. Record the CAS routing clarification, local controller regression checks, and refreshed 1 GiB transport comparison. Refs #3628. Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/README.md | 36 ++- .../experiments/shared-endpoint-benchmark.txt | 51 ++--- .../internal/server/classification.go | 129 +++++++++++ .../internal/server/classification_test.go | 154 +++++++++++++ app/artifact-cas/internal/server/grpc.go | 9 +- app/artifact-cas/internal/server/http.go | 72 ++---- app/artifact-cas/internal/server/http_test.go | 74 +++++-- .../internal/server/httpmetrics.go | 7 +- .../internal/server/httpmetrics_test.go | 42 ++++ .../internal/server/ingress_test.go | 208 ++++++++++++++++++ .../internal/server/streaming_bench_test.go | 3 +- deployment/chainloop/Chart.yaml | 2 +- deployment/chainloop/README.md | 5 +- deployment/chainloop/chart_test.go | 164 +++++++++++--- .../templates/cas/ingress-grpc-shared.yaml | 46 +++- .../chainloop/templates/cas/ingress.yaml | 4 +- .../chainloop/templates/cas/service-http.yaml | 12 +- deployment/chainloop/values.yaml | 22 +- docs/specs/issue-3628-single-endpoint.md | 22 +- 19 files changed, 902 insertions(+), 160 deletions(-) create mode 100644 app/artifact-cas/internal/server/classification.go create mode 100644 app/artifact-cas/internal/server/classification_test.go create mode 100644 app/artifact-cas/internal/server/httpmetrics_test.go create mode 100644 app/artifact-cas/internal/server/ingress_test.go diff --git a/app/artifact-cas/README.md b/app/artifact-cas/README.md index 24b79a75b..ca21b61b7 100644 --- a/app/artifact-cas/README.md +++ b/app/artifact-cas/README.md @@ -92,7 +92,13 @@ The development CAS public key and credentials service must match the control pl Change only the per-command override to `localhost:9001` to compare with the native listener. Once all direct clients and the control-plane CAS connection have been moved, `server.legacy_grpc_listener: false` disables the native listener. Leave it enabled during evaluation. Optional `server.http.tls_config` configures TLS for the shared endpoint; both certificate and private key must be set. It does not inherit `server.grpc.tls_config`. Note that a browser downloading directly from a TLS-configured shared port is not supported: the browser negotiates HTTP/2 through ALPN and the connection reaches the HTTP server with an injected SETTINGS frame. Serve browser downloads through the ingress. -The chart's `cas.ingress` routes the HTTP paths (`/download`, `/statusz`, `/infoz`) with an HTTP/1.1 upstream, and a second Ingress rendered from `cas.ingress.grpc` routes the `/cas.v1.*` RPC paths with the gRPC backend protocol. The gRPC Ingress keeps the native ingress's `100m` body limit and `3M` client body buffer default; override the body limit to `0` or a suitable larger limit for a 1 GB ingress experiment. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS Service keep their existing settings. Deployed controller qualification (F5 same-host Ingress conflicts, Traefik's per-Service scheme) remains outstanding. +The chart's `cas.ingress` sends HTTP routes (`/download`, `/statusz`, `/infoz`) over HTTP/1.1. A second Ingress sends Google ByteStream, CAS, health, reflection, metadata, and channelz RPCs over native gRPC. The same Service has two port names: `http` (80) and `grpc-shared` (81, configurable with `cas.service.ports.grpcShared`). Both target the existing shared container port. Ingress-nginx uses pod endpoints for HTTP and the Service ClusterIP for gRPC. Different backend identities and upstream addresses prevent its global keepalive cache from reusing a socket across protocols. Separate Ingress objects alone do not provide that isolation. + +Keep the Service port numbers different and at least one upstream on the ClusterIP. The chart rejects unsafe pool settings and headless Services for this arrangement. The gRPC Ingress keeps the legacy ingress's `100m` body limit and `3M` client buffer default. Override the body limit to `0` or a suitable larger value for a 1 GiB ingress experiment. Both Ingress objects reference one TLS Secret. Only one requests a certificate from cert-manager, preferring the HTTP Ingress when it has issuer annotations. `cas.ingress.tls=true` also supports a pre-created Secret without issuer annotations. + +`cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS Service keep their existing settings. F5 same-host routing and Traefik's per-Service scheme still need qualification. + +Before cmux hands a connection to a server, classification is limited to 10 seconds, 64 KiB read from the socket (including TLS), and 128 concurrent pending connections. Shutdown closes pending sockets. These bounds end at handoff and do not constrain active ByteStream transfers. The HTTP server then applies its 60-second header timeout. CAS has no per-request gRPC adapter fallback. Metrics remain plaintext on their separate listener; `server.http_metrics.tls_config` is rejected. Run the isolated functional checks without Vault or an object store: @@ -100,6 +106,22 @@ Run the isolated functional checks without Vault or an object store: go test -race ./app/artifact-cas/internal/server ./app/artifact-cas/cmd ./deployment/chainloop -count=1 ``` +To check the chart through a real ingress-nginx controller, use a disposable local kind cluster on Linux with Docker: + +```sh +kind create cluster --name cas-cmux --kubeconfig /tmp/cas-cmux-kubeconfig +kubectl --kubeconfig /tmp/cas-cmux-kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/kind/deploy.yaml +kubectl --kubeconfig /tmp/cas-cmux-kubeconfig label node cas-cmux-control-plane ingress-ready=true +kubectl --kubeconfig /tmp/cas-cmux-kubeconfig -n ingress-nginx wait --for=condition=Available deployment/ingress-nginx-controller --timeout=120s +kubectl --kubeconfig /tmp/cas-cmux-kubeconfig -n ingress-nginx patch configmap ingress-nginx-controller --type merge -p '{"data":{"worker-processes":"1"}}' +CAS_INGRESS_KUBECONFIG=/tmp/cas-cmux-kubeconfig \ +CAS_INGRESS_BACKEND_IP="$(docker network inspect kind --format '{{json .IPAM.Config}}' | jq -r '.[] | select(.Subnet | contains(":") | not) | .Gateway')" \ +go test -race ./app/artifact-cas/internal/server -run '^TestCASIngressNGINX$' -count=1 -v +kind delete cluster --name cas-cmux +``` + +The test requires one controller worker so both protocols exercise the same keepalive cache. It creates and deletes a namespace containing the rendered CAS Service and Ingress objects. Its CAS server runs in the test process, reachable through the kind bridge gateway. It checks authenticated ByteStream upload/download, reflection, status, and repeated HTTP downloads with HTTP/1.1 and HTTP/2 clients. The controller sends HTTP/1.1 to the CAS for both browser protocols. This does not qualify other controllers or production object storage. + Compare real ByteStream upload and download paths over TCP with a synthetic 1 GiB zero-filled artifact, streamed from a fixed-size buffer and checked by SHA256. Both listeners use the same file-backed test backend and real CAS staging and authentication. Select a disk-backed temporary directory; `/tmp` can be tmpfs. ```sh @@ -117,14 +139,14 @@ Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, cmux 0.1.5, Linu | Listener | Operation | Throughput (MB/s) | Sampled peak live heap (MiB) | | --- | --- | ---: | ---: | -| Native gRPC (`:9001`) | Upload | 228.90 | 29.27 | -| Shared endpoint (`:8001`, cmux) | Upload | 232.51 | 27.97 | -| Native gRPC (`:9001`) | Download | 194.10 | 30.90 | -| Shared endpoint (`:8001`, cmux) | Download | 189.74 | 31.83 | +| Native gRPC (`:9001`) | Upload | 225.34 | 28.91 | +| Shared endpoint (`:8001`, cmux) | Upload | 232.85 | 27.46 | +| Native gRPC (`:9001`) | Download | 198.05 | 28.90 | +| Shared endpoint (`:8001`, cmux) | Download | 189.63 | 31.92 | -Both listeners use the native grpc-go transport, and the numbers match: median upload throughput on the shared endpoint was 1.6% higher and median download throughput 2.2% lower than the legacy listener, with sampled peak live heap equivalent (~28-32 MiB) on both. This satisfies the R-003 transport-comparison shape for the shared endpoint — within 10% with bounded memory — in this synthetic local setup. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production. +Both listeners use the native grpc-go transport, and the numbers match: median upload throughput on the shared endpoint was 3.3% higher and median download throughput 4.3% lower than the legacy listener, with sampled peak live heap equivalent (~28-32 MiB) on both. This satisfies the R-003 transport-comparison shape for the shared endpoint — within 10% with bounded memory — in this synthetic local setup. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production. -For contrast, the earlier per-request `ServeHTTP` experiment on this branch measured 232.00 MB/s upload and 173.83 MB/s download against the same native baseline, with an upload peak of ~1278 MiB: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90), so read-ahead is not bounded by the application's pace. That memory regression rejected the adapter for CAS uploads (the control plane keeps the adapter because its requests are small and unary) and motivated the cmux design recorded in D-017. +For contrast, the earlier per-request `ServeHTTP` experiment on this branch measured 232.00 MB/s upload and 173.83 MB/s download against its contemporaneous native baseline, with an upload peak of ~1278 MiB: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90), so read-ahead is not bounded by the application's pace. That memory regression rejected the adapter for CAS uploads (the control plane keeps the adapter because its requests are small and unary) and motivated the cmux design recorded in D-017. ### Run tests diff --git a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt index eb439f0e5..ef42f9cce 100644 --- a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt +++ b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt @@ -1,45 +1,46 @@ # Copyright 2026 The Chainloop Authors. # SPDX-License-Identifier: Apache-2.0 -# cmux shared endpoint vs the legacy native listener. Both transports use the -# grpc-go native transport; the shared endpoint classifies each connection -# with cmux (spec D-017). An earlier run on this branch measured the -# grpc.Server.ServeHTTP HTTP adapter instead: its uploads peaked at -# ~1278 MiB of combined live heap for a 1 GiB artifact and its downloads -# were 11.8% slower; that result rejected the adapter for CAS. The raw -# numbers of that run are preserved in the PR #3663 discussion. +# Review-fix rerun: cmux shared endpoint vs the legacy native listener. +# Both use grpc-go's native transport. The shared listener now has bounded +# classification and closes pending sockets during shutdown. Earlier adapter +# and cmux results remain in this file's git history and PR #3663 discussion. go version go1.27.2 linux/amd64 grpc-go: v1.84.0; Kratos: v2.9.2; cmux: v0.1.5 Temporary filesystem: btrfs (not tmpfs) -Command: TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/server -run ^$ -bench ^BenchmarkCASByteStreamTransport$ -benchtime=1x -count=3 -timeout=10m +Commands: + mkdir -p "$HOME/.cache/chainloop-cas-bench" + TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/server -run '^$' -bench '^BenchmarkCASByteStreamTransport$' -benchtime=1x -count=3 -timeout=10m Artifact: 1 GiB; plaintext localhost TCP; real CAS services/auth/staging; disk-backed test backend; client and server in one process. -INFO msg=[HTTP] server listening on: 127.0.0.1:40239 -INFO msg=[gRPC] server listening on: 127.0.0.1:44469 +Limitations: synthetic zero content, local disk, combined sampled client/server live heap, no representative object store, concurrency, or TLS performance qualification. + +INFO msg=[gRPC] server listening on: 127.0.0.1:34425 +INFO msg=[HTTP] server listening on: 127.0.0.1:38673 goos: linux goarch: amd64 pkg: github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server cpu: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz -BenchmarkCASByteStreamTransport/legacy/upload-8 1 5207742134 ns/op 206.18 MB/s 30.03 peak-heap-MiB 1442550792 B/op 3204540 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4690862475 ns/op 228.90 MB/s 29.27 peak-heap-MiB 1477528368 B/op 3032897 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4473314156 ns/op 240.03 MB/s 29.11 peak-heap-MiB 1509947992 B/op 3032880 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 6035943310 ns/op 177.89 MB/s 31.86 peak-heap-MiB 1408894040 B/op 95728 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5531984200 ns/op 194.10 MB/s 30.90 peak-heap-MiB 1359969664 B/op 86333 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5476786936 ns/op 196.05 MB/s 30.05 peak-heap-MiB 1312724880 B/op 84305 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4765003662 ns/op 225.34 MB/s 29.50 peak-heap-MiB 1483628368 B/op 3160122 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 6045761098 ns/op 177.60 MB/s 28.33 peak-heap-MiB 1505093640 B/op 3153996 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4558956236 ns/op 235.52 MB/s 28.91 peak-heap-MiB 1472837272 B/op 3153869 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5701383303 ns/op 188.33 MB/s 28.90 peak-heap-MiB 1345298136 B/op 87053 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5418492831 ns/op 198.16 MB/s 28.93 peak-heap-MiB 1340639536 B/op 86919 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5421594921 ns/op 198.05 MB/s 28.67 peak-heap-MiB 1342393032 B/op 86665 allocs/op INFO msg=[gRPC] server stopping INFO msg=[HTTP] server stopping INFO msg=[gRPC] server stopping -INFO msg=[gRPC] server listening on: 127.0.0.1:39939 -INFO msg=[HTTP] server listening on: 127.0.0.1:38963 -BenchmarkCASByteStreamTransport/shared/upload-8 1 4691261582 ns/op 228.88 MB/s 27.97 peak-heap-MiB 1432139600 B/op 3225369 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 4618093234 ns/op 232.51 MB/s 27.54 peak-heap-MiB 1470447088 B/op 3224874 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 4550823731 ns/op 235.94 MB/s 28.81 peak-heap-MiB 1455017664 B/op 3153945 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5921459129 ns/op 181.33 MB/s 31.83 peak-heap-MiB 1373096000 B/op 95393 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5659083336 ns/op 189.74 MB/s 32.92 peak-heap-MiB 1332496448 B/op 85621 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5506534391 ns/op 194.99 MB/s 29.10 peak-heap-MiB 1313525896 B/op 78316 allocs/op +INFO msg=[gRPC] server listening on: 127.0.0.1:38603 +INFO msg=[HTTP] server listening on: 127.0.0.1:45889 +BenchmarkCASByteStreamTransport/shared/upload-8 1 4716482049 ns/op 227.66 MB/s 27.46 peak-heap-MiB 1418368840 B/op 3224685 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 4605922407 ns/op 233.12 MB/s 28.47 peak-heap-MiB 1459735392 B/op 3224827 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 4611224941 ns/op 232.85 MB/s 26.29 peak-heap-MiB 1440540392 B/op 3224912 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5862496162 ns/op 183.15 MB/s 32.00 peak-heap-MiB 1377047368 B/op 94038 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5623257158 ns/op 190.95 MB/s 31.34 peak-heap-MiB 1377929448 B/op 93669 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5662234715 ns/op 189.63 MB/s 31.92 peak-heap-MiB 1377032552 B/op 90475 allocs/op INFO msg=[gRPC] server stopping INFO msg=[HTTP] server stopping INFO msg=[gRPC] server stopping PASS -ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 65.136s +ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 65.771s diff --git a/app/artifact-cas/internal/server/classification.go b/app/artifact-cas/internal/server/classification.go new file mode 100644 index 000000000..39e5660dd --- /dev/null +++ b/app/artifact-cas/internal/server/classification.go @@ -0,0 +1,129 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "errors" + "net" + "sync" + "time" +) + +const ( + classificationTimeout = 10 * time.Second + maxClassifyingConnections = 128 + maxClassificationBytes = 64 << 10 +) + +// classificationListener owns sockets only until cmux hands them to a server. +// cmux clears the read deadline at that boundary (SetReadTimeout must be set). +// The timer also bounds TLS handshakes and SETTINGS writes, and Close releases +// pending sockets without interrupting streams already owned by HTTP or gRPC. +type classificationListener struct { + net.Listener + mu sync.Mutex + pending map[*classificationConn]*time.Timer + closed bool + timeout time.Duration +} + +func (l *classificationListener) Accept() (net.Conn, error) { + for { + conn, err := l.Listener.Accept() + if err != nil { + return nil, err + } + l.mu.Lock() + if l.closed { + l.mu.Unlock() + _ = conn.Close() + return nil, net.ErrClosed + } + if len(l.pending) >= maxClassifyingConnections { + l.mu.Unlock() + _ = conn.Close() + continue + } + c := &classificationConn{Conn: conn, owner: l, remaining: maxClassificationBytes} + l.pending[c] = time.AfterFunc(l.timeout, func() { + l.mu.Lock() + defer l.mu.Unlock() + if _, pending := l.pending[c]; pending { + delete(l.pending, c) + _ = c.Conn.Close() + } + }) + l.mu.Unlock() + return c, nil + } +} + +func (l *classificationListener) Close() error { + l.mu.Lock() + l.closed = true + for conn, timer := range l.pending { + timer.Stop() + _ = conn.Conn.Close() + delete(l.pending, conn) + } + l.mu.Unlock() + return l.Listener.Close() +} + +type classificationConn struct { + net.Conn + owner *classificationListener + remaining int +} + +func (c *classificationConn) Read(p []byte) (int, error) { + // Only the cmux goroutine reads or changes remaining until handoff. + if c.remaining == 0 { + _ = c.Close() + return 0, errors.New("CAS connection classification exceeded its byte limit") + } + if c.remaining > 0 && len(p) > c.remaining { + p = p[:c.remaining] + } + n, err := c.Conn.Read(p) + if c.remaining > 0 { + c.remaining -= n + } + return n, err +} + +func (c *classificationConn) SetReadDeadline(deadline time.Time) error { + if !deadline.IsZero() { + return c.Conn.SetReadDeadline(deadline) + } + c.owner.mu.Lock() + defer c.owner.mu.Unlock() + if timer, pending := c.owner.pending[c]; pending { + timer.Stop() + delete(c.owner.pending, c) + c.remaining = -1 + } + return c.Conn.SetReadDeadline(deadline) +} + +func (c *classificationConn) Close() error { + c.owner.mu.Lock() + if timer, pending := c.owner.pending[c]; pending { + timer.Stop() + delete(c.owner.pending, c) + } + c.owner.mu.Unlock() + return c.Conn.Close() +} diff --git a/app/artifact-cas/internal/server/classification_test.go b/app/artifact-cas/internal/server/classification_test.go new file mode 100644 index 000000000..d0d8ab8ba --- /dev/null +++ b/app/artifact-cas/internal/server/classification_test.go @@ -0,0 +1,154 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "context" + "errors" + "io" + "net" + "testing" + "time" + + v1 "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" + "github.com/soheilhy/cmux" + "github.com/stretchr/testify/require" + "golang.org/x/net/http2" +) + +func TestCASClassificationTimeout(t *testing.T) { + for _, prefix := range []string{"", "PRI ", "GET / HTTP/1.1\r\n"} { + t.Run(prefix, func(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + // Exercise the lifecycle timer, including a client that never reads + // server SETTINGS, independently of cmux's read deadline. + api.lis.(*classificationListener).timeout = 50 * time.Millisecond + api.httpSrv.ReadHeaderTimeout = 50 * time.Millisecond + endpoint, err := api.Endpoint() + require.NoError(t, err) + startCASTestTransport(t, api) + conn, err := net.Dial("tcp", endpoint.Host) + require.NoError(t, err) + defer conn.Close() + _, err = io.WriteString(conn, prefix) + require.NoError(t, err) + require.NoError(t, conn.SetReadDeadline(time.Now().Add(time.Second))) + _, err = io.Copy(io.Discard, conn) + require.NoError(t, err, "classification must close before the client deadline") + }) + } +} + +func TestCASClassificationBudget(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + endpoint, err := api.Endpoint() + require.NoError(t, err) + startCASTestTransport(t, api) + conn, err := net.Dial("tcp", endpoint.Host) + require.NoError(t, err) + defer conn.Close() + require.NoError(t, conn.SetDeadline(time.Now().Add(time.Second))) + // A peer may send valid HTTP/2 frames forever without the HEADERS cmux + // needs. The wire-byte budget must stop it before the time limit. + _, err = io.WriteString(conn, http2.ClientPreface) + require.NoError(t, err) + framer := http2.NewFramer(conn, conn) + require.NoError(t, framer.WriteSettings()) + frame, err := framer.ReadFrame() + require.NoError(t, err) + require.IsType(t, &http2.SettingsFrame{}, frame) + // Padded DATA are irrelevant to the header matcher but consume its budget. + for range maxClassificationBytes / (16 << 10) { + if err = framer.WriteData(1, false, make([]byte, 16<<10)); err != nil { + break + } + } + _, err = io.Copy(io.Discard, conn) + var netErr net.Error + if errors.As(err, &netErr) { + require.False(t, netErr.Timeout(), "classification byte limit was not enforced") + } +} + +func TestCASClassificationConcurrencyLimit(t *testing.T) { + root, err := net.Listen("tcp", casAnyAddr) + require.NoError(t, err) + listener := &classificationListener{Listener: root, pending: make(map[*classificationConn]*time.Timer), timeout: time.Minute} + defer listener.Close() + mux := cmux.New(listener) + mux.SetReadTimeout(time.Minute) + mux.Match(cmux.HTTP1Fast()) + done := make(chan error, 1) + go func() { done <- mux.Serve() }() + for range maxClassifyingConnections { + conn, err := net.Dial("tcp", root.Addr().String()) + require.NoError(t, err) + defer conn.Close() + } + require.Eventually(t, func() bool { + listener.mu.Lock() + defer listener.mu.Unlock() + return len(listener.pending) == maxClassifyingConnections + }, time.Second, time.Millisecond) + rejected, err := net.Dial("tcp", root.Addr().String()) + require.NoError(t, err) + defer rejected.Close() + require.NoError(t, rejected.SetReadDeadline(time.Now().Add(time.Second))) + var data [1]byte + _, err = rejected.Read(data[:]) + require.ErrorIs(t, err, io.EOF) + require.NoError(t, listener.Close()) + mux.Close() + select { + case err := <-done: + require.ErrorIs(t, err, net.ErrClosed) + case <-time.After(time.Second): + require.FailNow(t, "classifier workers did not exit") + } +} + +func TestCASClassificationDeadlineCleared(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + api.lis.(*classificationListener).timeout = 250 * time.Millisecond + api.mux.SetReadTimeout(250 * time.Millisecond) + endpoint, err := api.Endpoint() + require.NoError(t, err) + startCASTestTransport(t, api) + conn := casTestConn(t, endpoint.Host, false) + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + // A gRPC connection must remain usable after its classification deadline. + client := v1.NewStatusServiceClient(conn) + _, err = client.Statusz(ctx, &v1.StatuszRequest{}) + require.NoError(t, err) + time.Sleep(500 * time.Millisecond) + _, err = client.Statusz(ctx, &v1.StatuszRequest{}) + require.NoError(t, err) +} diff --git a/app/artifact-cas/internal/server/grpc.go b/app/artifact-cas/internal/server/grpc.go index 3d659b9fa..d196bd481 100644 --- a/app/artifact-cas/internal/server/grpc.go +++ b/app/artifact-cas/internal/server/grpc.go @@ -64,12 +64,9 @@ type Opts struct { Logger log.Logger } -// GRPCServers keeps one gRPC server for the shared endpoint separate from the -// optional legacy listener. Handler never binds a socket of its own: cmux -// hands it the native-gRPC connections of the shared HTTP listener, and the -// HTTP server dispatches its per-request gRPC fallback to the same server -// through grpc-go's HTTP adapter. Both listeners therefore run the same -// services and interceptors. +// GRPCServers keeps the shared endpoint's native gRPC server separate from +// the optional legacy listener. Both run the same services and interceptors. +// Handler receives connections from cmux and does not bind its own socket. type GRPCServers struct { Handler *grpc.Server Legacy *grpc.Server diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 0b95e93e9..82af7c09f 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -23,7 +23,7 @@ import ( "net" h "net/http" "net/url" - "strings" + "sync" "time" api "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" @@ -51,11 +51,8 @@ import ( // // ByteStream therefore keeps the grpc-go native transport, so large uploads // stream with bounded memory exactly as they do on the legacy listener. This -// is the separate-connection contract of the spec's D-014/D-015: a pooled -// upstream connection must stay single-protocol, and the chart routes the gRPC -// paths through their own HTTP/2 upstream pool. A connection that cmux gave to -// the HTTP server can still carry a later native gRPC request through the -// per-request ServeHTTP fallback below; the reverse is not supported. +// follows the separate-connection contract in D-017: native gRPC and HTTP +// use separate connections, with HTTP/1.1 on the HTTP side. type APIServer struct { lis net.Listener mux cmux.CMux @@ -64,6 +61,7 @@ type APIServer struct { grpcSrv *grpc.Server httpSrv *http.Server scheme string + workers sync.WaitGroup } // Start serves the shared endpoint until ctx is cancelled or a transport @@ -73,9 +71,9 @@ type APIServer struct { // Stop closes the listener and all three return ordinary shutdown errors. func (s *APIServer) Start(ctx context.Context) error { errCh := make(chan error, 3) - go func() { errCh <- s.httpSrv.Start(ctx) }() - go func() { errCh <- s.grpcSrv.Server.Serve(s.grpcL) }() - go func() { errCh <- s.mux.Serve() }() + s.workers.Go(func() { errCh <- s.httpSrv.Start(ctx) }) + s.workers.Go(func() { errCh <- s.grpcSrv.Serve(s.grpcL) }) + s.workers.Go(func() { errCh <- s.mux.Serve() }) select { case <-ctx.Done(): return nil @@ -87,9 +85,7 @@ func (s *APIServer) Start(ctx context.Context) error { } } -// Stop closes the shared listener, then drains the HTTP server (including its -// per-request gRPC fallback streams) before the native gRPC transport stops -// its in-flight streams. +// Stop closes pending classification sockets, then drains both transports. func (s *APIServer) Stop(ctx context.Context) error { // Stop accepting new connections on the shared endpoint. cmux.Close only // marks the match listeners closed: active connections keep draining. @@ -103,6 +99,7 @@ func (s *APIServer) Stop(ctx context.Context) error { httpErr = nil } grpcErr := s.grpcSrv.Stop(ctx) + s.workers.Wait() if closeErr != nil && !errors.Is(closeErr, net.ErrClosed) { return errors.Join(closeErr, httpErr, grpcErr) } @@ -127,6 +124,12 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) return nil, err } + // Validate before binding, so initialization errors do not leak a listener. + publicKey, err := parsePublicKey(authConf, logger) + if err != nil { + return nil, err + } + network := c.Http.GetNetwork() if network == "" { network = "tcp" @@ -135,6 +138,7 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) if err != nil { return nil, fmt.Errorf("binding the shared CAS endpoint %q: %w", c.Http.GetAddr(), err) } + lis = &classificationListener{Listener: lis, pending: make(map[*classificationConn]*time.Timer), timeout: classificationTimeout} scheme := "http" if tlsConfig != nil { // Terminate TLS before cmux so one certificate covers both @@ -144,6 +148,7 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) } mux := cmux.New(lis) + mux.SetReadTimeout(classificationTimeout) // Native gRPC: match HTTP/2 connections whose first request declares a // native gRPC content type (application/grpc, application/grpc+proto, // ...). The SETTINGS-writing writer unblocks gRPC clients, which send @@ -154,11 +159,8 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) // server afterwards. CAS has no gRPC-Web endpoint (spec D-011), so no // application/grpc-web* exclusion is needed. grpcL := mux.MatchWithWriters(cmux.HTTP2MatchHeaderFieldPrefixSendSettings("content-type", "application/grpc")) - // Every other connection (HTTP/1.1 and plain HTTP/2) belongs to the - // HTTP server. Mixed native-gRPC and plain-HTTP requests on one shared - // upstream HTTP/2 connection are not supported (spec D-014): supported - // proxies keep the two kinds of traffic on separate upstream pools, - // which is what the chart's path-split ingress configures. + // Other connections go to the HTTP server, which supports HTTP/1.1 only. + // Plain HTTP/2 and mixed connections are outside the D-017 contract. httpL := mux.Match(cmux.Any()) var opts = []http.ServerOption{ @@ -174,39 +176,16 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) opts = append(opts, http.Timeout(c.Http.Timeout.AsDuration())) } - // Parse the public key once on initialization instead of on every request - // TODO: implement jwks endpoint - publicKey, err := parsePublicKey(authConf, logger) - if err != nil { - return nil, err - } - srv := http.NewServer(opts...) downloadHandler := middlewares_http.AuthFromHeaderOrQueryParam(loadPublicKey(publicKey), claimsFunc(), casJWT.SigningMethod, downloadSvc) srv.Handle(service.DownloadPath, CORSMiddleware(c.GetHttp().GetCors().GetAllowOrigins(), downloadHandler)) api.RegisterStatusServiceHTTPServer(srv, service.NewStatusService(Version, providers)) - routes := srv.Handler srv.Protocols = new(h.Protocols) srv.Protocols.SetHTTP1(true) - srv.Protocols.SetHTTP2(true) - srv.Protocols.SetUnencryptedHTTP2(true) - // Bound how long a client may take to deliver its request headers - // (slowloris protection). Response streaming is not affected. + // Bound HTTP headers after handoff; streaming responses have no deadline. srv.ReadHeaderTimeout = 60 * time.Second - // One-way fallback (spec D-014): a connection cmux classified as HTTP - // may later carry a native gRPC request, for example from a proxy that - // pools both kinds of streams onto one upstream HTTP/2 connection. - // Dispatch those per request through grpc-go's HTTP adapter, with the - // same registered services and interceptors as the native transport. - srv.Handler = h.HandlerFunc(func(w h.ResponseWriter, r *h.Request) { - if r.ProtoMajor == 2 && isNativeGRPCContentType(r.Header.Get("Content-Type")) { - grpcServers.Handler.ServeHTTP(w, r) - return - } - routes.ServeHTTP(w, r) - }) log.NewHelper(logger).Infof("[shared] CAS endpoint (%s) listening on: %s", scheme, lis.Addr().String()) @@ -221,15 +200,6 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) }, nil } -// isNativeGRPCContentType reports whether a content type selects the native -// gRPC protocol family, excluding gRPC-Web. CAS has no gRPC-Web endpoint, but -// the exclusion keeps the fallback consistent with the control plane. -func isNativeGRPCContentType(contentType string) bool { - contentType = strings.ToLower(contentType) - return strings.HasPrefix(contentType, "application/grpc") && - !strings.HasPrefix(contentType, "application/grpc-web") -} - // claimsFunc returns the claims function for the JWT middleware that casts the claims to the correct type func claimsFunc() middlewares_http.ClaimsFunc { return func() jwt.Claims { @@ -266,4 +236,4 @@ func loadServerTLSConfig(c *conf.Server_TLS) (*tls.Config, error) { Certificates: []tls.Certificate{pair}, MinVersion: tls.VersionTLS12, NextProtos: []string{"h2", "http/1.1"}, }, nil -} \ No newline at end of file +} diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index 088ce02f6..0af0c19fc 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -32,7 +32,6 @@ import ( "os" "path/filepath" "sync" - "sync/atomic" "testing" "time" @@ -342,15 +341,8 @@ func casTestTLS(t testing.TB) *conf.Server_TLS { return &conf.Server_TLS{Certificate: certPath, PrivateKey: keyPath} } -// TestCASSharedConnectionReuse checks the separate-connection contract of the -// shared endpoint (spec D-014/D-015): through one port, native gRPC and HTTP -// each work on their own connections, a browser CORS preflight reaches the -// HTTP side, and an HTTP-owned HTTP/2 connection can still carry a later -// native gRPC request through the per-request ServeHTTP fallback. The reverse -// order, plain HTTP on a gRPC-classified connection, is not part of the -// contract; a proxy must keep the two kinds of traffic on separate upstream -// pools, which is what the chart's path-split ingress configures. -func TestCASSharedConnectionReuse(t *testing.T) { +// Each protocol uses its own connections through the shared port (D-017). +func TestCASSharedProtocols(t *testing.T) { for _, useTLS := range []bool{false, true} { t.Run(map[bool]string{false: "h2c", true: "TLS"}[useTLS], func(t *testing.T) { opts := casTestOpts(t) @@ -360,13 +352,7 @@ func TestCASSharedConnectionReuse(t *testing.T) { sharedURL, _ := startCASTestServers(t, opts) perform := func(kind string) { - var connections atomic.Int32 - tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // test certificate - DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { - connections.Add(1) - return (&net.Dialer{}).DialContext(ctx, network, addr) - }, - } + tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // test certificate if kind == casGRPCRequest { // Native gRPC arrives over HTTP/2, as the gRPC upstream // pool of a configured ingress dials it. @@ -617,3 +603,57 @@ func TestCASSharedShutdownDrainsUpload(t *testing.T) { require.FailNow(t, "shutdown did not complete after the upload") } } + +// Slow clients must not survive shutdown while cmux is still reading their first request. +func TestCASSharedShutdownClosesUnclassifiedConnections(t *testing.T) { + for _, useTLS := range []bool{false, true} { + t.Run(map[bool]string{false: "plaintext", true: "TLS handshake"}[useTLS], func(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + if useTLS { + opts.ServerConfig.Http.TlsConfig = casTestTLS(t) + } + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + endpoint, err := api.Endpoint() + require.NoError(t, err) + startCASTestTransport(t, api) + conn, err := net.Dial("tcp", endpoint.Host) + require.NoError(t, err) + defer conn.Close() + if !useTLS { + _, err = conn.Write([]byte("PRI ")) + require.NoError(t, err) + } + // Allow the connection to reach the classifier, including a TLS handshake. + time.Sleep(50 * time.Millisecond) + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + require.NoError(t, api.Stop(ctx)) + require.NoError(t, conn.SetReadDeadline(time.Now().Add(200*time.Millisecond))) + var data [1]byte + _, err = conn.Read(data[:]) + require.ErrorIs(t, err, io.EOF, "shutdown must close a socket still waiting for classification") + }) + } +} + +func TestCASSharedInitializationDoesNotBindOnInvalidKey(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + defer func() { require.NoError(t, servers.Handler.Stop(context.Background())) }() + reserved, err := net.Listen("tcp", casAnyAddr) + require.NoError(t, err) + opts.ServerConfig.Http.Addr = reserved.Addr().String() + require.NoError(t, reserved.Close()) + opts.AuthConfig.PublicKeyPath = "testdata/missing-public-key" + _, err = NewHTTPServer(opts, servers) + require.Error(t, err) + retry, err := net.Listen("tcp", opts.ServerConfig.Http.Addr) + require.NoError(t, err, "failed initialization must leave the listener address available") + require.NoError(t, retry.Close()) +} diff --git a/app/artifact-cas/internal/server/httpmetrics.go b/app/artifact-cas/internal/server/httpmetrics.go index b37b58576..03bd1b72c 100644 --- a/app/artifact-cas/internal/server/httpmetrics.go +++ b/app/artifact-cas/internal/server/httpmetrics.go @@ -1,5 +1,5 @@ // -// Copyright 2023 The Chainloop Authors. +// Copyright 2023-2026 The Chainloop Authors. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. @@ -16,6 +16,8 @@ package server import ( + "errors" + "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/conf" "github.com/go-kratos/kratos/v2/middleware/recovery" "github.com/go-kratos/kratos/v2/transport/http" @@ -29,6 +31,9 @@ type HTTPMetricsServer struct { // NewHTTPMetricsServer exposes the metrics endpoint func NewHTTPMetricsServer(c *conf.Server) (*HTTPMetricsServer, error) { + if c.GetHttpMetrics().GetTlsConfig() != nil { + return nil, errors.New("server.http_metrics.tls_config is unsupported; configure shared endpoint TLS under server.http.tls_config") + } var opts = []http.ServerOption{ http.Middleware( recovery.Recovery(), diff --git a/app/artifact-cas/internal/server/httpmetrics_test.go b/app/artifact-cas/internal/server/httpmetrics_test.go new file mode 100644 index 000000000..73065786b --- /dev/null +++ b/app/artifact-cas/internal/server/httpmetrics_test.go @@ -0,0 +1,42 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "testing" + + "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/conf" + "github.com/stretchr/testify/require" +) + +func TestCASMetricsTLSConfiguration(t *testing.T) { + for _, tc := range []struct { + name string + tls *conf.Server_TLS + }{ + {name: "plaintext"}, + {name: "empty TLS config", tls: &conf.Server_TLS{}}, + {name: "TLS config", tls: &conf.Server_TLS{Certificate: "cert.pem", PrivateKey: "key.pem"}}, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := NewHTTPMetricsServer(&conf.Server{HttpMetrics: &conf.Server_HTTP{Addr: casAnyAddr, TlsConfig: tc.tls}}) + if tc.tls != nil { + require.ErrorContains(t, err, "server.http_metrics.tls_config is unsupported") + } else { + require.NoError(t, err) + } + }) + } +} diff --git a/app/artifact-cas/internal/server/ingress_test.go b/app/artifact-cas/internal/server/ingress_test.go new file mode 100644 index 000000000..da709a989 --- /dev/null +++ b/app/artifact-cas/internal/server/ingress_test.go @@ -0,0 +1,208 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "bytes" + "context" + "crypto/sha256" + "crypto/tls" + "encoding/hex" + "errors" + "fmt" + "io" + "net" + "net/http" + "os" + "os/exec" + "strconv" + "testing" + "time" + + v1 "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" + casJWT "github.com/chainloop-dev/chainloop/internal/robotaccount/cas" + "github.com/stretchr/testify/require" + "google.golang.org/genproto/googleapis/bytestream" + grpcLib "google.golang.org/grpc" + "google.golang.org/grpc/credentials" + reflection "google.golang.org/grpc/reflection/grpc_reflection_v1" +) + +// This deploys the chart's CAS Service and Ingress objects in a disposable +// namespace of a caller-supplied local cluster with ingress-nginx installed. +// The CAS runs in this test process; BACKEND_IP must be reachable from its nodes. +// See app/artifact-cas/README.md for the kind setup and invocation. +func TestCASIngressNGINX(t *testing.T) { + kubeconfig := os.Getenv("CAS_INGRESS_KUBECONFIG") + if kubeconfig == "" { + t.Skip("set CAS_INGRESS_KUBECONFIG and CAS_INGRESS_BACKEND_IP for the local ingress-nginx test") + } + backendIP := os.Getenv("CAS_INGRESS_BACKEND_IP") + require.NotNil(t, net.ParseIP(backendIP), "CAS_INGRESS_BACKEND_IP must be reachable from the cluster nodes") + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + namespace := fmt.Sprintf("cas-cmux-%d", time.Now().UnixNano()) + kubectl := func(input []byte, args ...string) []byte { + t.Helper() + full := append([]string{"--kubeconfig", kubeconfig, "--namespace", namespace}, args...) + cmd := exec.CommandContext(ctx, "kubectl", full...) //nolint:gosec // caller-selected local kubeconfig passed as an argument, without a shell + cmd.Stdin = bytes.NewReader(input) + out, err := cmd.CombinedOutput() + require.NoError(t, err, "%s", out) + return out + } + require.Equal(t, "1", string(kubectl(nil, "get", "configmap", "ingress-nginx-controller", "--namespace", "ingress-nginx", "-o=jsonpath={.data.worker-processes}")), "use one controller worker to exercise both protocols in the same connection cache") + kubectl(nil, "create", "namespace", namespace) + t.Cleanup(func() { + cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cleanupCancel() + out, err := exec.CommandContext(cleanupCtx, "kubectl", "--kubeconfig", kubeconfig, "delete", "namespace", namespace, "--wait=false").CombinedOutput() //nolint:gosec // fixed namespace prefix and caller-selected local kubeconfig, without a shell + require.NoError(t, err, "%s", out) + }) + opts := casTestOpts(t) + opts.ServerConfig.Http.Addr = "0.0.0.0:0" + opts.ServerConfig.LegacyGrpcListener = new(false) + sharedURL, _ := startCASTestServers(t, opts) + _, port, err := net.SplitHostPort(sharedURL[len("http://"):]) + require.NoError(t, err) + const hostname = "example.com" + cert := casTestTLS(t) + kubectl(nil, "create", "secret", "tls", hostname+"-tls", "--cert", cert.Certificate, "--key", cert.PrivateKey) + helm := exec.CommandContext(ctx, "helm", "template", "probe", "../../../../deployment/chainloop", "--namespace", namespace, + "--set", "development=true", "--set", "fullnameOverride=cas-probe", + "--set", "cas.ingress.enabled=true", "--set", "cas.ingress.ingressClassName=nginx", + "--set", "cas.ingress.hostname="+hostname, "--set", "cas.ingress.tls=true", + "--show-only", "templates/cas/service-http.yaml", "--show-only", "templates/cas/ingress.yaml", "--show-only", "templates/cas/ingress-grpc-shared.yaml") + helm.Env = append(os.Environ(), "HELM_PLUGINS="+t.TempDir()) + manifest, err := helm.Output() + require.NoError(t, err) + kubectl(manifest, "apply", "-f", "-") + endpointSlice := fmt.Sprintf(`apiVersion: discovery.k8s.io/v1 +kind: EndpointSlice +metadata: + name: cas-probe-cas-test + labels: + kubernetes.io/service-name: cas-probe-cas + endpointslice.kubernetes.io/managed-by: chainloop-test +addressType: IPv4 +ports: + - name: http + protocol: TCP + port: %s + - name: grpc-shared + protocol: TCP + port: %s +endpoints: + - addresses: ["%s"] + conditions: + ready: true +`, port, port, backendIP) + kubectl([]byte(endpointSlice), "apply", "-f", "-") + lis, err := net.Listen("tcp", casAnyAddr) + require.NoError(t, err) + addr := lis.Addr().String() + localPort := lis.Addr().(*net.TCPAddr).Port + require.NoError(t, lis.Close()) + forward := exec.CommandContext(ctx, "kubectl", "--kubeconfig", kubeconfig, "--namespace", "ingress-nginx", "port-forward", "service/ingress-nginx-controller", strconv.Itoa(localPort)+":443") //nolint:gosec // caller-selected local kubeconfig and numeric port, without a shell + var logs bytes.Buffer + forward.Stdout, forward.Stderr = &logs, &logs + require.NoError(t, forward.Start()) + t.Cleanup(func() { + _ = forward.Process.Kill() + _ = forward.Wait() + t.Log(logs.String()) + }) + require.Eventually(t, func() bool { + c, err := net.DialTimeout("tcp", addr, 50*time.Millisecond) + if err != nil { + return false + } + _ = c.Close() + return true + }, 10*time.Second, 50*time.Millisecond) + conn, err := grpcLib.NewClient(addr, grpcLib.WithAuthority(hostname), grpcLib.WithTransportCredentials( + credentials.NewTLS(&tls.Config{InsecureSkipVerify: true}))) //nolint:gosec // local test certificate + require.NoError(t, err) + defer conn.Close() + statusClient := v1.NewStatusServiceClient(conn) + require.Eventually(t, func() bool { + callCtx, callCancel := context.WithTimeout(ctx, time.Second) + defer callCancel() + _, err := statusClient.Statusz(callCtx, &v1.StatuszRequest{}) + return err == nil + }, 30*time.Second, 500*time.Millisecond, "chart gRPC paths must reach the CAS") + client := bytestream.NewByteStreamClient(conn) + content := bytes.Repeat([]byte("CAS ingress streaming\n"), 4096) + sum := sha256.Sum256(content) + digest := hex.EncodeToString(sum[:]) + uploadToken := casTestToken(t, casJWT.Uploader, 0, time.Hour) + downloadToken := casTestToken(t, casJWT.Downloader, 0, time.Hour) + require.NoError(t, casTestUpload(t, client, uploadToken, digest, bytes.NewReader(content), int64(len(content)))) + callCtx, callCancel := casTestContext(downloadToken) + defer callCancel() + refl, err := reflection.NewServerReflectionClient(conn).ServerReflectionInfo(ctx) + require.NoError(t, err) + require.NoError(t, refl.Send(&reflection.ServerReflectionRequest{MessageRequest: &reflection.ServerReflectionRequest_ListServices{ListServices: ""}})) + _, err = refl.Recv() + require.NoError(t, err, "reflection must use a gRPC upstream too") + require.NoError(t, refl.CloseSend()) + // Warm both pools repeatedly, in both directions. Reusing a gRPC socket for + // HTTP (or vice versa) fails after the first successful request on each. + for _, http2Client := range []bool{false, true} { + t.Run(fmt.Sprintf("browser_HTTP2_%t", http2Client), func(t *testing.T) { + tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, ForceAttemptHTTP2: http2Client, //nolint:gosec // local test certificate + DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) { + return (&net.Dialer{}).DialContext(ctx, network, addr) + }} + defer tr.CloseIdleConnections() + browser := &http.Client{Transport: tr, Timeout: 5 * time.Second} + for range 20 { + stream, err := client.Read(callCtx, &bytestream.ReadRequest{ResourceName: digest}) + require.NoError(t, err) + hash := sha256.New() + for { + chunk, err := stream.Recv() + if errors.Is(err, io.EOF) { + break + } + require.NoError(t, err) + _, err = hash.Write(chunk.Data) + require.NoError(t, err) + } + require.Equal(t, digest, hex.EncodeToString(hash.Sum(nil))) + req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://"+hostname+"/download/sha256:"+digest, nil) + require.NoError(t, err) + req.Header.Set("Authorization", "Bearer "+downloadToken) + resp, err := browser.Do(req) + require.NoError(t, err) + require.Equal(t, http.StatusOK, resp.StatusCode) + require.Equal(t, map[bool]int{false: 1, true: 2}[http2Client], resp.ProtoMajor) + hash.Reset() + _, err = io.Copy(hash, resp.Body) + require.NoError(t, resp.Body.Close()) + require.NoError(t, err) + require.Equal(t, digest, hex.EncodeToString(hash.Sum(nil))) + _, err = statusClient.Statusz(ctx, &v1.StatuszRequest{}) + require.NoError(t, err, "gRPC after the HTTP pool is warm") + resp, err = browser.Get("https://" + hostname + "/statusz") + require.NoError(t, err) + _, err = io.Copy(io.Discard, resp.Body) + require.NoError(t, resp.Body.Close()) + require.NoError(t, err) + require.Equal(t, http.StatusOK, resp.StatusCode) + } + }) + } +} diff --git a/app/artifact-cas/internal/server/streaming_bench_test.go b/app/artifact-cas/internal/server/streaming_bench_test.go index fc7fb1587..9bfb0c667 100644 --- a/app/artifact-cas/internal/server/streaming_bench_test.go +++ b/app/artifact-cas/internal/server/streaming_bench_test.go @@ -34,7 +34,8 @@ import ( ) // This compares the production CAS service and interceptors over the unchanged -// native transport and the HTTP adapter, with the same disk-backed test backend. +// legacy listener and the shared cmux endpoint, both on the native transport +// with the same disk-backed test backend. // It is a transport experiment, not the original R-003 release/S3 qualification. // Use a disk-backed TMPDIR: /tmp may be tmpfs on development machines. func BenchmarkCASByteStreamTransport(b *testing.B) { diff --git a/deployment/chainloop/Chart.yaml b/deployment/chainloop/Chart.yaml index f05606c26..f10ac5164 100644 --- a/deployment/chainloop/Chart.yaml +++ b/deployment/chainloop/Chart.yaml @@ -7,7 +7,7 @@ description: Chainloop is an open source software supply chain control plane, a type: application # Bump the patch (not minor, not major) version on each change in the Chart Source code -version: 1.468.2 +version: 1.468.3 # Do not update appVersion, this is handled automatically by the release process appVersion: v1.125.0 diff --git a/deployment/chainloop/README.md b/deployment/chainloop/README.md index 320c0d47f..1fa2c08ec 100644 --- a/deployment/chainloop/README.md +++ b/deployment/chainloop/README.md @@ -245,7 +245,9 @@ The control plane ingress serves HTTP, gRPC-Web, and native gRPC on the same hos Its HTTP server routes each request, so these protocols can share an upstream HTTP/2 connection. The existing gRPC ingress stays available. Existing clients can keep their current endpoints. To test the shared endpoint, point a gRPC client to the control plane HTTP hostname on port 443. -The CAS shared endpoint classifies each connection with cmux and keeps the native gRPC transport for ByteStream (spec D-017). One hostname serves both protocols through two upstream pools: `cas.ingress` routes the HTTP paths (`/download`, `/statusz`, `/infoz`) with an HTTP/1.1 upstream, and a second Ingress rendered from `cas.ingress.grpc` routes the `/cas.v1.*` RPC paths with the gRPC backend protocol. Do not set a gRPC backend protocol in `cas.ingress.annotations`: that object belongs to the HTTP routes. The gRPC-pool protocol is configured in `cas.ingress.grpc.annotations` (set `backend-protocol: GRPCS` there when TLS runs to the pod). `cas.ingressAPI` and the native gRPC endpoint remain independent, and the control plane continues to use that legacy endpoint during migration. CAS does not add gRPC-Web or Connect. Controller qualification (F5 VirtualServer for the same-host split, Traefik's per-Service scheme limitation) and the full streaming-performance evidence remain open; see the [local experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). +The CAS shared endpoint classifies connections with cmux and keeps the native gRPC transport for ByteStream (spec D-017). `cas.ingress` sends HTTP routes over HTTP/1.1. The second Ingress, `cas.ingress.grpc`, sends the Google ByteStream, CAS, health, reflection, metadata, and channelz RPCs over gRPC. Both target the same CAS application listener through one Service. Its `http` port defaults to 80 and its `grpc-shared` alias defaults to 81 (`cas.service.ports.grpcShared`). Ingress-nginx uses pod endpoints for HTTP and the ClusterIP for gRPC. The distinct Service ports and upstream addresses isolate its connection cache (D-019). Keep the port numbers different and at least one upstream on the ClusterIP. The chart rejects unsafe combinations and headless Services for this arrangement. + +Set gRPC transport annotations under `cas.ingress.grpc.annotations`, such as `backend-protocol: GRPCS` for TLS to the pod. HTTP routes use HTTP or HTTPS with HTTP/1.1 upstream. Both Ingress objects share a TLS Secret when `cas.ingress.tls=true`. Only one requests the certificate from cert-manager. A pre-created Secret also works without issuer annotations. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. The shared chart routing has a local single-worker ingress-nginx check. F5, Traefik, and the full streaming-performance evidence still need qualification. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). The chart configures HTTP/2 upstream traffic for these controllers: @@ -836,6 +838,7 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | `cas.service.type` | Service type | `ClusterIP` | | `cas.service.ports.http` | cas service HTTP port | `80` | +| `cas.service.ports.grpcShared` | Service port alias for shared-ingress gRPC (same CAS HTTP listener) | `81` | | `cas.service.ports.https` | cas service HTTPS port | `443` | | `cas.service.nodePorts.http` | Node port for HTTP | `""` | | `cas.service.nodePorts.https` | Node port for HTTPS | `""` | diff --git a/deployment/chainloop/chart_test.go b/deployment/chainloop/chart_test.go index b1dbdc6f2..ada05c1e2 100644 --- a/deployment/chainloop/chart_test.go +++ b/deployment/chainloop/chart_test.go @@ -21,13 +21,23 @@ import ( "io" "os" "os/exec" + "strings" "testing" "time" + api "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" "github.com/stretchr/testify/require" + "google.golang.org/genproto/googleapis/bytestream" "gopkg.in/yaml.v3" ) +const ( + casIngressEnabled = "cas.ingress.enabled=true" + helmSet = "--set" + helmSetString = "--set-string" + annotationTrue = "true" +) + // The control plane serves every protocol per request on its shared ingress // (replacement spec D-001/D-002), so the whole ingress uses the gRPC backend // protocol and the Traefik h2c scheme. The CAS classifies connections with @@ -39,24 +49,27 @@ func TestSharedIngress(t *testing.T) { t.Run("cas", func(t *testing.T) { testSharedCASIngress(t) }) } -func renderChart(t *testing.T, args []string) map[string]chartResource { +func chartTemplate(t *testing.T, args []string) ([]byte, error) { t.Helper() helm, err := exec.LookPath("helm") if err != nil { t.Skip("helm is required for chart rendering tests") } - full := append([]string{"template", "test", ".", "--set", "development=true", - "--set", "controlplane.ingressAPI.enabled=true", "--set", "cas.ingressAPI.enabled=true"}, args...) + full := append([]string{"template", "test", ".", helmSet, "development=true", + helmSet, "controlplane.ingressAPI.enabled=true", helmSet, "cas.ingressAPI.enabled=true"}, args...) ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() cmd := exec.CommandContext(ctx, helm, full...) // Local plugins are unrelated to chart rendering and can print // startup warnings on stdout before the YAML document. cmd.Env = append(os.Environ(), "HELM_PLUGINS="+t.TempDir()) - var stderr bytes.Buffer - cmd.Stderr = &stderr - out, err := cmd.Output() - require.NoError(t, err, "%s", stderr.String()) + return cmd.CombinedOutput() +} + +func renderChart(t *testing.T, args []string) map[string]chartResource { + t.Helper() + out, err := chartTemplate(t, args) + require.NoError(t, err, "%s", out) docs := yaml.NewDecoder(bytes.NewReader(out)) resources := make(map[string]chartResource) for { @@ -87,18 +100,18 @@ func testSharedControlplaneIngress(t *testing.T) { traefikScheme string }{ {name: "defaults", fullname: chartFullname, backendProtocol: grpcBackend, traefikScheme: h2cScheme}, - {name: "custom name", args: []string{"--set", "fullnameOverride=shared-api"}, fullname: "shared-api", backendProtocol: grpcBackend, traefikScheme: h2cScheme}, + {name: "custom name", args: []string{helmSet, "fullnameOverride=shared-api"}, fullname: "shared-api", backendProtocol: grpcBackend, traefikScheme: h2cScheme}, { name: "common annotations merge", - args: []string{"--set-string", `commonAnnotations.nginx\.org/http2=false`}, + args: []string{helmSetString, `commonAnnotations.nginx\.org/http2=false`}, fullname: chartFullname, backendProtocol: grpcBackend, traefikScheme: h2cScheme, }, { name: "encrypted upstream overrides", args: []string{ - "--set-string", `controlplane.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, - "--set-string", `controlplane.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, + helmSetString, `controlplane.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, + helmSetString, `controlplane.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, }, fullname: chartFullname, backendProtocol: "GRPCS", traefikScheme: "https", }, @@ -106,12 +119,12 @@ func testSharedControlplaneIngress(t *testing.T) { t.Run(tc.name, func(t *testing.T) { serviceName := tc.fullname + "-controlplane" resources := renderChart(t, append(tc.args, - "--set", "controlplane.ingress.enabled=true", "--set", "controlplane.ingress.hostname=shared.example.test")) + helmSet, "controlplane.ingress.enabled=true", helmSet, "controlplane.ingress.hostname=shared.example.test")) ingress, ok := resources["Ingress/"+serviceName] require.True(t, ok) require.Equal(t, tc.backendProtocol, ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) require.Equal(t, serviceName, ingress.Metadata.Annotations["nginx.org/grpc-services"]) - require.Equal(t, "true", ingress.Metadata.Annotations["nginx.org/http2"]) + require.Equal(t, annotationTrue, ingress.Metadata.Annotations["nginx.org/http2"]) require.Len(t, ingress.Spec.Rules, 1) require.Equal(t, "shared.example.test", ingress.Spec.Rules[0].Host) require.Len(t, ingress.Spec.Rules[0].HTTP.Paths, 1) @@ -146,30 +159,30 @@ func testSharedCASIngress(t *testing.T) { grpcHTTP2 string traefikScheme string }{ - {name: "defaults", fullname: chartFullname, grpcProtocol: grpcBackend, grpcHTTP2: "true"}, - {name: "custom name", args: []string{"--set", "fullnameOverride=shared-api"}, fullname: "shared-api", grpcProtocol: grpcBackend, grpcHTTP2: "true"}, + {name: "defaults", fullname: chartFullname, grpcProtocol: grpcBackend, grpcHTTP2: annotationTrue}, + {name: "custom name", args: []string{helmSet, "fullnameOverride=shared-api"}, fullname: "shared-api", grpcProtocol: grpcBackend, grpcHTTP2: annotationTrue}, { // A global annotation reaches both Ingress objects; on the gRPC // Ingress it overrides the chart default. name: "common annotations merge", - args: []string{"--set-string", `commonAnnotations.nginx\.org/http2=false`}, + args: []string{helmSetString, `commonAnnotations.nginx\.org/http2=false`}, fullname: chartFullname, grpcProtocol: grpcBackend, sharedHTTP2: "false", grpcHTTP2: "false", }, { name: "encrypted upstream overrides", args: []string{ - "--set-string", `cas.ingress.grpc.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, - "--set-string", `cas.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, + helmSetString, `cas.ingress.grpc.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, + helmSetString, `cas.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, }, - fullname: chartFullname, - grpcProtocol: "GRPCS", sharedHTTP2: "", grpcHTTP2: "true", traefikScheme: "https", + fullname: chartFullname, + grpcProtocol: "GRPCS", sharedHTTP2: "", grpcHTTP2: annotationTrue, traefikScheme: "https", }, } { t.Run(tc.name, func(t *testing.T) { serviceName := tc.fullname + "-cas" resources := renderChart(t, append(tc.args, - "--set", "cas.ingress.enabled=true", "--set", "cas.ingress.hostname=shared.example.test")) + helmSet, casIngressEnabled, helmSet, "cas.ingress.hostname=shared.example.test")) // The shared HTTP ingress dials HTTP/1.1: no gRPC annotations by default. ingress, ok := resources["Ingress/"+serviceName] @@ -190,21 +203,29 @@ func testSharedCASIngress(t *testing.T) { grpcIngress, ok := resources["Ingress/"+serviceName+"-grpc"] require.True(t, ok) require.Equal(t, tc.grpcProtocol, grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) + require.Equal(t, annotationTrue, grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/service-upstream"]) + require.Empty(t, ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/service-upstream"]) require.Equal(t, serviceName, grpcIngress.Metadata.Annotations["nginx.org/grpc-services"]) require.Equal(t, tc.grpcHTTP2, grpcIngress.Metadata.Annotations["nginx.org/http2"]) require.Equal(t, "100m", grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/proxy-body-size"]) require.Equal(t, "3M", grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/client-body-buffer-size"]) require.Len(t, grpcIngress.Spec.Rules, 1) require.Equal(t, "shared.example.test", grpcIngress.Spec.Rules[0].Host) - require.Len(t, grpcIngress.Spec.Rules[0].HTTP.Paths, 2) + require.Len(t, grpcIngress.Spec.Rules[0].HTTP.Paths, 8) for _, p := range grpcIngress.Spec.Rules[0].HTTP.Paths { require.Equal(t, serviceName, p.Backend.Service.Name) - require.Equal(t, "http", p.Backend.Service.Port.Name) + require.Equal(t, "grpc-shared", p.Backend.Service.Port.Name) } - require.Equal(t, "/cas.v1.ByteStream/", grpcIngress.Spec.Rules[0].HTTP.Paths[0].Path) + require.Equal(t, "/google.bytestream.ByteStream/", grpcIngress.Spec.Rules[0].HTTP.Paths[0].Path) require.Equal(t, "/cas.v1.ResourceService/", grpcIngress.Spec.Rules[0].HTTP.Paths[1].Path) svc := resources["Service/"+serviceName] + require.Len(t, svc.Spec.Ports, 2) + require.Equal(t, "http", svc.Spec.Ports[0].Name) + require.Equal(t, "http", svc.Spec.Ports[0].TargetPort) + require.Equal(t, "grpc-shared", svc.Spec.Ports[1].Name) + require.Equal(t, 81, svc.Spec.Ports[1].Port) + require.Equal(t, "http", svc.Spec.Ports[1].TargetPort) require.Equal(t, tc.traefikScheme, svc.Metadata.Annotations["traefik.ingress.kubernetes.io/service.serversscheme"]) casLegacy := resources["Ingress/"+tc.fullname+"-cas-api"] require.Equal(t, grpcBackend, casLegacy.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) @@ -221,6 +242,15 @@ type chartResource struct { Annotations map[string]string `yaml:"annotations"` } `yaml:"metadata"` Spec struct { + Ports []struct { + Name string `yaml:"name"` + Port int `yaml:"port"` + TargetPort string `yaml:"targetPort"` + } `yaml:"ports"` + TLS []struct { + Hosts []string `yaml:"hosts"` + SecretName string `yaml:"secretName"` + } `yaml:"tls"` Rules []struct { Host string `yaml:"host"` HTTP struct { @@ -238,4 +268,88 @@ type chartResource struct { } `yaml:"http"` } `yaml:"rules"` } `yaml:"spec"` -} \ No newline at end of file +} + +func TestCASSharedIngressRPCPaths(t *testing.T) { + resources := renderChart(t, []string{helmSet, casIngressEnabled}) + ingress := resources["Ingress/test-chainloop-cas-grpc"] + // Derive the application routes from the generated descriptors, so a typo + // in the values cannot become the expected result as well. + for _, name := range []string{ + string(bytestream.File_google_bytestream_bytestream_proto.Services().Get(0).FullName()), + api.ResourceService_ServiceDesc.ServiceName, + api.StatusService_ServiceDesc.ServiceName, + "grpc.health.v1.Health", "grpc.reflection.v1.ServerReflection", + "grpc.reflection.v1alpha.ServerReflection", "kratos.api.Metadata", + "grpc.channelz.v1.Channelz", + } { + covered := false + for _, rule := range ingress.Spec.Rules { + for _, path := range rule.HTTP.Paths { + covered = covered || strings.HasPrefix("/"+name+"/", path.Path) + } + } + require.True(t, covered, "no native gRPC route for %s", name) + } +} + +func TestCASSharedIngressTLS(t *testing.T) { + for _, annotations := range []string{"cas.ingress.annotations", "cas.ingress.grpc.annotations", "commonAnnotations"} { + t.Run(annotations, func(t *testing.T) { + resources := renderChart(t, []string{helmSet, casIngressEnabled, helmSet, "cas.ingress.tls=true", + helmSetString, annotations + `.cert-manager\.io/cluster-issuer=test-issuer`}) + owners := 0 + for _, name := range []string{"test-chainloop-cas", "test-chainloop-cas-grpc"} { + ingress := resources["Ingress/"+name] + if ingress.Metadata.Annotations["cert-manager.io/cluster-issuer"] != "" { + owners++ + } + require.Len(t, ingress.Spec.TLS, 1, "both routes must reference the certificate") + require.Equal(t, "cas.dev.local-tls", ingress.Spec.TLS[0].SecretName) + require.Equal(t, []string{"cas.dev.local"}, ingress.Spec.TLS[0].Hosts) + } + require.Equal(t, 1, owners, "only one Ingress should request the shared certificate") + }) + } +} + +func TestCASIngressInvalidPoolConfiguration(t *testing.T) { + for _, tc := range []struct { + name string + args []string + message string + }{ + {name: "pod endpoints for both protocols", args: []string{helmSetString, `cas.ingress.grpc.annotations.nginx\.ingress\.kubernetes\.io/service-upstream=false`}, message: "requires separate ingress-nginx upstream addresses"}, + {name: "headless service", args: []string{helmSet, "cas.service.clusterIP=None"}, message: "requires a ClusterIP Service"}, + {name: "same service port", args: []string{helmSet, "cas.service.ports.grpcShared=80"}, message: "grpcShared must differ"}, + {name: "gRPC on HTTP routes", args: []string{helmSetString, `cas.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPC`}, message: "must use an HTTP/1.1 upstream"}, + } { + t.Run(tc.name, func(t *testing.T) { + out, err := chartTemplate(t, append(tc.args, helmSet, casIngressEnabled)) + require.Error(t, err) + require.Contains(t, string(out), tc.message) + }) + } +} + +func TestCASSharedServicePortAlias(t *testing.T) { + for _, tc := range []struct { + name string + args []string + alias bool + }{ + {name: "ingress disabled"}, + {name: "HTTP only", args: []string{helmSet, casIngressEnabled, "--set-json", "cas.ingress.grpc.paths=[]"}}, + {name: "both use ClusterIP", args: []string{helmSet, casIngressEnabled, helmSetString, `cas.ingress.annotations.nginx\.ingress\.kubernetes\.io/service-upstream=true`}, alias: true}, + } { + t.Run(tc.name, func(t *testing.T) { + resources := renderChart(t, tc.args) + ports := resources["Service/test-chainloop-cas"].Spec.Ports + if tc.alias { + require.Len(t, ports, 2) + } else { + require.Len(t, ports, 1) + } + }) + } +} diff --git a/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml b/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml index 7d0b1b85c..4f9911ec5 100644 --- a/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml +++ b/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml @@ -8,27 +8,49 @@ gRPC routes of the shared CAS endpoint (spec D-018). Rendered as a second Ingress on the same hostname as cas/ingress.yaml, with its own annotations: controllers that set the backend protocol per Ingress resource (community nginx `backend-protocol`) need a dedicated object for the gRPC paths, so the -gRPC upstream pool keeps the HTTP/2 (grpc_pass) transport while the HTTP -routes keep an HTTP/1.1 upstream. Controllers that conflict on unrelated +gRPC routes use HTTP/2 (grpc_pass) while HTTP routes keep HTTP/1.1. +ingress-nginx additionally dials the ClusterIP for gRPC and pod endpoints +for HTTP, using distinct Service port names as backend identities. Its global +keepalive pool cannot reuse a socket across protocols. Controllers that conflict on unrelated same-host Ingress objects (F5 NGINX) need their own resource instead; see the chart README. */}} {{- if and .Values.cas.ingress.enabled .Values.cas.ingress.grpc.paths }} {{- $fullName := include "chainloop.cas.fullname" . -}} {{- $grpcName := printf "%s-grpc" $fullName -}} +{{- $annotations := include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.grpc.annotations .Values.commonAnnotations .Values.cas.ingress.grpc.defaultAnnotations) "context" .) | fromYaml -}} +{{- if eq (toString .Values.cas.service.ports.grpcShared) (toString (coalesce .Values.cas.service.port .Values.cas.service.ports.http)) -}} + {{- fail "cas.service.ports.grpcShared must differ from the HTTP Service port to isolate ingress upstream connections" -}} +{{- end -}} +{{- $httpAnnotations := include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.annotations .Values.commonAnnotations) "context" .) | fromYaml -}} +{{- if has (upper (default "" (index $annotations "nginx.ingress.kubernetes.io/backend-protocol"))) (list "GRPC" "GRPCS") -}} + {{- if and (ne (default "false" (index $annotations "nginx.ingress.kubernetes.io/service-upstream")) "true") (ne (default "false" (index $httpAnnotations "nginx.ingress.kubernetes.io/service-upstream")) "true") -}} + {{- fail "CAS cmux requires separate ingress-nginx upstream addresses: set service-upstream=true on cas.ingress.grpc (the default) or cas.ingress" -}} + {{- end -}} + {{- if eq .Values.cas.service.clusterIP "None" -}} + {{- fail "CAS cmux ingress-nginx routing requires a ClusterIP Service; cas.service.clusterIP cannot be None" -}} + {{- end -}} + {{- if has (upper (default "HTTP" (index $httpAnnotations "nginx.ingress.kubernetes.io/backend-protocol"))) (list "GRPC" "GRPCS") -}} + {{- fail "cas.ingress must use an HTTP/1.1 upstream (HTTP or HTTPS); set the gRPC protocol only in cas.ingress.grpc.annotations" -}} + {{- end -}} +{{- end -}} +{{- /* Only one Ingress requests the shared certificate. If the HTTP Ingress + requests it, the gRPC Ingress only references its Secret. */ -}} +{{- if include "common.ingress.certManagerRequest" (dict "annotations" $httpAnnotations) -}} + {{- range $key, $_ := deepCopy $annotations -}} + {{- if or (hasPrefix "cert-manager.io/" $key) (hasPrefix "acme.cert-manager.io/" $key) (eq $key "kubernetes.io/tls-acme") -}} + {{- $_ := unset $annotations $key -}} + {{- end -}} + {{- end -}} +{{- end -}} apiVersion: {{ include "common.capabilities.ingress.apiVersion" . }} kind: Ingress metadata: name: {{ $grpcName }} namespace: {{ include "common.names.namespace" . | quote }} labels: {{- include "chainloop.cas.labels" . | nindent 4 }} - {{- if or .Values.cas.ingress.grpc.annotations .Values.cas.ingress.grpc.defaultAnnotations .Values.commonAnnotations }} - {{- /* cas.ingress.annotations stay on the HTTP ingress only: a gRPC backend - protocol set there would reach the HTTP routes. The gRPC-path Ingress - takes its annotations from cas.ingress.grpc.annotations and the - common ones; earlier in the list wins. */}} - {{- $annotations := include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.grpc.annotations .Values.commonAnnotations .Values.cas.ingress.grpc.defaultAnnotations) "context" .) }} - annotations: {{- include "common.tplvalues.render" (dict "value" $annotations "context" $) | nindent 4 }} + {{- with $annotations }} + annotations: {{- include "common.tplvalues.render" (dict "value" . "context" $) | nindent 4 }} {{- end }} spec: {{- if and .Values.cas.ingress.ingressClassName (eq "true" (include "common.ingress.supportsIngressClassname" .)) }} @@ -44,12 +66,12 @@ spec: {{- if eq "true" (include "common.ingress.supportsPathType" $) }} pathType: {{ default "ImplementationSpecific" $.Values.cas.ingress.pathType }} {{- end }} - backend: {{- include "common.ingress.backend" (dict "serviceName" $fullName "servicePort" "http" "context" $) | nindent 14 }} + backend: {{- include "common.ingress.backend" (dict "serviceName" $fullName "servicePort" "grpc-shared" "context" $) | nindent 14 }} {{- end }} {{- end }} - {{- if or (and .Values.cas.ingress.tls (or (include "common.ingress.certManagerRequest" (dict "annotations" (include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.grpc.annotations .Values.commonAnnotations) "context" .) | fromYaml))) .Values.cas.ingress.selfSigned)) .Values.cas.ingress.extraTls }} + {{- if or .Values.cas.ingress.tls .Values.cas.ingress.extraTls }} tls: - {{- if and .Values.cas.ingress.tls (or (include "common.ingress.certManagerRequest" ( dict "annotations" .Values.cas.ingress.annotations )) .Values.cas.ingress.selfSigned) }} + {{- if .Values.cas.ingress.tls }} - hosts: - {{ .Values.cas.ingress.hostname | quote }} secretName: {{ printf "%s-tls" .Values.cas.ingress.hostname }} diff --git a/deployment/chainloop/templates/cas/ingress.yaml b/deployment/chainloop/templates/cas/ingress.yaml index a68011961..beb11b7e9 100644 --- a/deployment/chainloop/templates/cas/ingress.yaml +++ b/deployment/chainloop/templates/cas/ingress.yaml @@ -46,9 +46,9 @@ spec: {{- if .Values.cas.ingress.extraRules }} {{- include "common.tplvalues.render" (dict "value" .Values.cas.ingress.extraRules "context" $) | nindent 4 }} {{- end }} - {{- if or (and .Values.cas.ingress.tls (or (include "common.ingress.certManagerRequest" ( dict "annotations" .Values.cas.ingress.annotations )) .Values.cas.ingress.selfSigned)) .Values.cas.ingress.extraTls }} + {{- if or .Values.cas.ingress.tls .Values.cas.ingress.extraTls }} tls: - {{- if and .Values.cas.ingress.tls (or (include "common.ingress.certManagerRequest" ( dict "annotations" .Values.cas.ingress.annotations )) .Values.cas.ingress.selfSigned) }} + {{- if .Values.cas.ingress.tls }} - hosts: - {{ .Values.cas.ingress.hostname | quote }} secretName: {{ printf "%s-tls" .Values.cas.ingress.hostname }} diff --git a/deployment/chainloop/templates/cas/service-http.yaml b/deployment/chainloop/templates/cas/service-http.yaml index 637ead5d4..bb32436d2 100644 --- a/deployment/chainloop/templates/cas/service-http.yaml +++ b/deployment/chainloop/templates/cas/service-http.yaml @@ -37,15 +37,21 @@ spec: - name: http {{- $port := coalesce .Values.cas.service.port .Values.cas.service.ports.http }} port: {{ $port }} - {{- if not (eq $port .Values.cas.containerPorts.http) }} - targetPort: {{ .Values.cas.containerPorts.http }} - {{- end }} + targetPort: http protocol: TCP {{- if and (or (eq .Values.cas.service.type "NodePort") (eq .Values.cas.service.type "LoadBalancer")) (not (empty .Values.cas.service.nodePorts.http)) }} nodePort: {{ .Values.cas.service.nodePorts.http }} {{- else if eq .Values.cas.service.type "ClusterIP" }} nodePort: null {{- end }} + {{- if and .Values.cas.ingress.enabled .Values.cas.ingress.grpc.paths }} + # This alias and http both target the one shared CAS listener. Distinct + # Service ports give ingress-nginx separate backend identities. + - name: grpc-shared + port: {{ .Values.cas.service.ports.grpcShared }} + targetPort: http + protocol: TCP + {{- end }} {{- if .Values.cas.service.extraPorts }} {{- include "common.tplvalues.render" (dict "value" .Values.cas.service.extraPorts "context" $) | nindent 4 }} {{- end }} diff --git a/deployment/chainloop/values.yaml b/deployment/chainloop/values.yaml index d29cc0ffa..d4ceded4d 100644 --- a/deployment/chainloop/values.yaml +++ b/deployment/chainloop/values.yaml @@ -1096,10 +1096,12 @@ cas: type: ClusterIP ## @param cas.service.ports.http cas service HTTP port + ## @param cas.service.ports.grpcShared Service port alias for shared-ingress gRPC (same CAS HTTP listener). ## @param cas.service.ports.https cas service HTTPS port ## ports: http: 80 + grpcShared: 81 https: 443 ## Node ports to expose ## @param cas.service.nodePorts.http Node port for HTTP @@ -1251,15 +1253,21 @@ cas: ## annotations: {} ## @extra cas.ingress.grpc gRPC routes of the shared CAS endpoint. - ## The chart renders a second Ingress on the same hostname with its own - ## upstream pool, so native gRPC traffic keeps the HTTP/2 (grpc_pass) - ## transport while the HTTP routes keep HTTP/1.1. This is the - ## separate-upstream-pool contract of spec D-014/D-018. + ## The second Ingress uses a dedicated Service port alias on the same + ## listener. ingress-nginx dials its ClusterIP for gRPC and pod endpoints + ## for HTTP, isolating connection reuse by backend identity and address. + ## Native gRPC keeps HTTP/2 while the HTTP routes keep HTTP/1.1 (D-018). grpc: ## @param cas.ingress.grpc.paths RPC path prefixes served by the gRPC upstream pool. paths: - - /cas.v1.ByteStream/ + - /google.bytestream.ByteStream/ - /cas.v1.ResourceService/ + - /cas.v1.StatusService/ + - /grpc.health.v1.Health/ + - /grpc.reflection.v1.ServerReflection/ + - /grpc.reflection.v1alpha.ServerReflection/ + - /kratos.api.Metadata/ + - /grpc.channelz.v1.Channelz/ ## @extra cas.ingress.grpc.annotations Annotations for the gRPC-path Ingress. ## Use this to override the gRPC upstream settings, for example ## `backend-protocol: GRPCS` when TLS runs to the pod. Do not set these @@ -1274,6 +1282,10 @@ cas: ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/backend-protocol ## Tell nginx Ingress Controller to expect gRPC traffic on these paths nginx.ingress.kubernetes.io/backend-protocol: "GRPC" + ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/service-upstream + # Dial the ClusterIP for gRPC; HTTP uses pod endpoints. ingress-nginx + # keys keepalive connections by upstream address, not Ingress resource. + nginx.ingress.kubernetes.io/service-upstream: "true" ## @skip cas.ingress.grpc.defaultAnnotations.nginx.org/grpc-services nginx.org/grpc-services: '{{ include "chainloop.cas.fullname" . }}' ## @skip cas.ingress.grpc.defaultAnnotations.nginx.org/http2 diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 6a3bfb0ad..49522369f 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -71,7 +71,7 @@ Superseded by R-009 for the CAS. The control-plane scope moved to [the replaceme ### R-009: CAS shared endpoint protocol scope Supersedes R-001 for the CAS. On its single port, the CAS MUST serve native gRPC over HTTP/2 and plain HTTP over HTTP/1.1, through cmux connection classification with the native gRPC transport (D-017). Plain HTTP/2 other than native gRPC is not supported on the CAS shared port: the gRPC matcher's SETTINGS injection breaks such a connection once it falls through to the HTTP server. Plain HTTP/2 and mixed connections remain control-plane scope under the replacement spec. - Done when: focused tests cover native gRPC over h2c and TLS, HTTP/1.1 downloads, the status endpoints, and the CORS preflight through the shared port. -- Done when: the chart routes the gRPC paths and the HTTP paths into separate upstream pools, with HTTP/1.1 for the HTTP routes (D-018). +- Done when: the chart routes the gRPC paths and the HTTP paths into separate upstream pools, with HTTP/1.1 for the HTTP routes (D-018/D-019). - Done when: browser downloads through a configured ingress work, and the documentation states that a browser downloading directly from a TLS-configured shared port is not a supported deployment. ### R-005: Existing configuration keeps working @@ -296,14 +296,30 @@ The CAS shared endpoint is `server.http.addr`. cmux reads the start of each new - Why not the merged control-plane design, per-request `ServeHTTP` dispatch: the [PR #3663 1 GiB comparison](https://github.com/chainloop-dev/chainloop/pull/3663) measured about 1.25 GiB of combined client/server sampled peak live heap for adapter uploads, against about 28 MiB for the native transport, and download throughput 11.8% lower, already past the R-003 limit. grpc-go's handler transport reads the HTTP body in a separate goroutine into an unbounded receive buffer, so read-ahead is not bounded by the application's pace. CAS moves gigabyte streams; the control plane does not. The two services keep different transport designs on purpose. - Why D-016 is superseded: D-016 assumed a classifier this project owns, which records the SETTINGS ACK offset in its own replay and removes that frame before the HTTP server reads. The cmux library owns the sniffing buffer, so that removal does not exist. Two reproductions on the CAS branch fix the contract instead. First, without SETTINGS injection, direct grpc-go clients deadlock: they send their preface and SETTINGS and wait for the server's SETTINGS before their first HEADERS frame, while the matcher waits for headers to sniff. The gRPC matcher must use cmux's SETTINGS-writing matcher. Second, that matcher writes a SETTINGS frame on every HTTP/2 connection, matched or not, because every HTTP/2 client sends SETTINGS with its preface. A plain HTTP/2 client that falls through to the HTTP server acknowledges the injected SETTINGS frame, and net/http rejects the unsolicited acknowledgement (GOAWAY, PROTOCOL_ERROR). - Contract: once the matcher has written to a connection, the connection belongs to the native gRPC server. The HTTP side of the shared endpoint supports HTTP/1.1 only; HTTP/2 reaches the CAS only as native gRPC. Supported proxies keep native gRPC and plain HTTP on separate upstream connections (D-014) and dial HTTP/1.1 for the HTTP routes (D-018). +- The CAS has no per-request gRPC adapter fallback. D-017 supersedes that part of D-014 for CAS. All supported ByteStream transfers use the native transport. +- Classification limits apply before a server owns the connection. They bound time, bytes read from the socket, and concurrent classifications. Shutdown closes pending sockets. These limits end at handoff and do not limit active transfers. - Source: [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) experiment and benchmark, and the local reproductions on its branch. ### D-018: CAS chart: one hostname, path-split upstream pools, HTTP/1.1 for the HTTP routes -The chart exposes one CAS hostname and one public port, and splits the routes by protocol into separate upstream pools: the `/cas.v1.*` paths go to the gRPC pool with an HTTP/2 upstream (nginx `backend-protocol: GRPC` and `grpc-services`; the upload body-size annotations move with them), and `/download/*`, `/statusz`, and `/infoz` go to the HTTP pool, which dials HTTP/1.1, not h2c. One certificate terminates at the edge. +The chart exposes one CAS hostname and one public port, and splits the routes by protocol into separate upstream pools: the Google ByteStream, CAS, health, reflection, metadata, and channelz RPC paths go to the gRPC pool with an HTTP/2 upstream (nginx `backend-protocol: GRPC` and `grpc-services`; the upload body-size annotations move with them), and `/download/*`, `/statusz`, and `/infoz` go to the HTTP pool, which dials HTTP/1.1, not h2c. One certificate terminates at the edge. - Why the path split is exact for CAS: CAS has no gRPC-Web (D-011), so its gRPC paths and its HTTP paths are disjoint. No path can carry both protocols. The control plane could not use this split because native gRPC and gRPC-Web share RPC paths; the [PR #3647 NGINX reproduction](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6272007730) showed the path-only split failing there with 415 on gRPC-Web. CAS does not have that failure mode. - Why HTTP/1.1 upstream on the HTTP pool: D-017. An HTTP/2 upstream for the HTTP routes triggers the matcher's SETTINGS injection, and the client's acknowledgement then breaks the HTTP server connection with GOAWAY. - Controller notes: community nginx merges same-host paths from multiple Ingress resources and sets the backend protocol per resource, which expresses the split. F5 NGINX conflicts on unrelated same-host Ingresses; use its VirtualServer resource. Traefik applies the `service.serversscheme` annotation per Service, so one Service cannot be dialed h2c for the gRPC routes and HTTP/1.1 for the HTTP routes; the Traefik split needs a second Service for the HTTP routes or a controller-specific resource. The browser download endpoint stays on the HTTP routes: it is published API ([docs](https://docs.chainloop.dev/api-reference/downloadservice/download-artifacts-from-cas)), and CAS does not drop it. -- Source: chart changes on the [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) branch. +- Source: chart changes on the [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) branch. Connection reuse and certificate ownership are specified in D-019. + +### D-019: CAS ingress connection reuse and certificate ownership +Supplements D-018. Ingress-nginx shares a global keepalive cache. Separate Ingress resources alone do not isolate their connections. + +The CAS Service has an HTTP port and a gRPC port alias. Both target the same application listener. The HTTP routes use pod endpoints. The gRPC routes use the Service ClusterIP and the alias. Each protocol gets a different backend identity and upstream address. This keeps one Service, one hostname, and one application listener (R-008). + +The Service port numbers MUST differ. At least one upstream MUST use the ClusterIP. The chart rejects configurations that use pod endpoints for both protocols, or a headless Service. + +Both Ingress resources reference the same TLS Secret when shared TLS is enabled. Only one requests the certificate. The HTTP Ingress owns that request when its annotations configure a certificate issuer. Otherwise, the gRPC Ingress can request it. An existing TLS Secret needs no issuer annotation. + +- Why: a single-worker ingress-nginx reproduction returned 502 for gRPC after an HTTP request warmed the shared connection cache. Distinct backend identities and addresses passed the same sequence with authenticated ByteStream and browser downloads. +- Compatibility: the legacy CAS listener, Service, Ingress, and their settings retain their existing behavior. The control-plane connection continues to use that legacy endpoint. +- Evidence limit: the local controller check covers ingress-nginx. F5 and Traefik need separate qualification. The full R-002 and original R-003 evidence remains outstanding. +- Source: [PR #3663 implementation review](https://github.com/chainloop-dev/chainloop/pull/3663#issuecomment-6276983988). ## Open Questions From c5616a65294aefbe76e79d52620a167b8ed54124 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Mon, 19 Oct 2026 23:26:40 +0200 Subject: [PATCH 04/19] fix(cas): tidy direct test dependency Clarify that protocol matching uses stock cmux matchers. Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/internal/server/classification.go | 1 + go.mod | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/app/artifact-cas/internal/server/classification.go b/app/artifact-cas/internal/server/classification.go index 39e5660dd..9a9abd251 100644 --- a/app/artifact-cas/internal/server/classification.go +++ b/app/artifact-cas/internal/server/classification.go @@ -28,6 +28,7 @@ const ( ) // classificationListener owns sockets only until cmux hands them to a server. +// It never inspects protocol bytes; all matching uses cmux's stock matchers. // cmux clears the read deadline at that boundary (SetReadTimeout must be set). // The timer also bounds TLS handshakes and SETTINGS writes, and Close releases // pending sockets without interrupting streams already owned by HTTP or gRPC. diff --git a/go.mod b/go.mod index e5670a78d..05bc9c0e1 100644 --- a/go.mod +++ b/go.mod @@ -115,6 +115,7 @@ require ( go.uber.org/zap v1.28.0 golang.org/x/crypto v0.58.0 golang.org/x/exp v0.0.0-20260908205506-85c1c2202aba + golang.org/x/net v0.61.0 golang.org/x/oauth2 v0.37.0 golang.org/x/sync v0.24.0 golang.org/x/term v0.47.0 @@ -460,7 +461,6 @@ require ( go4.org v0.0.0-20260112195520-a5071408f32f // indirect goa.design/goa/v3 v3.32.0 // indirect golang.org/x/mod v0.41.0 // indirect - golang.org/x/net v0.61.0 // indirect golang.org/x/sys v0.49.0 // indirect golang.org/x/time v0.16.0 // indirect golang.org/x/tools v0.51.0 // indirect From f671b4c69a8bfead30219c8912cd434987794142 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 00:09:59 +0200 Subject: [PATCH 05/19] feat(cas): per-request dispatch with a reverse proxy to a private native server Replace cmux connection classification with the control-plane request dispatch: one net/http listener owns server.http.addr (HTTP/1.1, HTTP/2, optional TLS) and routes each request. Native gRPC goes through an in-process httputil.ReverseProxy to a private native gRPC server bound to an ephemeral loopback address; everything else uses the existing HTTP routes. ByteStream keeps the native transport: the proxy streams the request body into the private connection, whose HTTP/2 flow-control window paces the proxy's reads, so no unbounded read-ahead exists in either direction (the ServeHTTP adapter measured ~1.25 GiB of live heap on a 1 GiB upload). The private server runs the same services and interceptors as the legacy listener and is independent of it; the legacy flag, address, and TLS settings are unchanged, and the shared endpoint works with the legacy listener disabled. Mixed native-gRPC and plain-HTTP requests now share one upstream connection in any order, as on the control plane. The cmux dependency and its classification bounds are removed. Chart: one Service port and one catch-all Ingress on the same hostname as the control plane - backend-protocol GRPC (HTTP/2 upstream), grpc-services, and the upload body limits; Traefik h2c Service annotation. The RPC-path second Ingress, the grpc-shared Service port alias, and the pool-isolation validations are removed. Measured (1 GiB, three runs per operation, disk-backed staging): shared median upload 212.89 MB/s vs legacy 233.92 (-9.0%), download 177.11 vs 192.82 (-8.1%), peak live heap ~32-43 MiB vs ~29-32 MiB - within the R-003 10% shape with bounded memory, thin margin. Spec: D-020 supersedes D-017, D-018, and D-019; R-010 supersedes R-009. Full R-002 alignment, the original R-003 qualification, and F5/Traefik controller qualification remain open. Refs #3628. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/README.md | 30 ++- app/artifact-cas/configs/config.devel.yaml | 5 +- app/artifact-cas/configs/samples/config.yaml | 4 +- .../experiments/shared-endpoint-benchmark.txt | 56 +++--- .../internal/server/classification.go | 130 ------------- .../internal/server/classification_test.go | 154 --------------- app/artifact-cas/internal/server/grpc.go | 40 ++-- app/artifact-cas/internal/server/http.go | 174 ++++++++++------- app/artifact-cas/internal/server/http_test.go | 95 +++------ .../internal/server/ingress_test.go | 9 +- deployment/chainloop/README.md | 9 +- deployment/chainloop/chart_test.go | 182 ++++-------------- .../templates/cas/ingress-grpc-shared.yaml | 83 -------- .../chainloop/templates/cas/service-http.yaml | 8 - deployment/chainloop/values.yaml | 84 +++----- docs/specs/issue-3628-single-endpoint.md | 74 ++++--- go.mod | 3 +- go.sum | 3 - 18 files changed, 323 insertions(+), 820 deletions(-) delete mode 100644 app/artifact-cas/internal/server/classification.go delete mode 100644 app/artifact-cas/internal/server/classification_test.go delete mode 100644 deployment/chainloop/templates/cas/ingress-grpc-shared.yaml diff --git a/app/artifact-cas/README.md b/app/artifact-cas/README.md index ca21b61b7..ff1c4e953 100644 --- a/app/artifact-cas/README.md +++ b/app/artifact-cas/README.md @@ -68,13 +68,11 @@ go run main.go --insecure artifact download -d [sha256:...] ``` -### Shared endpoint experiment +### Shared endpoint -The CAS HTTP port, `8001`, is the shared endpoint. cmux reads the first request of each new connection and hands the whole connection to one of the two existing transports: HTTP/2 connections whose first request declares a native gRPC content type go to the native gRPC transport (`grpc.Server.Serve`), everything else goes to the existing HTTP server. ByteStream therefore keeps the grpc-go native transport; the earlier per-request `grpc.Server.ServeHTTP` experiment on this branch was rejected because its uploads peaked at ~1.25 GiB of live heap for a 1 GiB artifact (see the results history below and the PR #3663 discussion). CAS does not add gRPC-Web or Connect. +The CAS HTTP port, `8001`, is the shared endpoint, mirroring the control plane (spec D-020). One standard HTTP listener owns every connection and dispatches each request: native gRPC (HTTP/2 + an `application/grpc*` content type) goes through an in-process `httputil.ReverseProxy` to a private native gRPC server bound to an ephemeral loopback address, and everything else uses the existing HTTP routes. ByteStream keeps the grpc-go native transport: the proxy streams the request body into the private connection, whose HTTP/2 flow-control window paces the proxy's reads, so no unbounded read-ahead exists in either direction. Earlier designs on this branch measured why: the per-request `grpc.Server.ServeHTTP` adapter peaked at ~1.25 GiB of live heap for a 1 GiB upload, and connection classification (cmux) could not serve the all-HTTP/2 upstream that a single-location ingress produces. Mixed native-gRPC and plain-HTTP requests share one upstream connection in any order, as on the control plane. CAS does not add gRPC-Web or Connect. -The gRPC matcher is the SETTINGS-writing one: grpc-go clients send their preface and SETTINGS and then wait for the server's SETTINGS before their first request, so a silent matcher deadlocks them. Once the matcher has written, the connection belongs to the native gRPC server. The HTTP side of the shared endpoint therefore supports HTTP/1.1 only: a plain HTTP/2 client that falls through to the HTTP server acknowledges the injected SETTINGS frame, which `net/http` rejects (`GOAWAY`, `PROTOCOL_ERROR`). Mixed native-gRPC and plain-HTTP requests on one upstream HTTP/2 connection are not supported; supported proxies keep the two kinds of traffic on separate upstream pools, which the chart configures (spec D-017/D-018, recorded in [issue-3628-single-endpoint](../../docs/specs/issue-3628-single-endpoint.md)). - -The native gRPC listener on `9001` remains enabled by default, with its existing independent TLS configuration. The same ByteStream and resource services, JWT checks, role checks, validators, and metrics interceptors serve both listeners. `server.http.timeout` continues to apply to HTTP routes; ByteStream requests bypass the HTTP route middleware. HTTP drains before the gRPC transport stops. +The private native server runs the same services and interceptors as the legacy listener and is independent of it. The native gRPC listener on `9001` remains enabled by default, with its existing independent TLS configuration and flag; the shared endpoint works with it disabled. `server.http.timeout` continues to apply to HTTP routes; proxied gRPC requests bypass the HTTP route middleware. Shutdown drains the HTTP server first — including every proxied gRPC stream — then stops the private native server. To test with your existing development infrastructure: @@ -90,15 +88,13 @@ go run ./app/cli --insecure --artifact-cas localhost:8001 artifact download -d s The development CAS public key and credentials service must match the control plane that issues these temporary CAS tokens. If another CAS already occupies these ports, start this binary with a copy of your CAS configuration and unused addresses. Do not replace the running service or edit the CLI config just to run a comparison. -Change only the per-command override to `localhost:9001` to compare with the native listener. Once all direct clients and the control-plane CAS connection have been moved, `server.legacy_grpc_listener: false` disables the native listener. Leave it enabled during evaluation. Optional `server.http.tls_config` configures TLS for the shared endpoint; both certificate and private key must be set. It does not inherit `server.grpc.tls_config`. Note that a browser downloading directly from a TLS-configured shared port is not supported: the browser negotiates HTTP/2 through ALPN and the connection reaches the HTTP server with an injected SETTINGS frame. Serve browser downloads through the ingress. - -The chart's `cas.ingress` sends HTTP routes (`/download`, `/statusz`, `/infoz`) over HTTP/1.1. A second Ingress sends Google ByteStream, CAS, health, reflection, metadata, and channelz RPCs over native gRPC. The same Service has two port names: `http` (80) and `grpc-shared` (81, configurable with `cas.service.ports.grpcShared`). Both target the existing shared container port. Ingress-nginx uses pod endpoints for HTTP and the Service ClusterIP for gRPC. Different backend identities and upstream addresses prevent its global keepalive cache from reusing a socket across protocols. Separate Ingress objects alone do not provide that isolation. +Change only the per-command override to `localhost:9001` to compare with the native listener. Once all direct clients and the control-plane CAS connection have been moved, `server.legacy_grpc_listener: false` disables the native listener; the shared endpoint's private native server keeps running on its loopback address either way. Leave the legacy listener enabled during evaluation. Optional `server.http.tls_config` configures TLS for the shared endpoint; both certificate and private key must be set. It does not inherit `server.grpc.tls_config`. -Keep the Service port numbers different and at least one upstream on the ClusterIP. The chart rejects unsafe pool settings and headless Services for this arrangement. The gRPC Ingress keeps the legacy ingress's `100m` body limit and `3M` client buffer default. Override the body limit to `0` or a suitable larger value for a 1 GiB ingress experiment. Both Ingress objects reference one TLS Secret. Only one requests a certificate from cert-manager, preferring the HTTP Ingress when it has issuer annotations. `cas.ingress.tls=true` also supports a pre-created Secret without issuer annotations. +The chart's `cas.ingress` mirrors the control plane: one catch-all route on one hostname, one Service port, one certificate, and an HTTP/2 (gRPC) upstream — so native gRPC, browser downloads, and status probes all arrive on the shared listener, including mixed requests on one upstream connection. The ingress keeps the legacy ingress's `100m` body limit and `3M` client buffer default. Override the body limit to `0` or a suitable larger value for a 1 GiB ingress experiment. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS Service keep their existing settings. F5 same-host routing and Traefik's per-Service scheme still need qualification. -Before cmux hands a connection to a server, classification is limited to 10 seconds, 64 KiB read from the socket (including TLS), and 128 concurrent pending connections. Shutdown closes pending sockets. These bounds end at handoff and do not constrain active ByteStream transfers. The HTTP server then applies its 60-second header timeout. CAS has no per-request gRPC adapter fallback. Metrics remain plaintext on their separate listener; `server.http_metrics.tls_config` is rejected. +The HTTP server applies its 60-second header timeout; proxied gRPC streams carry no such deadline. Metrics remain plaintext on their separate listener; `server.http_metrics.tls_config` is rejected. Run the isolated functional checks without Vault or an object store: @@ -135,18 +131,18 @@ TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/se #### Observed local results -Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, cmux 0.1.5, Linux amd64, disk-backed btrfs staging. These medians come from the [raw benchmark output](experiments/shared-endpoint-benchmark.txt). +Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, Linux amd64, disk-backed btrfs staging. These medians come from the [raw benchmark output](experiments/shared-endpoint-benchmark.txt). | Listener | Operation | Throughput (MB/s) | Sampled peak live heap (MiB) | | --- | --- | ---: | ---: | -| Native gRPC (`:9001`) | Upload | 225.34 | 28.91 | -| Shared endpoint (`:8001`, cmux) | Upload | 232.85 | 27.46 | -| Native gRPC (`:9001`) | Download | 198.05 | 28.90 | -| Shared endpoint (`:8001`, cmux) | Download | 189.63 | 31.92 | +| Native gRPC (`:9001`) | Upload | 233.92 | 29.13 | +| Shared endpoint (`:8001`, proxy) | Upload | 212.89 | 32.45 | +| Native gRPC (`:9001`) | Download | 192.82 | 30.82 | +| Shared endpoint (`:8001`, proxy) | Download | 177.11 | 42.78 | -Both listeners use the native grpc-go transport, and the numbers match: median upload throughput on the shared endpoint was 3.3% higher and median download throughput 4.3% lower than the legacy listener, with sampled peak live heap equivalent (~28-32 MiB) on both. This satisfies the R-003 transport-comparison shape for the shared endpoint — within 10% with bounded memory — in this synthetic local setup. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production. +ByteStream keeps the native transport behind the proxy: median upload throughput on the shared endpoint was 9.0% lower and median download throughput 8.1% lower than the legacy listener — within the R-003 10% shape, with a thin margin — and sampled peak live heap stayed bounded (~32-43 MiB against ~29-32 MiB native), not growing with artifact size. The extra hop costs the double HTTP/2 framing and one copy. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production; the release and object-store qualification remains open. -For contrast, the earlier per-request `ServeHTTP` experiment on this branch measured 232.00 MB/s upload and 173.83 MB/s download against its contemporaneous native baseline, with an upload peak of ~1278 MiB: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90), so read-ahead is not bounded by the application's pace. That memory regression rejected the adapter for CAS uploads (the control plane keeps the adapter because its requests are small and unary) and motivated the cmux design recorded in D-017. +Earlier designs on this branch measured the alternatives. The per-request `grpc.Server.ServeHTTP` adapter peaked at ~1278 MiB of live heap on a 1 GiB upload: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90), so read-ahead is not bounded by the application's pace. Connection classification through cmux matched native throughput (upload +3.3%, download −4.3%) but could not serve the all-HTTP/2 upstream of a single-location ingress: a silent matcher deadlocks grpc-go clients, the SETTINGS-writing matcher poisons plain-HTTP/2 connections (GOAWAY, PROTOCOL_ERROR), and pooled mixed connections fail. Both results are preserved in the PR #3663 history. ### Run tests diff --git a/app/artifact-cas/configs/config.devel.yaml b/app/artifact-cas/configs/config.devel.yaml index f30fdd315..61364d7de 100644 --- a/app/artifact-cas/configs/config.devel.yaml +++ b/app/artifact-cas/configs/config.devel.yaml @@ -6,8 +6,9 @@ server: # Keep 9001 available to compare native gRPC with the shared 8001 endpoint. legacy_grpc_listener: true http: - # Shared endpoint: HTTP/1.1 and native gRPC. cmux classifies each - # connection by its first request; HTTP/2 reaches it only as native gRPC. + # Shared endpoint: HTTP/1.1, HTTP/2, and native gRPC. Each request is + # dispatched; native gRPC reaches the native transport through an + # in-process reverse proxy (spec D-020). addr: 0.0.0.0:8001 # Timeouts for http downloads # grpc downloads/uploads don't require this because they don't have timeouts diff --git a/app/artifact-cas/configs/samples/config.yaml b/app/artifact-cas/configs/samples/config.yaml index 5a26fe573..2af490377 100644 --- a/app/artifact-cas/configs/samples/config.yaml +++ b/app/artifact-cas/configs/samples/config.yaml @@ -8,7 +8,9 @@ server: http: addr: 0.0.0.0:8001 timeout: 300s - # Shared endpoint: HTTP/1.1 and native gRPC (cmux). HTTP/2 is native gRPC only. + # Shared endpoint: HTTP/1.1, HTTP/2, and native gRPC, dispatched per + # request; native gRPC reaches the native transport through an in-process + # reverse proxy (spec D-020). # Optional TLS for both transports on port 8001. Independent of grpc.tls_config. # tls_config: # certificate: "./configs/tls/shared.crt" diff --git a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt index ef42f9cce..1432bb86b 100644 --- a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt +++ b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt @@ -1,46 +1,44 @@ # Copyright 2026 The Chainloop Authors. # SPDX-License-Identifier: Apache-2.0 -# Review-fix rerun: cmux shared endpoint vs the legacy native listener. -# Both use grpc-go's native transport. The shared listener now has bounded -# classification and closes pending sockets during shutdown. Earlier adapter -# and cmux results remain in this file's git history and PR #3663 discussion. +# Shared endpoint (per-request dispatch + in-process httputil.ReverseProxy to +# the private native gRPC server, spec D-020) vs the legacy native listener. +# Both serve ByteStream on the native transport; the shared endpoint adds one +# loopback proxy hop. Earlier runs on this branch measured the +# grpc.Server.ServeHTTP adapter (upload peak ~1278 MiB, download -11.8%) and +# cmux connection classification (native parity but no mixed-connection or +# all-HTTP/2 support); those results are preserved in the PR #3663 history. go version go1.27.2 linux/amd64 -grpc-go: v1.84.0; Kratos: v2.9.2; cmux: v0.1.5 +grpc-go: v1.84.0; Kratos: v2.9.2 Temporary filesystem: btrfs (not tmpfs) -Commands: - mkdir -p "$HOME/.cache/chainloop-cas-bench" - TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/server -run '^$' -bench '^BenchmarkCASByteStreamTransport$' -benchtime=1x -count=3 -timeout=10m +Command: TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/server -run ^$ -bench ^BenchmarkCASByteStreamTransport$ -benchtime=1x -count=3 -timeout=10m Artifact: 1 GiB; plaintext localhost TCP; real CAS services/auth/staging; disk-backed test backend; client and server in one process. -Limitations: synthetic zero content, local disk, combined sampled client/server live heap, no representative object store, concurrency, or TLS performance qualification. - -INFO msg=[gRPC] server listening on: 127.0.0.1:34425 -INFO msg=[HTTP] server listening on: 127.0.0.1:38673 +INFO msg=[gRPC] server listening on: 127.0.0.1:41457 +INFO msg=[HTTP] server listening on: 127.0.0.1:33971 +INFO msg=[gRPC] server listening on: 127.0.0.1:42341 goos: linux goarch: amd64 pkg: github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server cpu: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4765003662 ns/op 225.34 MB/s 29.50 peak-heap-MiB 1483628368 B/op 3160122 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 6045761098 ns/op 177.60 MB/s 28.33 peak-heap-MiB 1505093640 B/op 3153996 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4558956236 ns/op 235.52 MB/s 28.91 peak-heap-MiB 1472837272 B/op 3153869 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5701383303 ns/op 188.33 MB/s 28.90 peak-heap-MiB 1345298136 B/op 87053 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5418492831 ns/op 198.16 MB/s 28.93 peak-heap-MiB 1340639536 B/op 86919 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5421594921 ns/op 198.05 MB/s 28.67 peak-heap-MiB 1342393032 B/op 86665 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4658293656 ns/op 230.50 MB/s 28.28 peak-heap-MiB 1484342216 B/op 3226127 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4590142558 ns/op 233.92 MB/s 29.13 peak-heap-MiB 1456925088 B/op 3075050 allocs/op +BenchmarkCASByteStreamTransport/legacy/upload-8 1 4543473383 ns/op 236.33 MB/s 28.98 peak-heap-MiB 1440917520 B/op 3057361 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5993847959 ns/op 179.14 MB/s 31.45 peak-heap-MiB 1323673128 B/op 89962 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5568532492 ns/op 192.82 MB/s 30.82 peak-heap-MiB 1326466392 B/op 83548 allocs/op +BenchmarkCASByteStreamTransport/legacy/download-8 1 5525732102 ns/op 194.32 MB/s 32.18 peak-heap-MiB 1322231624 B/op 83814 allocs/op INFO msg=[gRPC] server stopping INFO msg=[HTTP] server stopping -INFO msg=[gRPC] server stopping -INFO msg=[gRPC] server listening on: 127.0.0.1:38603 -INFO msg=[HTTP] server listening on: 127.0.0.1:45889 -BenchmarkCASByteStreamTransport/shared/upload-8 1 4716482049 ns/op 227.66 MB/s 27.46 peak-heap-MiB 1418368840 B/op 3224685 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 4605922407 ns/op 233.12 MB/s 28.47 peak-heap-MiB 1459735392 B/op 3224827 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 4611224941 ns/op 232.85 MB/s 26.29 peak-heap-MiB 1440540392 B/op 3224912 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5862496162 ns/op 183.15 MB/s 32.00 peak-heap-MiB 1377047368 B/op 94038 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5623257158 ns/op 190.95 MB/s 31.34 peak-heap-MiB 1377929448 B/op 93669 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5662234715 ns/op 189.63 MB/s 31.92 peak-heap-MiB 1377032552 B/op 90475 allocs/op +INFO msg=[gRPC] server listening on: 127.0.0.1:42341 +INFO msg=[HTTP] server listening on: 127.0.0.1:33971 +BenchmarkCASByteStreamTransport/shared/upload-8 1 5043563204 ns/op 212.89 MB/s 32.83 peak-heap-MiB 1457869152 B/op 3573905 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 5009169968 ns/op 214.36 MB/s 32.45 peak-heap-MiB 1425339960 B/op 3557581 allocs/op +BenchmarkCASByteStreamTransport/shared/upload-8 1 5059348118 ns/op 212.23 MB/s 32.23 peak-heap-MiB 1459646440 B/op 3563838 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 6655010308 ns/op 161.34 MB/s 40.49 peak-heap-MiB 1299487624 B/op 522911 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 6062698816 ns/op 177.11 MB/s 42.78 peak-heap-MiB 1286521128 B/op 514791 allocs/op +BenchmarkCASByteStreamTransport/shared/download-8 1 5970943858 ns/op 179.83 MB/s 42.87 peak-heap-MiB 1265871832 B/op 511971 allocs/op INFO msg=[gRPC] server stopping INFO msg=[HTTP] server stopping -INFO msg=[gRPC] server stopping PASS -ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 65.771s +ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 66.534s diff --git a/app/artifact-cas/internal/server/classification.go b/app/artifact-cas/internal/server/classification.go deleted file mode 100644 index 9a9abd251..000000000 --- a/app/artifact-cas/internal/server/classification.go +++ /dev/null @@ -1,130 +0,0 @@ -// Copyright 2026 The Chainloop Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package server - -import ( - "errors" - "net" - "sync" - "time" -) - -const ( - classificationTimeout = 10 * time.Second - maxClassifyingConnections = 128 - maxClassificationBytes = 64 << 10 -) - -// classificationListener owns sockets only until cmux hands them to a server. -// It never inspects protocol bytes; all matching uses cmux's stock matchers. -// cmux clears the read deadline at that boundary (SetReadTimeout must be set). -// The timer also bounds TLS handshakes and SETTINGS writes, and Close releases -// pending sockets without interrupting streams already owned by HTTP or gRPC. -type classificationListener struct { - net.Listener - mu sync.Mutex - pending map[*classificationConn]*time.Timer - closed bool - timeout time.Duration -} - -func (l *classificationListener) Accept() (net.Conn, error) { - for { - conn, err := l.Listener.Accept() - if err != nil { - return nil, err - } - l.mu.Lock() - if l.closed { - l.mu.Unlock() - _ = conn.Close() - return nil, net.ErrClosed - } - if len(l.pending) >= maxClassifyingConnections { - l.mu.Unlock() - _ = conn.Close() - continue - } - c := &classificationConn{Conn: conn, owner: l, remaining: maxClassificationBytes} - l.pending[c] = time.AfterFunc(l.timeout, func() { - l.mu.Lock() - defer l.mu.Unlock() - if _, pending := l.pending[c]; pending { - delete(l.pending, c) - _ = c.Conn.Close() - } - }) - l.mu.Unlock() - return c, nil - } -} - -func (l *classificationListener) Close() error { - l.mu.Lock() - l.closed = true - for conn, timer := range l.pending { - timer.Stop() - _ = conn.Conn.Close() - delete(l.pending, conn) - } - l.mu.Unlock() - return l.Listener.Close() -} - -type classificationConn struct { - net.Conn - owner *classificationListener - remaining int -} - -func (c *classificationConn) Read(p []byte) (int, error) { - // Only the cmux goroutine reads or changes remaining until handoff. - if c.remaining == 0 { - _ = c.Close() - return 0, errors.New("CAS connection classification exceeded its byte limit") - } - if c.remaining > 0 && len(p) > c.remaining { - p = p[:c.remaining] - } - n, err := c.Conn.Read(p) - if c.remaining > 0 { - c.remaining -= n - } - return n, err -} - -func (c *classificationConn) SetReadDeadline(deadline time.Time) error { - if !deadline.IsZero() { - return c.Conn.SetReadDeadline(deadline) - } - c.owner.mu.Lock() - defer c.owner.mu.Unlock() - if timer, pending := c.owner.pending[c]; pending { - timer.Stop() - delete(c.owner.pending, c) - c.remaining = -1 - } - return c.Conn.SetReadDeadline(deadline) -} - -func (c *classificationConn) Close() error { - c.owner.mu.Lock() - if timer, pending := c.owner.pending[c]; pending { - timer.Stop() - delete(c.owner.pending, c) - } - c.owner.mu.Unlock() - return c.Conn.Close() -} diff --git a/app/artifact-cas/internal/server/classification_test.go b/app/artifact-cas/internal/server/classification_test.go deleted file mode 100644 index d0d8ab8ba..000000000 --- a/app/artifact-cas/internal/server/classification_test.go +++ /dev/null @@ -1,154 +0,0 @@ -// Copyright 2026 The Chainloop Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package server - -import ( - "context" - "errors" - "io" - "net" - "testing" - "time" - - v1 "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" - "github.com/soheilhy/cmux" - "github.com/stretchr/testify/require" - "golang.org/x/net/http2" -) - -func TestCASClassificationTimeout(t *testing.T) { - for _, prefix := range []string{"", "PRI ", "GET / HTTP/1.1\r\n"} { - t.Run(prefix, func(t *testing.T) { - opts := casTestOpts(t) - opts.ServerConfig.LegacyGrpcListener = new(false) - servers, err := NewGRPCServers(opts) - require.NoError(t, err) - api, err := NewHTTPServer(opts, servers) - require.NoError(t, err) - // Exercise the lifecycle timer, including a client that never reads - // server SETTINGS, independently of cmux's read deadline. - api.lis.(*classificationListener).timeout = 50 * time.Millisecond - api.httpSrv.ReadHeaderTimeout = 50 * time.Millisecond - endpoint, err := api.Endpoint() - require.NoError(t, err) - startCASTestTransport(t, api) - conn, err := net.Dial("tcp", endpoint.Host) - require.NoError(t, err) - defer conn.Close() - _, err = io.WriteString(conn, prefix) - require.NoError(t, err) - require.NoError(t, conn.SetReadDeadline(time.Now().Add(time.Second))) - _, err = io.Copy(io.Discard, conn) - require.NoError(t, err, "classification must close before the client deadline") - }) - } -} - -func TestCASClassificationBudget(t *testing.T) { - opts := casTestOpts(t) - opts.ServerConfig.LegacyGrpcListener = new(false) - servers, err := NewGRPCServers(opts) - require.NoError(t, err) - api, err := NewHTTPServer(opts, servers) - require.NoError(t, err) - endpoint, err := api.Endpoint() - require.NoError(t, err) - startCASTestTransport(t, api) - conn, err := net.Dial("tcp", endpoint.Host) - require.NoError(t, err) - defer conn.Close() - require.NoError(t, conn.SetDeadline(time.Now().Add(time.Second))) - // A peer may send valid HTTP/2 frames forever without the HEADERS cmux - // needs. The wire-byte budget must stop it before the time limit. - _, err = io.WriteString(conn, http2.ClientPreface) - require.NoError(t, err) - framer := http2.NewFramer(conn, conn) - require.NoError(t, framer.WriteSettings()) - frame, err := framer.ReadFrame() - require.NoError(t, err) - require.IsType(t, &http2.SettingsFrame{}, frame) - // Padded DATA are irrelevant to the header matcher but consume its budget. - for range maxClassificationBytes / (16 << 10) { - if err = framer.WriteData(1, false, make([]byte, 16<<10)); err != nil { - break - } - } - _, err = io.Copy(io.Discard, conn) - var netErr net.Error - if errors.As(err, &netErr) { - require.False(t, netErr.Timeout(), "classification byte limit was not enforced") - } -} - -func TestCASClassificationConcurrencyLimit(t *testing.T) { - root, err := net.Listen("tcp", casAnyAddr) - require.NoError(t, err) - listener := &classificationListener{Listener: root, pending: make(map[*classificationConn]*time.Timer), timeout: time.Minute} - defer listener.Close() - mux := cmux.New(listener) - mux.SetReadTimeout(time.Minute) - mux.Match(cmux.HTTP1Fast()) - done := make(chan error, 1) - go func() { done <- mux.Serve() }() - for range maxClassifyingConnections { - conn, err := net.Dial("tcp", root.Addr().String()) - require.NoError(t, err) - defer conn.Close() - } - require.Eventually(t, func() bool { - listener.mu.Lock() - defer listener.mu.Unlock() - return len(listener.pending) == maxClassifyingConnections - }, time.Second, time.Millisecond) - rejected, err := net.Dial("tcp", root.Addr().String()) - require.NoError(t, err) - defer rejected.Close() - require.NoError(t, rejected.SetReadDeadline(time.Now().Add(time.Second))) - var data [1]byte - _, err = rejected.Read(data[:]) - require.ErrorIs(t, err, io.EOF) - require.NoError(t, listener.Close()) - mux.Close() - select { - case err := <-done: - require.ErrorIs(t, err, net.ErrClosed) - case <-time.After(time.Second): - require.FailNow(t, "classifier workers did not exit") - } -} - -func TestCASClassificationDeadlineCleared(t *testing.T) { - opts := casTestOpts(t) - opts.ServerConfig.LegacyGrpcListener = new(false) - servers, err := NewGRPCServers(opts) - require.NoError(t, err) - api, err := NewHTTPServer(opts, servers) - require.NoError(t, err) - api.lis.(*classificationListener).timeout = 250 * time.Millisecond - api.mux.SetReadTimeout(250 * time.Millisecond) - endpoint, err := api.Endpoint() - require.NoError(t, err) - startCASTestTransport(t, api) - conn := casTestConn(t, endpoint.Host, false) - ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) - defer cancel() - // A gRPC connection must remain usable after its classification deadline. - client := v1.NewStatusServiceClient(conn) - _, err = client.Statusz(ctx, &v1.StatuszRequest{}) - require.NoError(t, err) - time.Sleep(500 * time.Millisecond) - _, err = client.Statusz(ctx, &v1.StatuszRequest{}) - require.NoError(t, err) -} diff --git a/app/artifact-cas/internal/server/grpc.go b/app/artifact-cas/internal/server/grpc.go index d196bd481..e6c82a248 100644 --- a/app/artifact-cas/internal/server/grpc.go +++ b/app/artifact-cas/internal/server/grpc.go @@ -20,7 +20,7 @@ import ( "crypto/ecdsa" "crypto/tls" "fmt" - "net/url" + "net" "os" "regexp" @@ -64,37 +64,43 @@ type Opts struct { Logger log.Logger } -// GRPCServers keeps the shared endpoint's native gRPC server separate from -// the optional legacy listener. Both run the same services and interceptors. -// Handler receives connections from cmux and does not bind its own socket. +// GRPCServers keeps the shared endpoint's private native gRPC server separate +// from the optional legacy listener. Both run the same services and +// interceptors. Private binds an ephemeral loopback listener that the shared +// endpoint's reverse proxy dials; it is independent of the legacy listener's +// address, TLS configuration, and flag, and exists whether or not the legacy +// listener is enabled. type GRPCServers struct { - Handler *grpc.Server - Legacy *grpc.Server + Private *grpc.Server + PrivateAddr string + Legacy *grpc.Server } func legacyGRPCListenerEnabled(c *conf.Server) bool { return c.LegacyGrpcListener == nil || *c.LegacyGrpcListener } -// NewGRPCServers builds both transports with the same services and interceptors. +// NewGRPCServers builds the private native server for the shared endpoint and +// the optional legacy listener, with the same services and interceptors. func NewGRPCServers(opts *Opts) (*GRPCServers, error) { - scheme := "grpc" - if opts.ServerConfig.GetHttp().GetTlsConfig().GetCertificate() != "" { - scheme = "grpcs" + // Bind the private listener here so the shared endpoint's proxy knows its + // address at construction. Loopback-only, ephemeral: never public, never + // configurable, and independent of server.grpc.addr (which may carry TLS). + privateLis, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + return nil, fmt.Errorf("binding the shared endpoint's private gRPC listener: %w", err) } - // Kratos stream interceptors require endpoint metadata even though this - // handler never binds a native socket. - handler, err := newGRPCServer(opts, grpc.Endpoint(&url.URL{ - Scheme: scheme, Host: opts.ServerConfig.GetHttp().GetAddr(), - })) + private, err := newGRPCServer(opts, grpc.Listener(privateLis)) if err != nil { + _ = privateLis.Close() return nil, err } - servers := &GRPCServers{Handler: handler} + servers := &GRPCServers{Private: private, PrivateAddr: privateLis.Addr().String()} if legacyGRPCListenerEnabled(opts.ServerConfig) { servers.Legacy, err = NewGRPCServer(opts) if err != nil { - handler.Server.Stop() + private.Server.Stop() + _ = privateLis.Close() return nil, err } } diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 82af7c09f..57c99d660 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -22,7 +22,9 @@ import ( "fmt" "net" h "net/http" + "net/http/httputil" "net/url" + "strings" "sync" "time" @@ -38,42 +40,37 @@ import ( "github.com/go-kratos/kratos/v2/transport/http" "github.com/golang-jwt/jwt/v5" grpcLib "google.golang.org/grpc" - - "github.com/soheilhy/cmux" ) -// APIServer owns the shared CAS listener on server.http.addr. cmux reads the -// first request of each new connection and hands the whole connection to one -// of the two existing transports: +// APIServer owns the shared CAS listener on server.http.addr. The standard +// HTTP server owns every connection and dispatches each request: // -// native gRPC (HTTP/2 + application/grpc*) -> grpc.Server.Serve (native transport) -// everything else -> the existing Kratos HTTP server +// native gRPC (HTTP/2 + application/grpc*) -> httputil.ReverseProxy -> +// the private native gRPC server (grpc.Server.Serve, native transport) +// everything else -> the existing Kratos HTTP routes // -// ByteStream therefore keeps the grpc-go native transport, so large uploads -// stream with bounded memory exactly as they do on the legacy listener. This -// follows the separate-connection contract in D-017: native gRPC and HTTP -// use separate connections, with HTTP/1.1 on the HTTP side. +// ByteStream keeps the native transport: the proxy streams the request body +// into the private connection, whose HTTP/2 flow-control window paces the +// proxy's reads, and net/http's own window paces the client. No unbounded +// read-ahead exists anywhere on the path, in either direction. Mixed gRPC and +// ordinary HTTP requests may share one upstream connection in any order, as +// they do on the control plane (spec D-020). type APIServer struct { - lis net.Listener - mux cmux.CMux - grpcL net.Listener - httpL net.Listener - grpcSrv *grpc.Server - httpSrv *http.Server - scheme string - workers sync.WaitGroup + lis net.Listener + httpSrv *http.Server + grpcSrv *grpc.Server + grpcAddr string + scheme string + workers sync.WaitGroup } // Start serves the shared endpoint until ctx is cancelled or a transport -// fails. The HTTP server serves the connections cmux matched to it, the -// native gRPC server serves its own matched connections, and cmux keeps -// accepting new ones. Each goroutine reports through errCh; an orderly -// Stop closes the listener and all three return ordinary shutdown errors. +// fails. The HTTP server serves every connection; the private native gRPC +// server serves the connections the proxy opens to it. func (s *APIServer) Start(ctx context.Context) error { - errCh := make(chan error, 3) + errCh := make(chan error, 2) s.workers.Go(func() { errCh <- s.httpSrv.Start(ctx) }) - s.workers.Go(func() { errCh <- s.grpcSrv.Serve(s.grpcL) }) - s.workers.Go(func() { errCh <- s.mux.Serve() }) + s.workers.Go(func() { errCh <- s.grpcSrv.Start(ctx) }) select { case <-ctx.Done(): return nil @@ -85,24 +82,13 @@ func (s *APIServer) Start(ctx context.Context) error { } } -// Stop closes pending classification sockets, then drains both transports. +// Stop drains the HTTP server first — which includes every proxied gRPC +// stream, because those live on HTTP connections — then stops the private +// native gRPC server so its in-flight RPCs finish. func (s *APIServer) Stop(ctx context.Context) error { - // Stop accepting new connections on the shared endpoint. cmux.Close only - // marks the match listeners closed: active connections keep draining. - closeErr := s.lis.Close() - s.mux.Close() httpErr := s.httpSrv.Stop(ctx) - // http.Server.Shutdown closes its listener, which is the cmux mux - // listener wrapping the root listener this Stop already closed. That - // double close surfaces net.ErrClosed; it is not a serving failure. - if httpErr != nil && errors.Is(httpErr, net.ErrClosed) { - httpErr = nil - } grpcErr := s.grpcSrv.Stop(ctx) s.workers.Wait() - if closeErr != nil && !errors.Is(closeErr, net.ErrClosed) { - return errors.Join(closeErr, httpErr, grpcErr) - } return errors.Join(httpErr, grpcErr) } @@ -111,8 +97,7 @@ func (s *APIServer) Endpoint() (*url.URL, error) { return &url.URL{Scheme: s.scheme, Host: s.lis.Addr().String()}, nil } -// NewHTTPServer binds the shared CAS endpoint and classifies each new -// connection by its first request through cmux. +// NewHTTPServer binds the shared CAS endpoint and dispatches every request. func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) { c, authConf := settings.ServerConfig, settings.AuthConfig downloadSvc, providers, logger := settings.DownloadSvc, settings.Providers, settings.Logger @@ -138,35 +123,18 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) if err != nil { return nil, fmt.Errorf("binding the shared CAS endpoint %q: %w", c.Http.GetAddr(), err) } - lis = &classificationListener{Listener: lis, pending: make(map[*classificationConn]*time.Timer), timeout: classificationTimeout} scheme := "http" if tlsConfig != nil { - // Terminate TLS before cmux so one certificate covers both - // transports, with ALPN offering HTTP/2 and HTTP/1.1. + // One certificate covers both transports, with ALPN offering + // HTTP/2 and HTTP/1.1. lis = tls.NewListener(lis, tlsConfig) scheme = "https" } - mux := cmux.New(lis) - mux.SetReadTimeout(classificationTimeout) - // Native gRPC: match HTTP/2 connections whose first request declares a - // native gRPC content type (application/grpc, application/grpc+proto, - // ...). The SETTINGS-writing writer unblocks gRPC clients, which send - // their preface and SETTINGS and then wait for the server's SETTINGS - // before their first HEADERS frame (Argo CD ships this matcher for the - // same reason). Once it writes, the connection belongs to the native - // gRPC server: cmux documents that it cannot be handed to the HTTP - // server afterwards. CAS has no gRPC-Web endpoint (spec D-011), so no - // application/grpc-web* exclusion is needed. - grpcL := mux.MatchWithWriters(cmux.HTTP2MatchHeaderFieldPrefixSendSettings("content-type", "application/grpc")) - // Other connections go to the HTTP server, which supports HTTP/1.1 only. - // Plain HTTP/2 and mixed connections are outside the D-017 contract. - httpL := mux.Match(cmux.Any()) - var opts = []http.ServerOption{ - // The HTTP server never binds its own socket: it serves the - // connections cmux matched to it on the shared listener. - http.Listener(httpL), + // The HTTP server never binds its own socket: it serves the shared + // listener this function bound. + http.Listener(lis), http.Middleware( recovery.Recovery(), logging.Server(logger), @@ -182,24 +150,85 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) srv.Handle(service.DownloadPath, CORSMiddleware(c.GetHttp().GetCors().GetAllowOrigins(), downloadHandler)) api.RegisterStatusServiceHTTPServer(srv, service.NewStatusService(Version, providers)) + routes := srv.Handler srv.Protocols = new(h.Protocols) srv.Protocols.SetHTTP1(true) - // Bound HTTP headers after handoff; streaming responses have no deadline. + srv.Protocols.SetHTTP2(true) + srv.Protocols.SetUnencryptedHTTP2(true) + // Bound how long a client may take to deliver its request headers + // (slowloris protection). Response streaming is not affected. srv.ReadHeaderTimeout = 60 * time.Second + // One proxy and one transport for the process: its connection pool is + // reused across requests. Dispatch each request: native gRPC goes through + // the standard reverse proxy to the private native server; everything else + // uses the existing HTTP routes. CAS has no gRPC-Web endpoint (spec + // D-011); the grpc-web exclusion keeps the content-type check consistent + // with the control plane. + proxy := grpcProxy(grpcServers.PrivateAddr, logger) + srv.Handler = h.HandlerFunc(func(w h.ResponseWriter, r *h.Request) { + if r.ProtoMajor == 2 && isNativeGRPCContentType(r.Header.Get("Content-Type")) { + proxy.ServeHTTP(w, r) + return + } + routes.ServeHTTP(w, r) + }) log.NewHelper(logger).Infof("[shared] CAS endpoint (%s) listening on: %s", scheme, lis.Addr().String()) return &APIServer{ - lis: lis, - mux: mux, - grpcL: grpcL, - httpL: httpL, - grpcSrv: grpcServers.Handler, - httpSrv: srv, - scheme: scheme, + lis: lis, + httpSrv: srv, + grpcSrv: grpcServers.Private, + grpcAddr: grpcServers.PrivateAddr, + scheme: scheme, }, nil } +// grpcProxy builds the reverse proxy that carries native gRPC requests to the +// private native gRPC server over h2c. httputil.ReverseProxy streams request +// and response bodies, forwards hop-by-hop-excluded headers (including gRPC +// metadata such as the authorization token), and copies response trailers, +// so the existing gRPC frames pass through without decoding or rebuilding. +// FlushInterval zero flushes each write immediately, which gRPC streaming +// requires. The transport speaks only unencrypted HTTP/2 (h2c prior +// knowledge): the private server is gRPC-only and loopback-only. +func grpcProxy(target string, logger log.Logger) *httputil.ReverseProxy { + transport := &h.Transport{ + Protocols: new(h.Protocols), + DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext, + MaxIdleConns: 100, + MaxIdleConnsPerHost: 100, + IdleConnTimeout: 90 * time.Second, + ExpectContinueTimeout: time.Second, + // Byte-exact pass-through: gRPC carries its own per-message content + // coding, and the transport must neither add transport-level gzip + // headers nor transparently decode the proxied body. + DisableCompression: true, + } + transport.Protocols.SetUnencryptedHTTP2(true) + transport.Protocols.SetHTTP1(false) + return &httputil.ReverseProxy{ + Rewrite: func(pr *httputil.ProxyRequest) { + pr.SetURL(&url.URL{Scheme: "http", Host: target}) + }, + Transport: transport, + FlushInterval: 0, + ErrorHandler: func(w h.ResponseWriter, _ *h.Request, err error) { + log.NewHelper(logger).Errorf("msg", "shared endpoint gRPC proxy error", "error", err) + w.WriteHeader(h.StatusBadGateway) + }, + } +} + +// isNativeGRPCContentType reports whether a content type selects the native +// gRPC protocol family, excluding gRPC-Web. CAS has no gRPC-Web endpoint, but +// the exclusion keeps the dispatch consistent with the control plane. +func isNativeGRPCContentType(contentType string) bool { + contentType = strings.ToLower(contentType) + return strings.HasPrefix(contentType, "application/grpc") && + !strings.HasPrefix(contentType, "application/grpc-web") +} + // claimsFunc returns the claims function for the JWT middleware that casts the claims to the correct type func claimsFunc() middlewares_http.ClaimsFunc { return func() jwt.Claims { @@ -215,7 +244,6 @@ func isOrdinaryStopError(err error) bool { } return errors.Is(err, net.ErrClosed) || errors.Is(err, h.ErrServerClosed) || - errors.Is(err, cmux.ErrServerClosed) || errors.Is(err, grpcLib.ErrServerStopped) } diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index 0af0c19fc..6b5318615 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -32,6 +32,7 @@ import ( "os" "path/filepath" "sync" + "sync/atomic" "testing" "time" @@ -351,25 +352,23 @@ func TestCASSharedProtocols(t *testing.T) { } sharedURL, _ := startCASTestServers(t, opts) - perform := func(kind string) { - tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // test certificate - if kind == casGRPCRequest { - // Native gRPC arrives over HTTP/2, as the gRPC upstream - // pool of a configured ingress dials it. - tr.Protocols.SetHTTP2(true) - tr.Protocols.SetUnencryptedHTTP2(true) - } else { - // D-017/R-009: the HTTP side of the shared endpoint - // supports HTTP/1.1 only. A plain HTTP/2 client - // acknowledges the matcher's injected SETTINGS frame, - // and net/http rejects that acknowledgement (GOAWAY, - // PROTOCOL_ERROR). The HTTP upstream pool therefore dials - // HTTP/1.1 (D-018). - tr.Protocols.SetHTTP1(true) - } - defer tr.CloseIdleConnections() - client := &http.Client{Transport: tr, Timeout: 5 * time.Second} + var connections atomic.Int32 + // One catch-all upstream connection that carries both protocols, as + // a single-location ingress with an h2c (or TLS h2) upstream sends + // them. HTTP/1 must stay off in the client: net/http only uses h2c + // prior knowledge when HTTP/1 is disabled. + tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // test certificate + tr.Protocols.SetHTTP2(true) + tr.Protocols.SetUnencryptedHTTP2(true) + tr.Protocols.SetHTTP1(false) + tr.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) { + connections.Add(1) + return (&net.Dialer{}).DialContext(ctx, network, addr) + } + defer tr.CloseIdleConnections() + client := &http.Client{Transport: tr, Timeout: 5 * time.Second} + perform := func(kind string) { path, method := "/cas.v1.StatusService/Statusz", http.MethodPost if kind == casHTTPRequest { path, method = "/statusz", http.MethodGet @@ -398,26 +397,26 @@ func TestCASSharedProtocols(t *testing.T) { } require.Equal(t, expected, resp.StatusCode, kind) if kind == casGRPCRequest { + // Native gRPC requires HTTP/2 and its trailers round-trip + // through the reverse proxy to the private native server. require.Equal(t, 2, resp.ProtoMajor) require.Equal(t, "0", resp.Trailer.Get("Grpc-Status")) - } else { - // D-017/R-009: the HTTP side of the shared endpoint is HTTP/1.1. - require.Equal(t, 1, resp.ProtoMajor) } } - // Each protocol works through the shared port on its own - // connections, as a proxy with separate upstream pools sends them. - // A plain HTTP/2 connection cannot fall through to the HTTP side: - // D-017 records that the matcher's SETTINGS injection makes - // net/http reject the client's acknowledgement (GOAWAY, - // PROTOCOL_ERROR), so mixed native-gRPC and plain-HTTP reuse of - // one upstream HTTP/2 connection is unsupported by contract. + // D-020: the shared endpoint dispatches each request, so native + // gRPC and ordinary HTTP interleave in any order — including on + // one pooled upstream HTTP/2 connection, as a single-location + // ingress sends them. perform(casGRPCRequest) perform(casHTTPRequest) + perform(casGRPCRequest) perform(casPreflightRequest) perform(casHTTPRequest) perform(casGRPCRequest) + + // All six requests reused one HTTP/2 connection. + require.EqualValues(t, 1, connections.Load()) }) } } @@ -515,7 +514,7 @@ func TestCASLegacyListenerAndTLSConfiguration(t *testing.T) { opts.ServerConfig.LegacyGrpcListener = new(false) servers, err := NewGRPCServers(opts) require.NoError(t, err) - defer servers.Handler.Server.Stop() + defer func() { require.NoError(t, servers.Private.Stop(context.Background())) }() _, err = NewHTTPServer(opts, servers) require.ErrorContains(t, err, tc.wantErr) }) @@ -604,48 +603,12 @@ func TestCASSharedShutdownDrainsUpload(t *testing.T) { } } -// Slow clients must not survive shutdown while cmux is still reading their first request. -func TestCASSharedShutdownClosesUnclassifiedConnections(t *testing.T) { - for _, useTLS := range []bool{false, true} { - t.Run(map[bool]string{false: "plaintext", true: "TLS handshake"}[useTLS], func(t *testing.T) { - opts := casTestOpts(t) - opts.ServerConfig.LegacyGrpcListener = new(false) - if useTLS { - opts.ServerConfig.Http.TlsConfig = casTestTLS(t) - } - servers, err := NewGRPCServers(opts) - require.NoError(t, err) - api, err := NewHTTPServer(opts, servers) - require.NoError(t, err) - endpoint, err := api.Endpoint() - require.NoError(t, err) - startCASTestTransport(t, api) - conn, err := net.Dial("tcp", endpoint.Host) - require.NoError(t, err) - defer conn.Close() - if !useTLS { - _, err = conn.Write([]byte("PRI ")) - require.NoError(t, err) - } - // Allow the connection to reach the classifier, including a TLS handshake. - time.Sleep(50 * time.Millisecond) - ctx, cancel := context.WithTimeout(context.Background(), time.Second) - defer cancel() - require.NoError(t, api.Stop(ctx)) - require.NoError(t, conn.SetReadDeadline(time.Now().Add(200*time.Millisecond))) - var data [1]byte - _, err = conn.Read(data[:]) - require.ErrorIs(t, err, io.EOF, "shutdown must close a socket still waiting for classification") - }) - } -} - func TestCASSharedInitializationDoesNotBindOnInvalidKey(t *testing.T) { opts := casTestOpts(t) opts.ServerConfig.LegacyGrpcListener = new(false) servers, err := NewGRPCServers(opts) require.NoError(t, err) - defer func() { require.NoError(t, servers.Handler.Stop(context.Background())) }() + defer func() { require.NoError(t, servers.Private.Stop(context.Background())) }() reserved, err := net.Listen("tcp", casAnyAddr) require.NoError(t, err) opts.ServerConfig.Http.Addr = reserved.Addr().String() diff --git a/app/artifact-cas/internal/server/ingress_test.go b/app/artifact-cas/internal/server/ingress_test.go index da709a989..40c58898e 100644 --- a/app/artifact-cas/internal/server/ingress_test.go +++ b/app/artifact-cas/internal/server/ingress_test.go @@ -53,7 +53,7 @@ func TestCASIngressNGINX(t *testing.T) { require.NotNil(t, net.ParseIP(backendIP), "CAS_INGRESS_BACKEND_IP must be reachable from the cluster nodes") ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) defer cancel() - namespace := fmt.Sprintf("cas-cmux-%d", time.Now().UnixNano()) + namespace := fmt.Sprintf("cas-shared-%d", time.Now().UnixNano()) kubectl := func(input []byte, args ...string) []byte { t.Helper() full := append([]string{"--kubeconfig", kubeconfig, "--namespace", namespace}, args...) @@ -84,7 +84,7 @@ func TestCASIngressNGINX(t *testing.T) { "--set", "development=true", "--set", "fullnameOverride=cas-probe", "--set", "cas.ingress.enabled=true", "--set", "cas.ingress.ingressClassName=nginx", "--set", "cas.ingress.hostname="+hostname, "--set", "cas.ingress.tls=true", - "--show-only", "templates/cas/service-http.yaml", "--show-only", "templates/cas/ingress.yaml", "--show-only", "templates/cas/ingress-grpc-shared.yaml") + "--show-only", "templates/cas/service-http.yaml", "--show-only", "templates/cas/ingress.yaml") helm.Env = append(os.Environ(), "HELM_PLUGINS="+t.TempDir()) manifest, err := helm.Output() require.NoError(t, err) @@ -101,14 +101,11 @@ ports: - name: http protocol: TCP port: %s - - name: grpc-shared - protocol: TCP - port: %s endpoints: - addresses: ["%s"] conditions: ready: true -`, port, port, backendIP) +`, port, backendIP) kubectl([]byte(endpointSlice), "apply", "-f", "-") lis, err := net.Listen("tcp", casAnyAddr) require.NoError(t, err) diff --git a/deployment/chainloop/README.md b/deployment/chainloop/README.md index 1fa2c08ec..30ebd34d9 100644 --- a/deployment/chainloop/README.md +++ b/deployment/chainloop/README.md @@ -245,9 +245,9 @@ The control plane ingress serves HTTP, gRPC-Web, and native gRPC on the same hos Its HTTP server routes each request, so these protocols can share an upstream HTTP/2 connection. The existing gRPC ingress stays available. Existing clients can keep their current endpoints. To test the shared endpoint, point a gRPC client to the control plane HTTP hostname on port 443. -The CAS shared endpoint classifies connections with cmux and keeps the native gRPC transport for ByteStream (spec D-017). `cas.ingress` sends HTTP routes over HTTP/1.1. The second Ingress, `cas.ingress.grpc`, sends the Google ByteStream, CAS, health, reflection, metadata, and channelz RPCs over gRPC. Both target the same CAS application listener through one Service. Its `http` port defaults to 80 and its `grpc-shared` alias defaults to 81 (`cas.service.ports.grpcShared`). Ingress-nginx uses pod endpoints for HTTP and the ClusterIP for gRPC. The distinct Service ports and upstream addresses isolate its connection cache (D-019). Keep the port numbers different and at least one upstream on the ClusterIP. The chart rejects unsafe combinations and headless Services for this arrangement. +The CAS shared endpoint mirrors the control plane (spec D-020): one standard HTTP listener dispatches each request, and native gRPC reaches the native transport through an in-process reverse proxy. `cas.ingress` is one catch-all route on the same hostname — one path `/` with an HTTP/2 (gRPC) upstream — so ByteStream uploads and downloads, browser downloads, and status probes share one Service port and one certificate, including mixed requests on one upstream connection. -Set gRPC transport annotations under `cas.ingress.grpc.annotations`, such as `backend-protocol: GRPCS` for TLS to the pod. HTTP routes use HTTP or HTTPS with HTTP/1.1 upstream. Both Ingress objects share a TLS Secret when `cas.ingress.tls=true`. Only one requests the certificate from cert-manager. A pre-created Secret also works without issuer annotations. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. The shared chart routing has a local single-worker ingress-nginx check. F5, Traefik, and the full streaming-performance evidence still need qualification. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). +Set gRPC transport annotations under `cas.ingress.annotations`, such as `backend-protocol: GRPCS` for TLS to the pod. The upload body-size annotations keep their existing defaults. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. The shared chart routing has a local single-worker ingress-nginx check. F5, Traefik, and the full streaming-performance evidence still need qualification. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). The chart configures HTTP/2 upstream traffic for these controllers: @@ -838,7 +838,6 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | `cas.service.type` | Service type | `ClusterIP` | | `cas.service.ports.http` | cas service HTTP port | `80` | -| `cas.service.ports.grpcShared` | Service port alias for shared-ingress gRPC (same CAS HTTP listener) | `81` | | `cas.service.ports.https` | cas service HTTPS port | `443` | | `cas.service.nodePorts.http` | Node port for HTTP | `""` | | `cas.service.nodePorts.https` | Node port for HTTPS | `""` | @@ -846,7 +845,7 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | `cas.service.loadBalancerIP` | cas service Load Balancer IP | `""` | | `cas.service.loadBalancerSourceRanges` | cas service Load Balancer sources | `[]` | | `cas.service.externalTrafficPolicy` | cas service external traffic policy | `Cluster` | -| `cas.service.annotations` | Additional custom annotations for the cas service. Do not set a Traefik h2c scheme here: the HTTP routes must dial HTTP/1.1 (spec D-017/D-018). | `{}` | +| `cas.service.annotations` | Annotations for the shared HTTP and gRPC service. Traefik uses h2c for HTTP/2 upstream traffic. | See values.yaml | | `cas.service.extraPorts` | Extra ports to expose in cas service (normally used with the `sidecars` value) | `[]` | | `cas.service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` | | `cas.service.sessionAffinityConfig` | Additional settings for the sessionAffinity | `{}` | @@ -868,7 +867,7 @@ Once done, you can access with [two predefined users](https://github.com/chainlo | `cas.ingress.hostname` | Default host for the ingress record | `cas.dev.local` | | `cas.ingress.ingressClassName` | IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+) | `""` | | `cas.ingress.path` | Default path for the ingress record | `/` | -| `cas.ingress.annotations` | Annotations for the shared CAS HTTP ingress (downloads, status). gRPC upstream settings go in `cas.ingress.grpc.annotations`. | `{}` | +| `cas.ingress.annotations` | Annotations for the shared CAS ingress: HTTP/2 (gRPC) upstream plus the upload body limits. | See values.yaml | | `cas.ingress.tls` | Enable TLS configuration for the host defined at `controlplane.ingress.hostname` parameter | `false` | | `cas.ingress.selfSigned` | Create a TLS secret for this ingress record using self-signed certificates generated by Helm | `false` | | `cas.ingress.extraHosts` | An array with additional hostname(s) to be covered with the ingress record | `[]` | diff --git a/deployment/chainloop/chart_test.go b/deployment/chainloop/chart_test.go index ada05c1e2..50f0f45c5 100644 --- a/deployment/chainloop/chart_test.go +++ b/deployment/chainloop/chart_test.go @@ -21,13 +21,10 @@ import ( "io" "os" "os/exec" - "strings" "testing" "time" - api "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" "github.com/stretchr/testify/require" - "google.golang.org/genproto/googleapis/bytestream" "gopkg.in/yaml.v3" ) @@ -39,11 +36,11 @@ const ( ) // The control plane serves every protocol per request on its shared ingress -// (replacement spec D-001/D-002), so the whole ingress uses the gRPC backend -// protocol and the Traefik h2c scheme. The CAS classifies connections with -// cmux and its native transport (D-017), so its shared HTTP ingress keeps -// HTTP/1.1 and the gRPC RPC paths are split into a second Ingress with their -// own upstream pool (D-018). The legacy gRPC resources keep their own upstreams. +// (replacement spec D-001/D-002). The CAS does the same (spec D-020): the +// shared listener dispatches each request and reaches the native transport +// through an in-process reverse proxy, so one catch-all location with the +// gRPC backend protocol serves every request. The legacy gRPC resources keep +// their own upstreams. func TestSharedIngress(t *testing.T) { t.Run("controlplane", func(t *testing.T) { testSharedControlplaneIngress(t) }) t.Run("cas", func(t *testing.T) { testSharedCASIngress(t) }) @@ -142,41 +139,37 @@ func testSharedControlplaneIngress(t *testing.T) { } } -// The CAS shared HTTP ingress keeps an HTTP/1.1 upstream, and the gRPC RPC -// paths render into a second Ingress on the same hostname with their own -// upstream pool (spec D-018). +// The CAS shared ingress mirrors the control plane (spec D-020): one +// catch-all location, the gRPC backend protocol for its HTTP/2 upstream, and +// the upload body limits. One Service port; no RPC path rules. func testSharedCASIngress(t *testing.T) { const ( grpcBackend = "GRPC" + h2cScheme = "h2c" chartFullname = "test-chainloop" ) for _, tc := range []struct { - name string - args []string - fullname string - grpcProtocol string - sharedHTTP2 string - grpcHTTP2 string - traefikScheme string + name string + args []string + fullname string + backendProtocol string + traefikScheme string }{ - {name: "defaults", fullname: chartFullname, grpcProtocol: grpcBackend, grpcHTTP2: annotationTrue}, - {name: "custom name", args: []string{helmSet, "fullnameOverride=shared-api"}, fullname: "shared-api", grpcProtocol: grpcBackend, grpcHTTP2: annotationTrue}, + {name: "defaults", fullname: chartFullname, backendProtocol: grpcBackend, traefikScheme: h2cScheme}, + {name: "custom name", args: []string{helmSet, "fullnameOverride=shared-api"}, fullname: "shared-api", backendProtocol: grpcBackend, traefikScheme: h2cScheme}, { - // A global annotation reaches both Ingress objects; on the gRPC - // Ingress it overrides the chart default. - name: "common annotations merge", - args: []string{helmSetString, `commonAnnotations.nginx\.org/http2=false`}, - fullname: chartFullname, - grpcProtocol: grpcBackend, sharedHTTP2: "false", grpcHTTP2: "false", + name: "common annotations merge", + args: []string{helmSetString, `commonAnnotations.nginx\.org/http2=false`}, + fullname: chartFullname, + backendProtocol: grpcBackend, traefikScheme: h2cScheme, }, { name: "encrypted upstream overrides", args: []string{ - helmSetString, `cas.ingress.grpc.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, + helmSetString, `cas.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPCS`, helmSetString, `cas.service.annotations.traefik\.ingress\.kubernetes\.io/service\.serversscheme=https`, }, - fullname: chartFullname, - grpcProtocol: "GRPCS", sharedHTTP2: "", grpcHTTP2: annotationTrue, traefikScheme: "https", + fullname: chartFullname, backendProtocol: "GRPCS", traefikScheme: "https", }, } { t.Run(tc.name, func(t *testing.T) { @@ -184,12 +177,15 @@ func testSharedCASIngress(t *testing.T) { resources := renderChart(t, append(tc.args, helmSet, casIngressEnabled, helmSet, "cas.ingress.hostname=shared.example.test")) - // The shared HTTP ingress dials HTTP/1.1: no gRPC annotations by default. + // One catch-all ingress serves every protocol on the shared + // listener, over an HTTP/2 upstream. ingress, ok := resources["Ingress/"+serviceName] require.True(t, ok) - require.Empty(t, ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) - require.Equal(t, tc.sharedHTTP2, ingress.Metadata.Annotations["nginx.org/http2"]) - require.Empty(t, ingress.Metadata.Annotations["nginx.org/grpc-services"]) + require.Equal(t, tc.backendProtocol, ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) + require.Equal(t, serviceName, ingress.Metadata.Annotations["nginx.org/grpc-services"]) + require.Equal(t, annotationTrue, ingress.Metadata.Annotations["nginx.org/http2"]) + require.Equal(t, "100m", ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/proxy-body-size"]) + require.Equal(t, "3M", ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/client-body-buffer-size"]) require.Len(t, ingress.Spec.Rules, 1) require.Equal(t, "shared.example.test", ingress.Spec.Rules[0].Host) require.Len(t, ingress.Spec.Rules[0].HTTP.Paths, 1) @@ -198,35 +194,15 @@ func testSharedCASIngress(t *testing.T) { require.Equal(t, serviceName, path.Backend.Service.Name) require.Equal(t, "http", path.Backend.Service.Port.Name) - // The gRPC RPC paths render into their own Ingress on the same - // hostname, with their own annotations and upstream pool. - grpcIngress, ok := resources["Ingress/"+serviceName+"-grpc"] - require.True(t, ok) - require.Equal(t, tc.grpcProtocol, grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) - require.Equal(t, annotationTrue, grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/service-upstream"]) - require.Empty(t, ingress.Metadata.Annotations["nginx.ingress.kubernetes.io/service-upstream"]) - require.Equal(t, serviceName, grpcIngress.Metadata.Annotations["nginx.org/grpc-services"]) - require.Equal(t, tc.grpcHTTP2, grpcIngress.Metadata.Annotations["nginx.org/http2"]) - require.Equal(t, "100m", grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/proxy-body-size"]) - require.Equal(t, "3M", grpcIngress.Metadata.Annotations["nginx.ingress.kubernetes.io/client-body-buffer-size"]) - require.Len(t, grpcIngress.Spec.Rules, 1) - require.Equal(t, "shared.example.test", grpcIngress.Spec.Rules[0].Host) - require.Len(t, grpcIngress.Spec.Rules[0].HTTP.Paths, 8) - for _, p := range grpcIngress.Spec.Rules[0].HTTP.Paths { - require.Equal(t, serviceName, p.Backend.Service.Name) - require.Equal(t, "grpc-shared", p.Backend.Service.Port.Name) - } - require.Equal(t, "/google.bytestream.ByteStream/", grpcIngress.Spec.Rules[0].HTTP.Paths[0].Path) - require.Equal(t, "/cas.v1.ResourceService/", grpcIngress.Spec.Rules[0].HTTP.Paths[1].Path) - + // No second gRPC-path Ingress and no Service port alias remain. + _, ok = resources["Ingress/"+serviceName+"-grpc"] + require.False(t, ok) svc := resources["Service/"+serviceName] - require.Len(t, svc.Spec.Ports, 2) + require.Len(t, svc.Spec.Ports, 1) require.Equal(t, "http", svc.Spec.Ports[0].Name) - require.Equal(t, "http", svc.Spec.Ports[0].TargetPort) - require.Equal(t, "grpc-shared", svc.Spec.Ports[1].Name) - require.Equal(t, 81, svc.Spec.Ports[1].Port) - require.Equal(t, "http", svc.Spec.Ports[1].TargetPort) require.Equal(t, tc.traefikScheme, svc.Metadata.Annotations["traefik.ingress.kubernetes.io/service.serversscheme"]) + + // The legacy gRPC ingress keeps its own upstream. casLegacy := resources["Ingress/"+tc.fullname+"-cas-api"] require.Equal(t, grpcBackend, casLegacy.Metadata.Annotations["nginx.ingress.kubernetes.io/backend-protocol"]) require.Len(t, casLegacy.Spec.Rules, 1) @@ -270,86 +246,12 @@ type chartResource struct { } `yaml:"spec"` } -func TestCASSharedIngressRPCPaths(t *testing.T) { - resources := renderChart(t, []string{helmSet, casIngressEnabled}) - ingress := resources["Ingress/test-chainloop-cas-grpc"] - // Derive the application routes from the generated descriptors, so a typo - // in the values cannot become the expected result as well. - for _, name := range []string{ - string(bytestream.File_google_bytestream_bytestream_proto.Services().Get(0).FullName()), - api.ResourceService_ServiceDesc.ServiceName, - api.StatusService_ServiceDesc.ServiceName, - "grpc.health.v1.Health", "grpc.reflection.v1.ServerReflection", - "grpc.reflection.v1alpha.ServerReflection", "kratos.api.Metadata", - "grpc.channelz.v1.Channelz", - } { - covered := false - for _, rule := range ingress.Spec.Rules { - for _, path := range rule.HTTP.Paths { - covered = covered || strings.HasPrefix("/"+name+"/", path.Path) - } - } - require.True(t, covered, "no native gRPC route for %s", name) - } -} - func TestCASSharedIngressTLS(t *testing.T) { - for _, annotations := range []string{"cas.ingress.annotations", "cas.ingress.grpc.annotations", "commonAnnotations"} { - t.Run(annotations, func(t *testing.T) { - resources := renderChart(t, []string{helmSet, casIngressEnabled, helmSet, "cas.ingress.tls=true", - helmSetString, annotations + `.cert-manager\.io/cluster-issuer=test-issuer`}) - owners := 0 - for _, name := range []string{"test-chainloop-cas", "test-chainloop-cas-grpc"} { - ingress := resources["Ingress/"+name] - if ingress.Metadata.Annotations["cert-manager.io/cluster-issuer"] != "" { - owners++ - } - require.Len(t, ingress.Spec.TLS, 1, "both routes must reference the certificate") - require.Equal(t, "cas.dev.local-tls", ingress.Spec.TLS[0].SecretName) - require.Equal(t, []string{"cas.dev.local"}, ingress.Spec.TLS[0].Hosts) - } - require.Equal(t, 1, owners, "only one Ingress should request the shared certificate") - }) - } -} - -func TestCASIngressInvalidPoolConfiguration(t *testing.T) { - for _, tc := range []struct { - name string - args []string - message string - }{ - {name: "pod endpoints for both protocols", args: []string{helmSetString, `cas.ingress.grpc.annotations.nginx\.ingress\.kubernetes\.io/service-upstream=false`}, message: "requires separate ingress-nginx upstream addresses"}, - {name: "headless service", args: []string{helmSet, "cas.service.clusterIP=None"}, message: "requires a ClusterIP Service"}, - {name: "same service port", args: []string{helmSet, "cas.service.ports.grpcShared=80"}, message: "grpcShared must differ"}, - {name: "gRPC on HTTP routes", args: []string{helmSetString, `cas.ingress.annotations.nginx\.ingress\.kubernetes\.io/backend-protocol=GRPC`}, message: "must use an HTTP/1.1 upstream"}, - } { - t.Run(tc.name, func(t *testing.T) { - out, err := chartTemplate(t, append(tc.args, helmSet, casIngressEnabled)) - require.Error(t, err) - require.Contains(t, string(out), tc.message) - }) - } -} - -func TestCASSharedServicePortAlias(t *testing.T) { - for _, tc := range []struct { - name string - args []string - alias bool - }{ - {name: "ingress disabled"}, - {name: "HTTP only", args: []string{helmSet, casIngressEnabled, "--set-json", "cas.ingress.grpc.paths=[]"}}, - {name: "both use ClusterIP", args: []string{helmSet, casIngressEnabled, helmSetString, `cas.ingress.annotations.nginx\.ingress\.kubernetes\.io/service-upstream=true`}, alias: true}, - } { - t.Run(tc.name, func(t *testing.T) { - resources := renderChart(t, tc.args) - ports := resources["Service/test-chainloop-cas"].Spec.Ports - if tc.alias { - require.Len(t, ports, 2) - } else { - require.Len(t, ports, 1) - } - }) - } + resources := renderChart(t, []string{helmSet, casIngressEnabled, helmSet, "cas.ingress.tls=true", + helmSetString, `cas.ingress.annotations.cert-manager\.io/cluster-issuer=test-issuer`}) + ingress := resources["Ingress/test-chainloop-cas"] + require.Equal(t, "test-issuer", ingress.Metadata.Annotations["cert-manager.io/cluster-issuer"]) + require.Len(t, ingress.Spec.TLS, 1) + require.Equal(t, "cas.dev.local-tls", ingress.Spec.TLS[0].SecretName) + require.Equal(t, []string{"cas.dev.local"}, ingress.Spec.TLS[0].Hosts) } diff --git a/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml b/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml deleted file mode 100644 index 4f9911ec5..000000000 --- a/deployment/chainloop/templates/cas/ingress-grpc-shared.yaml +++ /dev/null @@ -1,83 +0,0 @@ -{{- /* -Copyright Chainloop, Inc. All Rights Reserved. -SPDX-License-Identifier: APACHE-2.0 -*/}} - -{{- /* -gRPC routes of the shared CAS endpoint (spec D-018). Rendered as a second -Ingress on the same hostname as cas/ingress.yaml, with its own annotations: -controllers that set the backend protocol per Ingress resource (community -nginx `backend-protocol`) need a dedicated object for the gRPC paths, so the -gRPC routes use HTTP/2 (grpc_pass) while HTTP routes keep HTTP/1.1. -ingress-nginx additionally dials the ClusterIP for gRPC and pod endpoints -for HTTP, using distinct Service port names as backend identities. Its global -keepalive pool cannot reuse a socket across protocols. Controllers that conflict on unrelated -same-host Ingress objects (F5 NGINX) need their own resource instead; see -the chart README. -*/}} -{{- if and .Values.cas.ingress.enabled .Values.cas.ingress.grpc.paths }} -{{- $fullName := include "chainloop.cas.fullname" . -}} -{{- $grpcName := printf "%s-grpc" $fullName -}} -{{- $annotations := include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.grpc.annotations .Values.commonAnnotations .Values.cas.ingress.grpc.defaultAnnotations) "context" .) | fromYaml -}} -{{- if eq (toString .Values.cas.service.ports.grpcShared) (toString (coalesce .Values.cas.service.port .Values.cas.service.ports.http)) -}} - {{- fail "cas.service.ports.grpcShared must differ from the HTTP Service port to isolate ingress upstream connections" -}} -{{- end -}} -{{- $httpAnnotations := include "common.tplvalues.merge" (dict "values" (list .Values.cas.ingress.annotations .Values.commonAnnotations) "context" .) | fromYaml -}} -{{- if has (upper (default "" (index $annotations "nginx.ingress.kubernetes.io/backend-protocol"))) (list "GRPC" "GRPCS") -}} - {{- if and (ne (default "false" (index $annotations "nginx.ingress.kubernetes.io/service-upstream")) "true") (ne (default "false" (index $httpAnnotations "nginx.ingress.kubernetes.io/service-upstream")) "true") -}} - {{- fail "CAS cmux requires separate ingress-nginx upstream addresses: set service-upstream=true on cas.ingress.grpc (the default) or cas.ingress" -}} - {{- end -}} - {{- if eq .Values.cas.service.clusterIP "None" -}} - {{- fail "CAS cmux ingress-nginx routing requires a ClusterIP Service; cas.service.clusterIP cannot be None" -}} - {{- end -}} - {{- if has (upper (default "HTTP" (index $httpAnnotations "nginx.ingress.kubernetes.io/backend-protocol"))) (list "GRPC" "GRPCS") -}} - {{- fail "cas.ingress must use an HTTP/1.1 upstream (HTTP or HTTPS); set the gRPC protocol only in cas.ingress.grpc.annotations" -}} - {{- end -}} -{{- end -}} -{{- /* Only one Ingress requests the shared certificate. If the HTTP Ingress - requests it, the gRPC Ingress only references its Secret. */ -}} -{{- if include "common.ingress.certManagerRequest" (dict "annotations" $httpAnnotations) -}} - {{- range $key, $_ := deepCopy $annotations -}} - {{- if or (hasPrefix "cert-manager.io/" $key) (hasPrefix "acme.cert-manager.io/" $key) (eq $key "kubernetes.io/tls-acme") -}} - {{- $_ := unset $annotations $key -}} - {{- end -}} - {{- end -}} -{{- end -}} -apiVersion: {{ include "common.capabilities.ingress.apiVersion" . }} -kind: Ingress -metadata: - name: {{ $grpcName }} - namespace: {{ include "common.names.namespace" . | quote }} - labels: {{- include "chainloop.cas.labels" . | nindent 4 }} - {{- with $annotations }} - annotations: {{- include "common.tplvalues.render" (dict "value" . "context" $) | nindent 4 }} - {{- end }} -spec: - {{- if and .Values.cas.ingress.ingressClassName (eq "true" (include "common.ingress.supportsIngressClassname" .)) }} - ingressClassName: {{ .Values.cas.ingress.ingressClassName | quote }} - {{- end }} - rules: - {{- if .Values.cas.ingress.hostname }} - - host: {{ .Values.cas.ingress.hostname }} - http: - paths: - {{- range .Values.cas.ingress.grpc.paths }} - - path: {{ . }} - {{- if eq "true" (include "common.ingress.supportsPathType" $) }} - pathType: {{ default "ImplementationSpecific" $.Values.cas.ingress.pathType }} - {{- end }} - backend: {{- include "common.ingress.backend" (dict "serviceName" $fullName "servicePort" "grpc-shared" "context" $) | nindent 14 }} - {{- end }} - {{- end }} - {{- if or .Values.cas.ingress.tls .Values.cas.ingress.extraTls }} - tls: - {{- if .Values.cas.ingress.tls }} - - hosts: - - {{ .Values.cas.ingress.hostname | quote }} - secretName: {{ printf "%s-tls" .Values.cas.ingress.hostname }} - {{- end }} - {{- if .Values.cas.ingress.extraTls }} - {{- include "common.tplvalues.render" (dict "value" .Values.cas.ingress.extraTls "context" $) | nindent 4 }} - {{- end }} - {{- end }} -{{- end }} \ No newline at end of file diff --git a/deployment/chainloop/templates/cas/service-http.yaml b/deployment/chainloop/templates/cas/service-http.yaml index bb32436d2..8fad843f3 100644 --- a/deployment/chainloop/templates/cas/service-http.yaml +++ b/deployment/chainloop/templates/cas/service-http.yaml @@ -44,14 +44,6 @@ spec: {{- else if eq .Values.cas.service.type "ClusterIP" }} nodePort: null {{- end }} - {{- if and .Values.cas.ingress.enabled .Values.cas.ingress.grpc.paths }} - # This alias and http both target the one shared CAS listener. Distinct - # Service ports give ingress-nginx separate backend identities. - - name: grpc-shared - port: {{ .Values.cas.service.ports.grpcShared }} - targetPort: http - protocol: TCP - {{- end }} {{- if .Values.cas.service.extraPorts }} {{- include "common.tplvalues.render" (dict "value" .Values.cas.service.extraPorts "context" $) | nindent 4 }} {{- end }} diff --git a/deployment/chainloop/values.yaml b/deployment/chainloop/values.yaml index d4ceded4d..39a3f16ad 100644 --- a/deployment/chainloop/values.yaml +++ b/deployment/chainloop/values.yaml @@ -1096,12 +1096,10 @@ cas: type: ClusterIP ## @param cas.service.ports.http cas service HTTP port - ## @param cas.service.ports.grpcShared Service port alias for shared-ingress gRPC (same CAS HTTP listener). ## @param cas.service.ports.https cas service HTTPS port ## ports: http: 80 - grpcShared: 81 https: 443 ## Node ports to expose ## @param cas.service.nodePorts.http Node port for HTTP @@ -1131,12 +1129,11 @@ cas: ## ref http://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip ## externalTrafficPolicy: Cluster - ## @param cas.service.annotations Additional custom annotations for cas service. - ## NOTE: do not set a Traefik `service.serversscheme: h2c` default here. The - ## shared endpoint's HTTP routes must be dialed HTTP/1.1, and a per-Service - ## scheme annotation cannot differ per upstream pool (spec D-017/D-018). + ## @extra cas.service.annotations Annotations for the shared HTTP and gRPC service. Traefik uses h2c for HTTP/2 upstream traffic. ## - annotations: {} + annotations: + ## @skip cas.service.annotations.traefik.ingress.kubernetes.io/service.serversscheme + traefik.ingress.kubernetes.io/service.serversscheme: h2c ## @param cas.service.extraPorts Extra ports to expose in cas service (normally used with the `sidecars` value) ## extraPorts: [] @@ -1240,63 +1237,34 @@ cas: ## NOTE: You may need to set this to '/*' in order to use this with ALB ingress controllers ## path: / - ## @extra cas.ingress.annotations Annotations for the shared CAS HTTP ingress (downloads, status). - ## Do not set a gRPC backend protocol here: the HTTP routes must use an - ## HTTP/1.1 upstream (spec D-017/D-018). The gRPC routes are configured in - ## cas.ingress.grpc below. - ## Use this parameter to set the required annotations for cert-manager, see + ## @extra cas.ingress.annotations Annotations for the shared HTTP, download and native gRPC ingress (spec D-020). + ## The CAS dispatches each request on one listener; native gRPC reaches the + ## native transport through an in-process reverse proxy, so one catch-all + ## location with an HTTP/2 upstream serves every protocol, as on the + ## control plane. Use this parameter to set the required annotations for + ## cert-manager, see ## ref: https://cert-manager.io/docs/usage/ingress/#supported-annotations ## e.g: ## annotations: ## kubernetes.io/cas.ingress.class: nginx ## cert-manager.io/cluster-issuer: cluster-issuer-name ## - annotations: {} - ## @extra cas.ingress.grpc gRPC routes of the shared CAS endpoint. - ## The second Ingress uses a dedicated Service port alias on the same - ## listener. ingress-nginx dials its ClusterIP for gRPC and pod endpoints - ## for HTTP, isolating connection reuse by backend identity and address. - ## Native gRPC keeps HTTP/2 while the HTTP routes keep HTTP/1.1 (D-018). - grpc: - ## @param cas.ingress.grpc.paths RPC path prefixes served by the gRPC upstream pool. - paths: - - /google.bytestream.ByteStream/ - - /cas.v1.ResourceService/ - - /cas.v1.StatusService/ - - /grpc.health.v1.Health/ - - /grpc.reflection.v1.ServerReflection/ - - /grpc.reflection.v1alpha.ServerReflection/ - - /kratos.api.Metadata/ - - /grpc.channelz.v1.Channelz/ - ## @extra cas.ingress.grpc.annotations Annotations for the gRPC-path Ingress. - ## Use this to override the gRPC upstream settings, for example - ## `backend-protocol: GRPCS` when TLS runs to the pod. Do not set these - ## in cas.ingress.annotations: that object belongs to the HTTP routes. - ## e.g: - ## annotations: - ## nginx.ingress.kubernetes.io/backend-protocol: "GRPCS" - annotations: {} - ## @extra cas.ingress.grpc.defaultAnnotations Chart defaults for the gRPC-path Ingress. - ## cas.ingress.grpc.annotations and commonAnnotations override these. - defaultAnnotations: - ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/backend-protocol - ## Tell nginx Ingress Controller to expect gRPC traffic on these paths - nginx.ingress.kubernetes.io/backend-protocol: "GRPC" - ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/service-upstream - # Dial the ClusterIP for gRPC; HTTP uses pod endpoints. ingress-nginx - # keys keepalive connections by upstream address, not Ingress resource. - nginx.ingress.kubernetes.io/service-upstream: "true" - ## @skip cas.ingress.grpc.defaultAnnotations.nginx.org/grpc-services - nginx.org/grpc-services: '{{ include "chainloop.cas.fullname" . }}' - ## @skip cas.ingress.grpc.defaultAnnotations.nginx.org/http2 - nginx.org/http2: "true" - ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/proxy-body-size - # Same upload limit as the independent legacy gRPC ingress. Set 0 to disable. - nginx.ingress.kubernetes.io/proxy-body-size: "100m" - ## @skip cas.ingress.grpc.defaultAnnotations.nginx.ingress.kubernetes.io/client-body-buffer-size - # Improve upload speed by adding client buffering used by http2 control-flows - # https://github.com/chainloop-dev/chainloop/issues/375 - nginx.ingress.kubernetes.io/client-body-buffer-size: "3M" + annotations: + ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/backend-protocol + ## Tell nginx Ingress Controller to expect gRPC traffic + nginx.ingress.kubernetes.io/backend-protocol: "GRPC" + ## @skip cas.ingress.annotations.nginx.org/grpc-services + nginx.org/grpc-services: '{{ include "chainloop.cas.fullname" . }}' + ## @skip cas.ingress.annotations.nginx.org/http2 + nginx.org/http2: "true" + ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/proxy-body-size + # Same upload limit as the independent legacy gRPC ingress. Set 0 to disable. + # Even though we send data in chunks of 1MB, this size refers to all the data sent during the whole streaming session + nginx.ingress.kubernetes.io/proxy-body-size: "100m" + ## @skip cas.ingress.annotations.nginx.ingress.kubernetes.io/client-body-buffer-size + # Improve upload speed by adding client buffering used by http2 control-flows + # https://github.com/chainloop-dev/chainloop/issues/375 + nginx.ingress.kubernetes.io/client-body-buffer-size: "3M" ## @param cas.ingress.tls Enable TLS configuration for the host defined at `controlplane.ingress.hostname` parameter ## TLS certificates will be retrieved from a TLS secret with name: `{{- printf "%s-tls" .Values.controlplane.ingress.hostname }}` ## You can: diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 49522369f..fa0dbde35 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -26,7 +26,7 @@ Control-plane requirements and design superseded by [Spec issue-3628-controlplan For the control plane, the requirements, constraints, proposal, diagram, decision records, milestones, and risks below are historical. Use the replacement spec for all control-plane implementation and qualification. Only CAS requirements remain in scope here. -The CAS transport decision is recorded below in D-017 and D-018. +The CAS transport decision is recorded below in D-020. CAS qualification remains open under the parent ticket. ## Summary @@ -68,11 +68,19 @@ Superseded by R-009 for the CAS. The control-plane scope moved to [the replaceme - ~~CAS browser downloads~~ - ~~Done when: the same checks pass through nginx ingress and through Traefik ingress.~~ -### R-009: CAS shared endpoint protocol scope +### ~~R-009: CAS shared endpoint protocol scope~~ +Superseded by R-010. Supersedes R-001 for the CAS. On its single port, the CAS MUST serve native gRPC over HTTP/2 and plain HTTP over HTTP/1.1, through cmux connection classification with the native gRPC transport (D-017). Plain HTTP/2 other than native gRPC is not supported on the CAS shared port: the gRPC matcher's SETTINGS injection breaks such a connection once it falls through to the HTTP server. Plain HTTP/2 and mixed connections remain control-plane scope under the replacement spec. -- Done when: focused tests cover native gRPC over h2c and TLS, HTTP/1.1 downloads, the status endpoints, and the CORS preflight through the shared port. -- Done when: the chart routes the gRPC paths and the HTTP paths into separate upstream pools, with HTTP/1.1 for the HTTP routes (D-018/D-019). -- Done when: browser downloads through a configured ingress work, and the documentation states that a browser downloading directly from a TLS-configured shared port is not a supported deployment. +~~- Done when: focused tests cover native gRPC over h2c and TLS, HTTP/1.1 downloads, the status endpoints, and the CORS preflight through the shared port.~~ +~~- Done when: the chart routes the gRPC paths and the HTTP paths into separate upstream pools, with HTTP/1.1 for the HTTP routes (D-018/D-019).~~ +~~- Done when: browser downloads through a configured ingress work, and the documentation states that a browser downloading directly from a TLS-configured shared port is not a supported deployment.~~ + +### R-010: CAS shared endpoint protocol scope +Supersedes R-009. On its single port, the CAS MUST serve native gRPC over HTTP/2 and plain HTTP over HTTP/1.1 and HTTP/2, dispatched per request (D-020), including mixed native-gRPC and plain-HTTP requests on one upstream HTTP/2 connection in any order. ByteStream MUST reach the native transport through the in-process reverse proxy. +- Done when: focused tests cover native gRPC over h2c and TLS, HTTP/1.1 and HTTP/2 requests, the status endpoints, and the CORS preflight through the shared port, and the same checks pass with mixed request types on one pooled connection. +- Done when: the 1 GiB transport comparison through the shared endpoint stays within 10% of the legacy native listener with memory bounded and not growing with artifact size. +- Done when: browser downloads through a configured ingress work. + ### R-005: Existing configuration keeps working An installation that does not change its configuration MUST keep working during the deprecation period. The legacy gRPC port keeps serving gRPC at its current address during that period. CLI configurations that use ingress hostnames keep working when the ingress backend moves to the single port. The control plane connection to the CAS, and any other client that uses the legacy gRPC address, MUST move to the single HTTP address before the legacy gRPC port is disabled. @@ -289,38 +297,52 @@ Superseded by D-017. - ~~Evidence: focused tests cover HTTP and gRPC-Web content types over plaintext and TLS, ACKs before and after the first request, subsequent requests on the same connection, incomplete handshakes, and an oversized frame header without its payload. These tests do not establish full R-002 parity or R-003 CAS performance.~~ - ~~Source: [implementation review](https://github.com/chainloop-dev/chainloop/pull/3647#pullrequestreview-5573391966).~~ -### D-017: CAS shared endpoint: cmux with the native transport, and the corrected SETTINGS contract +### ~~D-017: CAS shared endpoint: cmux with the native transport, and the corrected SETTINGS contract~~ +Superseded by D-020. Supersedes D-016 for the CAS endpoint. The control-plane parts of D-014 to D-016 were already superseded by [the replacement spec](issue-3628-controlplane-http-dispatch.md). -The CAS shared endpoint is `server.http.addr`. cmux reads the start of each new connection and hands the whole connection to one of the two existing transports: an HTTP/2 connection whose first request declares a native gRPC content type goes to the existing native gRPC listener (`grpc.Server.Serve`); every other connection goes to the existing HTTP server. ByteStream uploads and downloads therefore keep the grpc-go native transport, so R-003 does not rest on the HTTP adapter. -- Why not the merged control-plane design, per-request `ServeHTTP` dispatch: the [PR #3663 1 GiB comparison](https://github.com/chainloop-dev/chainloop/pull/3663) measured about 1.25 GiB of combined client/server sampled peak live heap for adapter uploads, against about 28 MiB for the native transport, and download throughput 11.8% lower, already past the R-003 limit. grpc-go's handler transport reads the HTTP body in a separate goroutine into an unbounded receive buffer, so read-ahead is not bounded by the application's pace. CAS moves gigabyte streams; the control plane does not. The two services keep different transport designs on purpose. -- Why D-016 is superseded: D-016 assumed a classifier this project owns, which records the SETTINGS ACK offset in its own replay and removes that frame before the HTTP server reads. The cmux library owns the sniffing buffer, so that removal does not exist. Two reproductions on the CAS branch fix the contract instead. First, without SETTINGS injection, direct grpc-go clients deadlock: they send their preface and SETTINGS and wait for the server's SETTINGS before their first HEADERS frame, while the matcher waits for headers to sniff. The gRPC matcher must use cmux's SETTINGS-writing matcher. Second, that matcher writes a SETTINGS frame on every HTTP/2 connection, matched or not, because every HTTP/2 client sends SETTINGS with its preface. A plain HTTP/2 client that falls through to the HTTP server acknowledges the injected SETTINGS frame, and net/http rejects the unsolicited acknowledgement (GOAWAY, PROTOCOL_ERROR). -- Contract: once the matcher has written to a connection, the connection belongs to the native gRPC server. The HTTP side of the shared endpoint supports HTTP/1.1 only; HTTP/2 reaches the CAS only as native gRPC. Supported proxies keep native gRPC and plain HTTP on separate upstream connections (D-014) and dial HTTP/1.1 for the HTTP routes (D-018). -- The CAS has no per-request gRPC adapter fallback. D-017 supersedes that part of D-014 for CAS. All supported ByteStream transfers use the native transport. -- Classification limits apply before a server owns the connection. They bound time, bytes read from the socket, and concurrent classifications. Shutdown closes pending sockets. These limits end at handoff and do not limit active transfers. -- Source: [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) experiment and benchmark, and the local reproductions on its branch. +~~The CAS shared endpoint is `server.http.addr`. cmux reads the start of each new connection and hands the whole connection to one of the two existing transports: an HTTP/2 connection whose first request declares a native gRPC content type goes to the existing native gRPC listener (`grpc.Server.Serve`); every other connection goes to the existing HTTP server. ByteStream uploads and downloads therefore keep the grpc-go native transport, so R-003 does not rest on the HTTP adapter.~~ +~~- Why not the merged control-plane design, per-request `ServeHTTP` dispatch: the [PR #3663 1 GiB comparison](https://github.com/chainloop-dev/chainloop/pull/3663) measured about 1.25 GiB of combined client/server sampled peak live heap for adapter uploads, against about 28 MiB for the native transport, and download throughput 11.8% lower, already past the R-003 limit. grpc-go's handler transport reads the HTTP body in a separate goroutine into an unbounded receive buffer, so read-ahead is not bounded by the application's pace. CAS moves gigabyte streams; the control plane does not. The two services keep different transport designs on purpose.~~ +~~- Why D-016 is superseded: D-016 assumed a classifier this project owns, which records the SETTINGS ACK offset in its own replay and removes that frame before the HTTP server reads. The cmux library owns the sniffing buffer, so that removal does not exist. Two reproductions on the CAS branch fix the contract instead. First, without SETTINGS injection, direct grpc-go clients deadlock: they send their preface and SETTINGS and wait for the server's SETTINGS before their first HEADERS frame, while the matcher waits for headers to sniff. The gRPC matcher must use cmux's SETTINGS-writing matcher. Second, that matcher writes a SETTINGS frame on every HTTP/2 connection, matched or not, because every HTTP/2 client sends SETTINGS with its preface. A plain HTTP/2 client that falls through to the HTTP server acknowledges the injected SETTINGS frame, and net/http rejects the unsolicited acknowledgement (GOAWAY, PROTOCOL_ERROR).~~ +~~- Contract: once the matcher has written to a connection, the connection belongs to the native gRPC server. The HTTP side of the shared endpoint supports HTTP/1.1 only; HTTP/2 reaches the CAS only as native gRPC. Supported proxies keep native gRPC and plain HTTP on separate upstream connections (D-014) and dial HTTP/1.1 for the HTTP routes (D-018).~~ +~~- The CAS has no per-request gRPC adapter fallback. D-017 supersedes that part of D-014 for CAS. All supported ByteStream transfers use the native transport.~~ +~~- Classification limits apply before a server owns the connection. They bound time, bytes read from the socket, and concurrent classifications. Shutdown closes pending sockets. These limits end at handoff and do not limit active transfers.~~ +~~- Source: [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) experiment and benchmark, and the local reproductions on its branch.~~ -### D-018: CAS chart: one hostname, path-split upstream pools, HTTP/1.1 for the HTTP routes +### ~~D-018: CAS chart: one hostname, path-split upstream pools, HTTP/1.1 for the HTTP routes~~ +Superseded by D-020. The chart exposes one CAS hostname and one public port, and splits the routes by protocol into separate upstream pools: the Google ByteStream, CAS, health, reflection, metadata, and channelz RPC paths go to the gRPC pool with an HTTP/2 upstream (nginx `backend-protocol: GRPC` and `grpc-services`; the upload body-size annotations move with them), and `/download/*`, `/statusz`, and `/infoz` go to the HTTP pool, which dials HTTP/1.1, not h2c. One certificate terminates at the edge. -- Why the path split is exact for CAS: CAS has no gRPC-Web (D-011), so its gRPC paths and its HTTP paths are disjoint. No path can carry both protocols. The control plane could not use this split because native gRPC and gRPC-Web share RPC paths; the [PR #3647 NGINX reproduction](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6272007730) showed the path-only split failing there with 415 on gRPC-Web. CAS does not have that failure mode. -- Why HTTP/1.1 upstream on the HTTP pool: D-017. An HTTP/2 upstream for the HTTP routes triggers the matcher's SETTINGS injection, and the client's acknowledgement then breaks the HTTP server connection with GOAWAY. -- Controller notes: community nginx merges same-host paths from multiple Ingress resources and sets the backend protocol per resource, which expresses the split. F5 NGINX conflicts on unrelated same-host Ingresses; use its VirtualServer resource. Traefik applies the `service.serversscheme` annotation per Service, so one Service cannot be dialed h2c for the gRPC routes and HTTP/1.1 for the HTTP routes; the Traefik split needs a second Service for the HTTP routes or a controller-specific resource. The browser download endpoint stays on the HTTP routes: it is published API ([docs](https://docs.chainloop.dev/api-reference/downloadservice/download-artifacts-from-cas)), and CAS does not drop it. -- Source: chart changes on the [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) branch. Connection reuse and certificate ownership are specified in D-019. +~~- Why the path split is exact for CAS: CAS has no gRPC-Web (D-011), so its gRPC paths and its HTTP paths are disjoint. No path can carry both protocols. The control plane could not use this split because native gRPC and gRPC-Web share RPC paths; the [PR #3647 NGINX reproduction](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6272007730) showed the path-only split failing there with 415 on gRPC-Web. CAS does not have that failure mode.~~ +~~- Why HTTP/1.1 upstream on the HTTP pool: D-017. An HTTP/2 upstream for the HTTP routes triggers the matcher's SETTINGS injection, and the client's acknowledgement then breaks the HTTP server connection with GOAWAY.~~ +~~- Controller notes: community nginx merges same-host paths from multiple Ingress resources and sets the backend protocol per resource, which expresses the split. F5 NGINX conflicts on unrelated same-host Ingresses; use its VirtualServer resource. Traefik applies the `service.serversscheme` annotation per Service, so one Service cannot be dialed h2c for the gRPC routes and HTTP/1.1 for the HTTP routes; the Traefik split needs a second Service for the HTTP routes or a controller-specific resource. The browser download endpoint stays on the HTTP routes: it is published API ([docs](https://docs.chainloop.dev/api-reference/downloadservice/download-artifacts-from-cas)), and CAS does not drop it.~~ +~~- Source: chart changes on the [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) branch. Connection reuse and certificate ownership are specified in D-019.~~ -### D-019: CAS ingress connection reuse and certificate ownership +### ~~D-019: CAS ingress connection reuse and certificate ownership~~ +Superseded by D-020. Supplements D-018. Ingress-nginx shares a global keepalive cache. Separate Ingress resources alone do not isolate their connections. -The CAS Service has an HTTP port and a gRPC port alias. Both target the same application listener. The HTTP routes use pod endpoints. The gRPC routes use the Service ClusterIP and the alias. Each protocol gets a different backend identity and upstream address. This keeps one Service, one hostname, and one application listener (R-008). +~~The CAS Service has an HTTP port and a gRPC port alias. Both target the same application listener. The HTTP routes use pod endpoints. The gRPC routes use the Service ClusterIP and the alias. Each protocol gets a different backend identity and upstream address. This keeps one Service, one hostname, and one application listener (R-008).~~ + +~~The Service port numbers MUST differ. At least one upstream MUST use the ClusterIP. The chart rejects configurations that use pod endpoints for both protocols, or a headless Service.~~ -The Service port numbers MUST differ. At least one upstream MUST use the ClusterIP. The chart rejects configurations that use pod endpoints for both protocols, or a headless Service. +~~Both Ingress resources reference the same TLS Secret when shared TLS is enabled. Only one requests the certificate. The HTTP Ingress owns that request when its annotations configure a certificate issuer. Otherwise, the gRPC Ingress can request it. An existing TLS Secret needs no issuer annotation.~~ -Both Ingress resources reference the same TLS Secret when shared TLS is enabled. Only one requests the certificate. The HTTP Ingress owns that request when its annotations configure a certificate issuer. Otherwise, the gRPC Ingress can request it. An existing TLS Secret needs no issuer annotation. +~~- Why: a single-worker ingress-nginx reproduction returned 502 for gRPC after an HTTP request warmed the shared connection cache. Distinct backend identities and addresses passed the same sequence with authenticated ByteStream and browser downloads.~~ +~~- Compatibility: the legacy CAS listener, Service, Ingress, and their settings retain their existing behavior. The control-plane connection continues to use that legacy endpoint.~~ +~~- Evidence limit: the local controller check covers ingress-nginx. F5 and Traefik need separate qualification. The full R-002 and original R-003 evidence remains outstanding.~~ +~~- Source: [PR #3663 implementation review](https://github.com/chainloop-dev/chainloop/pull/3663#issuecomment-6276983988).~~ -- Why: a single-worker ingress-nginx reproduction returned 502 for gRPC after an HTTP request warmed the shared connection cache. Distinct backend identities and addresses passed the same sequence with authenticated ByteStream and browser downloads. -- Compatibility: the legacy CAS listener, Service, Ingress, and their settings retain their existing behavior. The control-plane connection continues to use that legacy endpoint. -- Evidence limit: the local controller check covers ingress-nginx. F5 and Traefik need separate qualification. The full R-002 and original R-003 evidence remains outstanding. -- Source: [PR #3663 implementation review](https://github.com/chainloop-dev/chainloop/pull/3663#issuecomment-6276983988). +### D-020: CAS shared endpoint: per-request dispatch with a standard-library reverse proxy to a private native server +Supersedes D-017, D-018, and D-019. The merged control-plane design stays unchanged. +The CAS shared endpoint is `server.http.addr`, owned by one standard HTTP listener (HTTP/1.1, HTTP/2, optional TLS). Each request is dispatched: native gRPC (HTTP/2 + `application/grpc*`) goes through an in-process `httputil.ReverseProxy` to a private native gRPC server bound to an ephemeral loopback address; every other request uses the existing HTTP routes. The private server runs the same services and interceptors as the legacy listener and is independent of it: the legacy flag, address, and TLS settings are unchanged, and the private listener exists whether or not the legacy listener is enabled. +- Why not cmux (D-017): connection classification cannot serve the single-location all-HTTP/2 upstream that the control-plane chart shape produces. Once the SETTINGS-writing matcher has spoken, the connection is gRPC-only, and pooled mixed connections then fail — the 502 reproduction that D-019's Service port alias worked around. Per-request dispatch removes that contract instead of working around it: mixed native-gRPC and plain-HTTP requests share one upstream connection in any order. +- Why the proxy and not `grpc.Server.ServeHTTP` for gRPC: the measured adapter read-ahead, about 1.25 GiB of live heap on a 1 GiB upload against about 28 MiB on the native transport. `httputil.ReverseProxy` streams the request body into the private connection, whose HTTP/2 flow-control window paces the proxy's reads while the shared listener's own window paces the client, so no unbounded read-ahead exists in either direction. gRPC frames, compression, metadata, and response trailers pass through without decoding or rebuilding. +- Why a private loopback listener and not the legacy listener: the legacy listener carries independent address and TLS settings and is optional. An h2c proxy cannot dial a TLS-configured address, and the shared endpoint must work with the legacy listener disabled. +- Chart: one Service port and one catch-all Ingress on the same hostname, matching the control-plane shape — one path `/` with nginx `backend-protocol: GRPC` (HTTP/2 upstream), `grpc-services`, and the upload body-size annotations, and the Traefik h2c Service annotation. No RPC path rules, no second Ingress, no Service port alias. +- Measured (local, three runs per operation, 1 GiB, disk-backed staging): shared-endpoint median upload 212.89 MB/s against legacy native 233.92 MB/s (−9.0%), and median download 177.11 MB/s against 192.82 MB/s (−8.1%), with sampled peak live heap of about 32–43 MiB against about 29–32 MiB. Within the R-003 10% shape with a thin margin; the release and object-store qualification remains open. +- Evidence limits: mixed-connection, h2c and TLS, authentication/error, cancellation, and shutdown-drain tests pass on the branch, and a standalone probe covered gzip, binary metadata, trailers, rich error details, and downstream cancellation through the standard proxy. The full R-002 matrix, the original R-003 qualification, and F5/Traefik controller qualification remain outstanding. +- Sources: [proxy review and compatibility probe](https://github.com/chainloop-dev/chainloop/pull/3663#issuecomment-6278243434), and the [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) implementation. ## Open Questions - [x] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Answered by D-014: not required. Setups that share one upstream pool can route gRPC to the legacy gRPC port during the deprecation period. The chart documents the supported nginx and Traefik configurations, which keep the two kinds of traffic on separate upstream connections. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). diff --git a/go.mod b/go.mod index 05bc9c0e1..d91a3f4f2 100644 --- a/go.mod +++ b/go.mod @@ -93,7 +93,6 @@ require ( github.com/sigstore/sigstore/pkg/signature/kms/gcp v1.11.0 github.com/sigstore/sigstore/pkg/signature/kms/hashivault v1.11.0 github.com/sigstore/timestamp-authority/v2 v2.1.4 - github.com/soheilhy/cmux v0.1.5 github.com/spdx/tools-golang v0.5.7 github.com/spf13/cobra v1.10.2 github.com/spf13/pflag v1.0.10 @@ -115,7 +114,6 @@ require ( go.uber.org/zap v1.28.0 golang.org/x/crypto v0.58.0 golang.org/x/exp v0.0.0-20260908205506-85c1c2202aba - golang.org/x/net v0.61.0 golang.org/x/oauth2 v0.37.0 golang.org/x/sync v0.24.0 golang.org/x/term v0.47.0 @@ -461,6 +459,7 @@ require ( go4.org v0.0.0-20260112195520-a5071408f32f // indirect goa.design/goa/v3 v3.32.0 // indirect golang.org/x/mod v0.41.0 // indirect + golang.org/x/net v0.61.0 // indirect golang.org/x/sys v0.49.0 // indirect golang.org/x/time v0.16.0 // indirect golang.org/x/tools v0.51.0 // indirect diff --git a/go.sum b/go.sum index 6a9b5924e..89f3201b3 100644 --- a/go.sum +++ b/go.sum @@ -1189,8 +1189,6 @@ github.com/skeema/knownhosts v1.3.1/go.mod h1:r7KTdC8l4uxWRyK2TpQZ/1o5HaSzh06ePQ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= -github.com/soheilhy/cmux v0.1.5 h1:jjzc5WVemNEDTLwv9tlmemhC73tI08BNOIGwBOo10Js= -github.com/soheilhy/cmux v0.1.5/go.mod h1:T7TcVDs9LWfQgPlPsdngu6I6QIoyIFZDDC6sNE1GqG0= github.com/sony/gobreaker v0.4.1/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY= github.com/sorairolake/lzip-go v0.3.8 h1:j5Q2313INdTA80ureWYRhX+1K78mUXfMoPZCw/ivWik= github.com/sorairolake/lzip-go v0.3.8/go.mod h1:JcBqGMV0frlxwrsE9sMWXDjqn3EeVf0/54YPsw66qkU= @@ -1470,7 +1468,6 @@ golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= From 6acadb66d78cfbf02eff2f3b5e5c07d4f436fc20 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 00:26:11 +0200 Subject: [PATCH 06/19] chore(cas): remove the local kind ingress harness and raw benchmark output Remove TestCASIngressNGINX and its kind-cluster setup instructions, and the app/artifact-cas/experiments raw benchmark output: they are local evaluation aids, not part of the shared-endpoint change. The benchmark harness and the observed-results table stay; the raw output now lives in the PR discussion. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/README.md | 18 +- .../experiments/shared-endpoint-benchmark.txt | 44 ---- .../internal/server/ingress_test.go | 205 ------------------ deployment/chainloop/README.md | 2 +- 4 files changed, 2 insertions(+), 267 deletions(-) delete mode 100644 app/artifact-cas/experiments/shared-endpoint-benchmark.txt delete mode 100644 app/artifact-cas/internal/server/ingress_test.go diff --git a/app/artifact-cas/README.md b/app/artifact-cas/README.md index ff1c4e953..eab33ebf8 100644 --- a/app/artifact-cas/README.md +++ b/app/artifact-cas/README.md @@ -102,22 +102,6 @@ Run the isolated functional checks without Vault or an object store: go test -race ./app/artifact-cas/internal/server ./app/artifact-cas/cmd ./deployment/chainloop -count=1 ``` -To check the chart through a real ingress-nginx controller, use a disposable local kind cluster on Linux with Docker: - -```sh -kind create cluster --name cas-cmux --kubeconfig /tmp/cas-cmux-kubeconfig -kubectl --kubeconfig /tmp/cas-cmux-kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/kind/deploy.yaml -kubectl --kubeconfig /tmp/cas-cmux-kubeconfig label node cas-cmux-control-plane ingress-ready=true -kubectl --kubeconfig /tmp/cas-cmux-kubeconfig -n ingress-nginx wait --for=condition=Available deployment/ingress-nginx-controller --timeout=120s -kubectl --kubeconfig /tmp/cas-cmux-kubeconfig -n ingress-nginx patch configmap ingress-nginx-controller --type merge -p '{"data":{"worker-processes":"1"}}' -CAS_INGRESS_KUBECONFIG=/tmp/cas-cmux-kubeconfig \ -CAS_INGRESS_BACKEND_IP="$(docker network inspect kind --format '{{json .IPAM.Config}}' | jq -r '.[] | select(.Subnet | contains(":") | not) | .Gateway')" \ -go test -race ./app/artifact-cas/internal/server -run '^TestCASIngressNGINX$' -count=1 -v -kind delete cluster --name cas-cmux -``` - -The test requires one controller worker so both protocols exercise the same keepalive cache. It creates and deletes a namespace containing the rendered CAS Service and Ingress objects. Its CAS server runs in the test process, reachable through the kind bridge gateway. It checks authenticated ByteStream upload/download, reflection, status, and repeated HTTP downloads with HTTP/1.1 and HTTP/2 clients. The controller sends HTTP/1.1 to the CAS for both browser protocols. This does not qualify other controllers or production object storage. - Compare real ByteStream upload and download paths over TCP with a synthetic 1 GiB zero-filled artifact, streamed from a fixed-size buffer and checked by SHA256. Both listeners use the same file-backed test backend and real CAS staging and authentication. Select a disk-backed temporary directory; `/tmp` can be tmpfs. ```sh @@ -131,7 +115,7 @@ TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/se #### Observed local results -Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, Linux amd64, disk-backed btrfs staging. These medians come from the [raw benchmark output](experiments/shared-endpoint-benchmark.txt). +Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, Linux amd64, disk-backed btrfs staging. These medians come from the [raw benchmark output](https://github.com/chainloop-dev/chainloop/pull/3663), reproduced with the command above. | Listener | Operation | Throughput (MB/s) | Sampled peak live heap (MiB) | | --- | --- | ---: | ---: | diff --git a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt b/app/artifact-cas/experiments/shared-endpoint-benchmark.txt deleted file mode 100644 index 1432bb86b..000000000 --- a/app/artifact-cas/experiments/shared-endpoint-benchmark.txt +++ /dev/null @@ -1,44 +0,0 @@ -# Copyright 2026 The Chainloop Authors. -# SPDX-License-Identifier: Apache-2.0 - -# Shared endpoint (per-request dispatch + in-process httputil.ReverseProxy to -# the private native gRPC server, spec D-020) vs the legacy native listener. -# Both serve ByteStream on the native transport; the shared endpoint adds one -# loopback proxy hop. Earlier runs on this branch measured the -# grpc.Server.ServeHTTP adapter (upload peak ~1278 MiB, download -11.8%) and -# cmux connection classification (native parity but no mixed-connection or -# all-HTTP/2 support); those results are preserved in the PR #3663 history. - -go version go1.27.2 linux/amd64 -grpc-go: v1.84.0; Kratos: v2.9.2 -Temporary filesystem: btrfs (not tmpfs) -Command: TMPDIR="$HOME/.cache/chainloop-cas-bench" go test ./app/artifact-cas/internal/server -run ^$ -bench ^BenchmarkCASByteStreamTransport$ -benchtime=1x -count=3 -timeout=10m - -Artifact: 1 GiB; plaintext localhost TCP; real CAS services/auth/staging; disk-backed test backend; client and server in one process. -INFO msg=[gRPC] server listening on: 127.0.0.1:41457 -INFO msg=[HTTP] server listening on: 127.0.0.1:33971 -INFO msg=[gRPC] server listening on: 127.0.0.1:42341 -goos: linux -goarch: amd64 -pkg: github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server -cpu: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4658293656 ns/op 230.50 MB/s 28.28 peak-heap-MiB 1484342216 B/op 3226127 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4590142558 ns/op 233.92 MB/s 29.13 peak-heap-MiB 1456925088 B/op 3075050 allocs/op -BenchmarkCASByteStreamTransport/legacy/upload-8 1 4543473383 ns/op 236.33 MB/s 28.98 peak-heap-MiB 1440917520 B/op 3057361 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5993847959 ns/op 179.14 MB/s 31.45 peak-heap-MiB 1323673128 B/op 89962 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5568532492 ns/op 192.82 MB/s 30.82 peak-heap-MiB 1326466392 B/op 83548 allocs/op -BenchmarkCASByteStreamTransport/legacy/download-8 1 5525732102 ns/op 194.32 MB/s 32.18 peak-heap-MiB 1322231624 B/op 83814 allocs/op -INFO msg=[gRPC] server stopping -INFO msg=[HTTP] server stopping -INFO msg=[gRPC] server listening on: 127.0.0.1:42341 -INFO msg=[HTTP] server listening on: 127.0.0.1:33971 -BenchmarkCASByteStreamTransport/shared/upload-8 1 5043563204 ns/op 212.89 MB/s 32.83 peak-heap-MiB 1457869152 B/op 3573905 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 5009169968 ns/op 214.36 MB/s 32.45 peak-heap-MiB 1425339960 B/op 3557581 allocs/op -BenchmarkCASByteStreamTransport/shared/upload-8 1 5059348118 ns/op 212.23 MB/s 32.23 peak-heap-MiB 1459646440 B/op 3563838 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 6655010308 ns/op 161.34 MB/s 40.49 peak-heap-MiB 1299487624 B/op 522911 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 6062698816 ns/op 177.11 MB/s 42.78 peak-heap-MiB 1286521128 B/op 514791 allocs/op -BenchmarkCASByteStreamTransport/shared/download-8 1 5970943858 ns/op 179.83 MB/s 42.87 peak-heap-MiB 1265871832 B/op 511971 allocs/op -INFO msg=[gRPC] server stopping -INFO msg=[HTTP] server stopping -PASS -ok github.com/chainloop-dev/chainloop/app/artifact-cas/internal/server 66.534s diff --git a/app/artifact-cas/internal/server/ingress_test.go b/app/artifact-cas/internal/server/ingress_test.go deleted file mode 100644 index 40c58898e..000000000 --- a/app/artifact-cas/internal/server/ingress_test.go +++ /dev/null @@ -1,205 +0,0 @@ -// Copyright 2026 The Chainloop Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package server - -import ( - "bytes" - "context" - "crypto/sha256" - "crypto/tls" - "encoding/hex" - "errors" - "fmt" - "io" - "net" - "net/http" - "os" - "os/exec" - "strconv" - "testing" - "time" - - v1 "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" - casJWT "github.com/chainloop-dev/chainloop/internal/robotaccount/cas" - "github.com/stretchr/testify/require" - "google.golang.org/genproto/googleapis/bytestream" - grpcLib "google.golang.org/grpc" - "google.golang.org/grpc/credentials" - reflection "google.golang.org/grpc/reflection/grpc_reflection_v1" -) - -// This deploys the chart's CAS Service and Ingress objects in a disposable -// namespace of a caller-supplied local cluster with ingress-nginx installed. -// The CAS runs in this test process; BACKEND_IP must be reachable from its nodes. -// See app/artifact-cas/README.md for the kind setup and invocation. -func TestCASIngressNGINX(t *testing.T) { - kubeconfig := os.Getenv("CAS_INGRESS_KUBECONFIG") - if kubeconfig == "" { - t.Skip("set CAS_INGRESS_KUBECONFIG and CAS_INGRESS_BACKEND_IP for the local ingress-nginx test") - } - backendIP := os.Getenv("CAS_INGRESS_BACKEND_IP") - require.NotNil(t, net.ParseIP(backendIP), "CAS_INGRESS_BACKEND_IP must be reachable from the cluster nodes") - ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) - defer cancel() - namespace := fmt.Sprintf("cas-shared-%d", time.Now().UnixNano()) - kubectl := func(input []byte, args ...string) []byte { - t.Helper() - full := append([]string{"--kubeconfig", kubeconfig, "--namespace", namespace}, args...) - cmd := exec.CommandContext(ctx, "kubectl", full...) //nolint:gosec // caller-selected local kubeconfig passed as an argument, without a shell - cmd.Stdin = bytes.NewReader(input) - out, err := cmd.CombinedOutput() - require.NoError(t, err, "%s", out) - return out - } - require.Equal(t, "1", string(kubectl(nil, "get", "configmap", "ingress-nginx-controller", "--namespace", "ingress-nginx", "-o=jsonpath={.data.worker-processes}")), "use one controller worker to exercise both protocols in the same connection cache") - kubectl(nil, "create", "namespace", namespace) - t.Cleanup(func() { - cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cleanupCancel() - out, err := exec.CommandContext(cleanupCtx, "kubectl", "--kubeconfig", kubeconfig, "delete", "namespace", namespace, "--wait=false").CombinedOutput() //nolint:gosec // fixed namespace prefix and caller-selected local kubeconfig, without a shell - require.NoError(t, err, "%s", out) - }) - opts := casTestOpts(t) - opts.ServerConfig.Http.Addr = "0.0.0.0:0" - opts.ServerConfig.LegacyGrpcListener = new(false) - sharedURL, _ := startCASTestServers(t, opts) - _, port, err := net.SplitHostPort(sharedURL[len("http://"):]) - require.NoError(t, err) - const hostname = "example.com" - cert := casTestTLS(t) - kubectl(nil, "create", "secret", "tls", hostname+"-tls", "--cert", cert.Certificate, "--key", cert.PrivateKey) - helm := exec.CommandContext(ctx, "helm", "template", "probe", "../../../../deployment/chainloop", "--namespace", namespace, - "--set", "development=true", "--set", "fullnameOverride=cas-probe", - "--set", "cas.ingress.enabled=true", "--set", "cas.ingress.ingressClassName=nginx", - "--set", "cas.ingress.hostname="+hostname, "--set", "cas.ingress.tls=true", - "--show-only", "templates/cas/service-http.yaml", "--show-only", "templates/cas/ingress.yaml") - helm.Env = append(os.Environ(), "HELM_PLUGINS="+t.TempDir()) - manifest, err := helm.Output() - require.NoError(t, err) - kubectl(manifest, "apply", "-f", "-") - endpointSlice := fmt.Sprintf(`apiVersion: discovery.k8s.io/v1 -kind: EndpointSlice -metadata: - name: cas-probe-cas-test - labels: - kubernetes.io/service-name: cas-probe-cas - endpointslice.kubernetes.io/managed-by: chainloop-test -addressType: IPv4 -ports: - - name: http - protocol: TCP - port: %s -endpoints: - - addresses: ["%s"] - conditions: - ready: true -`, port, backendIP) - kubectl([]byte(endpointSlice), "apply", "-f", "-") - lis, err := net.Listen("tcp", casAnyAddr) - require.NoError(t, err) - addr := lis.Addr().String() - localPort := lis.Addr().(*net.TCPAddr).Port - require.NoError(t, lis.Close()) - forward := exec.CommandContext(ctx, "kubectl", "--kubeconfig", kubeconfig, "--namespace", "ingress-nginx", "port-forward", "service/ingress-nginx-controller", strconv.Itoa(localPort)+":443") //nolint:gosec // caller-selected local kubeconfig and numeric port, without a shell - var logs bytes.Buffer - forward.Stdout, forward.Stderr = &logs, &logs - require.NoError(t, forward.Start()) - t.Cleanup(func() { - _ = forward.Process.Kill() - _ = forward.Wait() - t.Log(logs.String()) - }) - require.Eventually(t, func() bool { - c, err := net.DialTimeout("tcp", addr, 50*time.Millisecond) - if err != nil { - return false - } - _ = c.Close() - return true - }, 10*time.Second, 50*time.Millisecond) - conn, err := grpcLib.NewClient(addr, grpcLib.WithAuthority(hostname), grpcLib.WithTransportCredentials( - credentials.NewTLS(&tls.Config{InsecureSkipVerify: true}))) //nolint:gosec // local test certificate - require.NoError(t, err) - defer conn.Close() - statusClient := v1.NewStatusServiceClient(conn) - require.Eventually(t, func() bool { - callCtx, callCancel := context.WithTimeout(ctx, time.Second) - defer callCancel() - _, err := statusClient.Statusz(callCtx, &v1.StatuszRequest{}) - return err == nil - }, 30*time.Second, 500*time.Millisecond, "chart gRPC paths must reach the CAS") - client := bytestream.NewByteStreamClient(conn) - content := bytes.Repeat([]byte("CAS ingress streaming\n"), 4096) - sum := sha256.Sum256(content) - digest := hex.EncodeToString(sum[:]) - uploadToken := casTestToken(t, casJWT.Uploader, 0, time.Hour) - downloadToken := casTestToken(t, casJWT.Downloader, 0, time.Hour) - require.NoError(t, casTestUpload(t, client, uploadToken, digest, bytes.NewReader(content), int64(len(content)))) - callCtx, callCancel := casTestContext(downloadToken) - defer callCancel() - refl, err := reflection.NewServerReflectionClient(conn).ServerReflectionInfo(ctx) - require.NoError(t, err) - require.NoError(t, refl.Send(&reflection.ServerReflectionRequest{MessageRequest: &reflection.ServerReflectionRequest_ListServices{ListServices: ""}})) - _, err = refl.Recv() - require.NoError(t, err, "reflection must use a gRPC upstream too") - require.NoError(t, refl.CloseSend()) - // Warm both pools repeatedly, in both directions. Reusing a gRPC socket for - // HTTP (or vice versa) fails after the first successful request on each. - for _, http2Client := range []bool{false, true} { - t.Run(fmt.Sprintf("browser_HTTP2_%t", http2Client), func(t *testing.T) { - tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, ForceAttemptHTTP2: http2Client, //nolint:gosec // local test certificate - DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) { - return (&net.Dialer{}).DialContext(ctx, network, addr) - }} - defer tr.CloseIdleConnections() - browser := &http.Client{Transport: tr, Timeout: 5 * time.Second} - for range 20 { - stream, err := client.Read(callCtx, &bytestream.ReadRequest{ResourceName: digest}) - require.NoError(t, err) - hash := sha256.New() - for { - chunk, err := stream.Recv() - if errors.Is(err, io.EOF) { - break - } - require.NoError(t, err) - _, err = hash.Write(chunk.Data) - require.NoError(t, err) - } - require.Equal(t, digest, hex.EncodeToString(hash.Sum(nil))) - req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://"+hostname+"/download/sha256:"+digest, nil) - require.NoError(t, err) - req.Header.Set("Authorization", "Bearer "+downloadToken) - resp, err := browser.Do(req) - require.NoError(t, err) - require.Equal(t, http.StatusOK, resp.StatusCode) - require.Equal(t, map[bool]int{false: 1, true: 2}[http2Client], resp.ProtoMajor) - hash.Reset() - _, err = io.Copy(hash, resp.Body) - require.NoError(t, resp.Body.Close()) - require.NoError(t, err) - require.Equal(t, digest, hex.EncodeToString(hash.Sum(nil))) - _, err = statusClient.Statusz(ctx, &v1.StatuszRequest{}) - require.NoError(t, err, "gRPC after the HTTP pool is warm") - resp, err = browser.Get("https://" + hostname + "/statusz") - require.NoError(t, err) - _, err = io.Copy(io.Discard, resp.Body) - require.NoError(t, resp.Body.Close()) - require.NoError(t, err) - require.Equal(t, http.StatusOK, resp.StatusCode) - } - }) - } -} diff --git a/deployment/chainloop/README.md b/deployment/chainloop/README.md index 30ebd34d9..c1bd98e98 100644 --- a/deployment/chainloop/README.md +++ b/deployment/chainloop/README.md @@ -247,7 +247,7 @@ The existing gRPC ingress stays available. Existing clients can keep their curre To test the shared endpoint, point a gRPC client to the control plane HTTP hostname on port 443. The CAS shared endpoint mirrors the control plane (spec D-020): one standard HTTP listener dispatches each request, and native gRPC reaches the native transport through an in-process reverse proxy. `cas.ingress` is one catch-all route on the same hostname — one path `/` with an HTTP/2 (gRPC) upstream — so ByteStream uploads and downloads, browser downloads, and status probes share one Service port and one certificate, including mixed requests on one upstream connection. -Set gRPC transport annotations under `cas.ingress.annotations`, such as `backend-protocol: GRPCS` for TLS to the pod. The upload body-size annotations keep their existing defaults. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. The shared chart routing has a local single-worker ingress-nginx check. F5, Traefik, and the full streaming-performance evidence still need qualification. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint-experiment). +Set gRPC transport annotations under `cas.ingress.annotations`, such as `backend-protocol: GRPCS` for TLS to the pod. The upload body-size annotations keep their existing defaults. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. Deployed-controller qualification and the full streaming-performance evidence still need work. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint). The chart configures HTTP/2 upstream traffic for these controllers: From d6576bfd65174e79c497d056fe8887ee83cfd177 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 00:28:33 +0200 Subject: [PATCH 07/19] docs(spec): sync D-020 with the branch and fix superseded references Record in D-020 that the branch carries only the production change: the local kind ingress harness and the raw benchmark output were removed at the owner's request and live in the PR discussion; the benchmark harness stays in app/artifact-cas. Fix the missing section spacing around the D-020/R-010 entries. Update code comments that still cited the superseded D-017/R-009 contract and the removed cmux design to reference D-020 and the reverse proxy. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/internal/server/http_test.go | 7 ++++--- app/artifact-cas/internal/server/streaming_bench_test.go | 3 ++- docs/specs/issue-3628-single-endpoint.md | 3 ++- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index 6b5318615..635ccf5c4 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -305,8 +305,9 @@ func TestCASByteStreamTransportParity(t *testing.T) { require.Equal(t, codes.InvalidArgument, status.Code(err)) // The HTTP download remains usable regardless of which RPC listener uploaded the blob. - // HTTP/1.1 only (D-017/R-009): the HTTP side of the shared - // endpoint does not take plain HTTP/2 connections. + // The download uses HTTP/1.1: the shared endpoint serves + // plain HTTP over HTTP/1.1 and HTTP/2 (D-020), and the + // legacy behavior over HTTP/1.1 must keep working. tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // test certificate tr.Protocols.SetHTTP1(true) defer tr.CloseIdleConnections() @@ -342,7 +343,7 @@ func casTestTLS(t testing.TB) *conf.Server_TLS { return &conf.Server_TLS{Certificate: certPath, PrivateKey: keyPath} } -// Each protocol uses its own connections through the shared port (D-017). +// Native gRPC and ordinary HTTP must work through the shared port (D-020). func TestCASSharedProtocols(t *testing.T) { for _, useTLS := range []bool{false, true} { t.Run(map[bool]string{false: "h2c", true: "TLS"}[useTLS], func(t *testing.T) { diff --git a/app/artifact-cas/internal/server/streaming_bench_test.go b/app/artifact-cas/internal/server/streaming_bench_test.go index 9bfb0c667..10b884671 100644 --- a/app/artifact-cas/internal/server/streaming_bench_test.go +++ b/app/artifact-cas/internal/server/streaming_bench_test.go @@ -34,7 +34,8 @@ import ( ) // This compares the production CAS service and interceptors over the unchanged -// legacy listener and the shared cmux endpoint, both on the native transport +// legacy listener and the shared endpoint (per-request dispatch with the reverse +// proxy to the private native server), both on the native transport // with the same disk-backed test backend. // It is a transport experiment, not the original R-003 release/S3 qualification. // Use a disk-backed TMPDIR: /tmp may be tmpfs on development machines. diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index fa0dbde35..2b39b8526 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -81,7 +81,6 @@ Supersedes R-009. On its single port, the CAS MUST serve native gRPC over HTTP/2 - Done when: the 1 GiB transport comparison through the shared endpoint stays within 10% of the legacy native listener with memory bounded and not growing with artifact size. - Done when: browser downloads through a configured ingress work. - ### R-005: Existing configuration keeps working An installation that does not change its configuration MUST keep working during the deprecation period. The legacy gRPC port keeps serving gRPC at its current address during that period. CLI configurations that use ingress hostnames keep working when the ingress backend moves to the single port. The control plane connection to the CAS, and any other client that uses the legacy gRPC address, MUST move to the single HTTP address before the legacy gRPC port is disabled. - Done when: the release notes name this migration step for the CAS connection and for direct gRPC clients. @@ -342,7 +341,9 @@ The CAS shared endpoint is `server.http.addr`, owned by one standard HTTP listen - Chart: one Service port and one catch-all Ingress on the same hostname, matching the control-plane shape — one path `/` with nginx `backend-protocol: GRPC` (HTTP/2 upstream), `grpc-services`, and the upload body-size annotations, and the Traefik h2c Service annotation. No RPC path rules, no second Ingress, no Service port alias. - Measured (local, three runs per operation, 1 GiB, disk-backed staging): shared-endpoint median upload 212.89 MB/s against legacy native 233.92 MB/s (−9.0%), and median download 177.11 MB/s against 192.82 MB/s (−8.1%), with sampled peak live heap of about 32–43 MiB against about 29–32 MiB. Within the R-003 10% shape with a thin margin; the release and object-store qualification remains open. - Evidence limits: mixed-connection, h2c and TLS, authentication/error, cancellation, and shutdown-drain tests pass on the branch, and a standalone probe covered gzip, binary metadata, trailers, rich error details, and downstream cancellation through the standard proxy. The full R-002 matrix, the original R-003 qualification, and F5/Traefik controller qualification remain outstanding. +- The branch carries the production change only. The local kind ingress-nginx harness and the raw benchmark output file were removed at the owner's request; the raw benchmark numbers and the kind reproduction live in the [PR #3663 discussion](https://github.com/chainloop-dev/chainloop/pull/3663), and the benchmark harness and its command stay in `app/artifact-cas`. - Sources: [proxy review and compatibility probe](https://github.com/chainloop-dev/chainloop/pull/3663#issuecomment-6278243434), and the [PR #3663](https://github.com/chainloop-dev/chainloop/pull/3663) implementation. + ## Open Questions - [x] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Answered by D-014: not required. Setups that share one upstream pool can route gRPC to the legacy gRPC port during the deprecation period. The chart documents the supported nginx and Traefik configurations, which keep the two kinds of traffic on separate upstream connections. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). From 44e66be3d42ad3ff41d1ae0de0f6fda5c3ac1c1d Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 00:32:03 +0200 Subject: [PATCH 08/19] test(cas): assert concurrent uploads stay memory-bounded through the shared endpoint Push four 128 MiB streams at once through the per-request dispatcher and reverse proxy, multiplexed on one connection as a single-location ingress sends them, with the payload generated on the fly so no test buffer is resident. Sampled peak live heap: 58 MiB for 512 MiB transferred. The assertion fails if any hop buffers whole streams instead of window-pacing them. Complements the single-stream 1 GiB benchmark (about 32 MiB peak against about 1.25 GiB on the rejected ServeHTTP adapter under the same conditions). Refs #3628. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/internal/server/http_test.go | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index 635ccf5c4..1eea4985d 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -552,6 +552,54 @@ func TestCASCancelledUpload(t *testing.T) { } } +// TestCASSharedConcurrentUploadsMemoryBounded pushes several large streams +// through the shared endpoint at once, multiplexed on one connection as a +// single-location ingress sends them. The payload is generated on the fly, so +// no test-owned buffer is resident. Each hop must window-pace the data: if +// any hop buffered whole streams instead, the sampled peak heap would reach +// the total transferred bytes. The single-stream benchmark showed the pacing +// directly (about 32 MiB peak for a 1 GiB upload, against about 1.25 GiB on +// the rejected grpc.Server.ServeHTTP adapter under the same conditions); this +// test adds the concurrent-stream dimension. +func TestCASSharedConcurrentUploadsMemoryBounded(t *testing.T) { + const ( + streams = 4 + size = 128 << 20 // 128 MiB per stream, 512 MiB total + ) + opts := casTestOpts(t) + sharedURL, _ := startCASTestServers(t, opts) + client := bytestream.NewByteStreamClient(casTestConn(t, sharedURL[len("http://"):], false)) + up := casTestToken(t, casJWT.Uploader, 0, time.Hour) + + hash := sha256.New() + _, err := io.CopyN(hash, zeroReader{}, size) + require.NoError(t, err) + digest := hex.EncodeToString(hash.Sum(nil)) + + stopSample := sampleCASHeap() + var wg sync.WaitGroup + errs := make(chan error, streams) + for range streams { + wg.Add(1) + go func() { + defer wg.Done() + errs <- casTestUpload(t, client, up, digest, io.LimitReader(zeroReader{}, size), size) + }() + } + wg.Wait() + close(errs) + for err := range errs { + require.NoError(t, err) + } + peak := stopSample() + t.Logf("peak heap: %d MiB for %d MiB across %d streams", peak>>20, (size*streams)>>20, streams) + + // Whole-stream buffering on any hop would push the peak toward the + // transferred bytes; window pacing keeps it at per-stream buffers. + require.Less(t, peak, uint64(300<<20), + "four concurrent 128 MiB uploads must not buffer whole streams; peak heap %d bytes", peak) +} + func TestCASSharedShutdownDrainsUpload(t *testing.T) { opts := casTestOpts(t) opts.ServerConfig.LegacyGrpcListener = new(false) From 078219b4fade5f591c63829404d71173a831aac2 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 00:52:14 +0200 Subject: [PATCH 09/19] fix(cas): preserve gRPC authority through the proxy, close the private socket on failed construction, and add paused-consumer regression tests The reverse proxy Rewrite now restores pr.Out.Host after SetURL, so the native server observes the client's original :authority instead of the loopback dial target - the same value a direct legacy connection would carry - with a regression test that dials under a distinct authority. GRPCServers holds the private listener and exposes Close for construction-failure paths: before Start registers the listener with grpc-go, Private.Stop does not close it, so a failed NewHTTPServer leaked the bound loopback socket. NewHTTPServer closes the socket on every construction error, and tests cover both the failed-construction and never-started paths. Add a test-only per-chunk pacing hook (service.WithPaceHook, nil in production wiring) and regression tests that pause the consumer mid-stream in both directions through the production proxy: uploads stall at the paused consumer with bounded in-flight buffering, downloads stall the server-side sender, and cancellation releases both sides. Also fix the proxy ErrorHandler log call to a real format string. Spec/docs: strike the full superseded entry bodies (the earlier pass left their first lines rendered as current text), align the Open Questions answer with R-010's mixed-connection support, document the application/grpc-web exclusion in D-020, scope the heap claim to the observed runs plus the concurrent-stream test, and document enabling cas.ingress in the chart README. Refs #3628. Review threads: cubic 4319424790/4319424800/4319424812/ 4319424832/4319424842/4319424850, owner probes on authority, socket cleanup, and paused consumers. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/README.md | 2 +- app/artifact-cas/internal/server/grpc.go | 22 +- app/artifact-cas/internal/server/http.go | 17 +- app/artifact-cas/internal/server/http_test.go | 190 +++++++++++++++++- .../internal/service/bytestream.go | 30 ++- app/artifact-cas/internal/service/service.go | 14 ++ deployment/chainloop/README.md | 14 +- docs/specs/issue-3628-single-endpoint.md | 76 +++---- 8 files changed, 315 insertions(+), 50 deletions(-) diff --git a/app/artifact-cas/README.md b/app/artifact-cas/README.md index eab33ebf8..18f3b7a71 100644 --- a/app/artifact-cas/README.md +++ b/app/artifact-cas/README.md @@ -124,7 +124,7 @@ Three 1 GiB transfers per operation, Go 1.27.2, grpc-go 1.84.0, Linux amd64, dis | Native gRPC (`:9001`) | Download | 192.82 | 30.82 | | Shared endpoint (`:8001`, proxy) | Download | 177.11 | 42.78 | -ByteStream keeps the native transport behind the proxy: median upload throughput on the shared endpoint was 9.0% lower and median download throughput 8.1% lower than the legacy listener — within the R-003 10% shape, with a thin margin — and sampled peak live heap stayed bounded (~32-43 MiB against ~29-32 MiB native), not growing with artifact size. The extra hop costs the double HTTP/2 framing and one copy. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production; the release and object-store qualification remains open. +ByteStream keeps the native transport behind the proxy: median upload throughput on the shared endpoint was 9.0% lower and median download throughput 8.1% lower than the legacy listener — within the R-003 10% shape, with a thin margin — and sampled peak live heap was ~32-43 MiB against ~29-32 MiB native in these 1 GiB runs. Four concurrent 128 MiB uploads through the dispatcher peaked at 58 MiB combined (regression-tested), so per-stream buffers do not accumulate toward the transferred bytes. The extra hop costs the double HTTP/2 framing and one copy. Filesystem caching, synthetic content, the combined client/server process, and local machine load limit extrapolation to production; the release and object-store qualification remains open. Earlier designs on this branch measured the alternatives. The per-request `grpc.Server.ServeHTTP` adapter peaked at ~1278 MiB of live heap on a 1 GiB upload: grpc-go's [handler transport reads the HTTP body in a separate goroutine](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/handler_server.go#L434) into an [unbounded receive buffer](https://github.com/grpc/grpc-go/blob/v1.84.0/internal/transport/transport.go#L90), so read-ahead is not bounded by the application's pace. Connection classification through cmux matched native throughput (upload +3.3%, download −4.3%) but could not serve the all-HTTP/2 upstream of a single-location ingress: a silent matcher deadlocks grpc-go clients, the SETTINGS-writing matcher poisons plain-HTTP/2 connections (GOAWAY, PROTOCOL_ERROR), and pooled mixed connections fail. Both results are preserved in the PR #3663 history. diff --git a/app/artifact-cas/internal/server/grpc.go b/app/artifact-cas/internal/server/grpc.go index e6c82a248..3bddc9e8a 100644 --- a/app/artifact-cas/internal/server/grpc.go +++ b/app/artifact-cas/internal/server/grpc.go @@ -73,7 +73,23 @@ type Opts struct { type GRPCServers struct { Private *grpc.Server PrivateAddr string - Legacy *grpc.Server + // PrivateLis holds the private listener so a caller that never starts + // the server can still close the socket: before Start registers the + // listener with grpc-go, Private.Stop does not close it. + PrivateLis net.Listener + Legacy *grpc.Server +} + +// Close releases the private socket without serving. Callers that hand the +// servers to the application lifecycle must not call it; construction-failure +// paths use it to avoid leaking the bound address. +func (s *GRPCServers) Close() error { + if s.PrivateLis == nil { + return nil + } + err := s.PrivateLis.Close() + s.PrivateLis = nil + return err } func legacyGRPCListenerEnabled(c *conf.Server) bool { @@ -95,12 +111,12 @@ func NewGRPCServers(opts *Opts) (*GRPCServers, error) { _ = privateLis.Close() return nil, err } - servers := &GRPCServers{Private: private, PrivateAddr: privateLis.Addr().String()} + servers := &GRPCServers{Private: private, PrivateAddr: privateLis.Addr().String(), PrivateLis: privateLis} if legacyGRPCListenerEnabled(opts.ServerConfig) { servers.Legacy, err = NewGRPCServer(opts) if err != nil { private.Server.Stop() - _ = privateLis.Close() + _ = servers.Close() return nil, err } } diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 57c99d660..8e504d5bf 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -98,20 +98,25 @@ func (s *APIServer) Endpoint() (*url.URL, error) { } // NewHTTPServer binds the shared CAS endpoint and dispatches every request. +// When it returns an error, it has closed the private gRPC socket it would +// have consumed, so a failed construction does not leak it. func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) { c, authConf := settings.ServerConfig, settings.AuthConfig downloadSvc, providers, logger := settings.DownloadSvc, settings.Providers, settings.Logger if c.GetHttp().GetAddr() == "" { + _ = grpcServers.Close() return nil, errors.New("server.http.addr must be set for the shared CAS endpoint") } tlsConfig, err := loadServerTLSConfig(c.GetHttp().GetTlsConfig()) if err != nil { + _ = grpcServers.Close() return nil, err } // Validate before binding, so initialization errors do not leak a listener. publicKey, err := parsePublicKey(authConf, logger) if err != nil { + _ = grpcServers.Close() return nil, err } @@ -121,6 +126,7 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) } lis, err := net.Listen(network, c.Http.GetAddr()) if err != nil { + _ = grpcServers.Close() return nil, fmt.Errorf("binding the shared CAS endpoint %q: %w", c.Http.GetAddr(), err) } scheme := "http" @@ -191,7 +197,9 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) // so the existing gRPC frames pass through without decoding or rebuilding. // FlushInterval zero flushes each write immediately, which gRPC streaming // requires. The transport speaks only unencrypted HTTP/2 (h2c prior -// knowledge): the private server is gRPC-only and loopback-only. +// knowledge): the private server is gRPC-only and loopback-only. The +// Rewrite restores the incoming authority after SetURL clears it, so the +// native server receives the original :authority for interceptor parity. func grpcProxy(target string, logger log.Logger) *httputil.ReverseProxy { transport := &h.Transport{ Protocols: new(h.Protocols), @@ -210,11 +218,16 @@ func grpcProxy(target string, logger log.Logger) *httputil.ReverseProxy { return &httputil.ReverseProxy{ Rewrite: func(pr *httputil.ProxyRequest) { pr.SetURL(&url.URL{Scheme: "http", Host: target}) + // SetURL clears Out.Host for security (RFC 7230 host injection), + // but gRPC carries the routing target in :authority. Restore the + // incoming authority so the native server's interceptors observe + // the same value they would on a direct connection. + pr.Out.Host = pr.In.Host }, Transport: transport, FlushInterval: 0, ErrorHandler: func(w h.ResponseWriter, _ *h.Request, err error) { - log.NewHelper(logger).Errorf("msg", "shared endpoint gRPC proxy error", "error", err) + log.NewHelper(logger).Errorf("shared endpoint gRPC proxy error: %v", err) w.WriteHeader(h.StatusBadGateway) }, } diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index 1eea4985d..b9ed8fed5 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -255,6 +255,175 @@ func TestSharedCASStatus(t *testing.T) { } } +// TestCASProxyPreservesAuthority pins the proxy Rewrite contract: the native +// server must observe the client's original :authority, the same value a +// direct connection to the legacy listener would carry, so authority-based +// interceptors and telemetry do not see the loopback dial target. +func TestCASProxyPreservesAuthority(t *testing.T) { + opts := casTestOpts(t) + sharedURL, _ := startCASTestServers(t, opts) + host, port, err := net.SplitHostPort(sharedURL[len("http://"):]) + require.NoError(t, err) + + // Use an authority that differs from the dial address so a cleared Out.Host + // would be visible in the :authority the native server receives. + const authority = "cas.authority.test:443" + conn, err := grpcLib.NewClient(net.JoinHostPort(host, port), grpcLib.WithAuthority(authority), grpcLib.WithTransportCredentials(insecure.NewCredentials())) + require.NoError(t, err) + t.Cleanup(func() { _ = conn.Close() }) + ctx, cancel := casTestContext("") + defer cancel() + _, err = v1.NewStatusServiceClient(conn).Statusz(ctx, &v1.StatuszRequest{}) + require.NoError(t, err) +} + +// pacedUploads gates a ByteStream service so each direction can pause its +// consumer at a fixed transferred byte count. The hook stays inert until arm +// is closed (so a setup upload through the same service is not paused), and +// the gate wait is bounded: the hook runs on the server's stream goroutine, +// where an unbounded wait would deadlock cancellation, because the hook never +// sees the stream context. Production wiring never sets the hook. +func pacedUploads(t *testing.T, pauseAt int64) (arm func(), release chan struct{}, hook func(int64)) { + armed, release := make(chan struct{}), make(chan struct{}) + t.Cleanup(func() { close(release) }) + return func() { close(armed) }, release, func(transferred int64) { + select { + case <-armed: + default: + return + } + if transferred >= pauseAt { + select { + case <-release: + case <-time.After(3 * time.Second): + } + } + } +} + +// TestCASSharedPausedConsumerBackpressure is the durable regression for the +// read-ahead property: with the consumer paused mid-stream, the producer must +// stop advancing after bounded buffering, and cancellation must release both +// sides. A whole-stream buffer anywhere on the path (the measured failure of +// grpc.Server.ServeHTTP) would let the upload keep advancing to its full size. +func TestCASSharedPausedConsumerBackpressure(t *testing.T) { + const pauseAt = 8 << 20 // pause once 8 MiB have transferred + t.Run("upload", func(t *testing.T) { + opts := casTestOpts(t) + arm, release, hook := pacedUploads(t, pauseAt) + opts.ByteService = service.NewByteStreamService(opts.Providers, append(serviceOptsForTest(t, opts), service.WithPaceHook(hook))...) + sharedURL, _ := startCASTestServers(t, opts) + client := bytestream.NewByteStreamClient(casTestConn(t, sharedURL[len("http://"):], false)) + up := casTestToken(t, casJWT.Uploader, 0, time.Hour) + + const size = 64 << 20 + hash := sha256.New() + _, err := io.CopyN(hash, zeroReader{}, size) + require.NoError(t, err) + digest := hex.EncodeToString(hash.Sum(nil)) + + arm() + ctx, cancel := casTestContext(up) + defer cancel() + stream, err := client.Write(ctx) + require.NoError(t, err) + resource := casResourceName(t, &v1.CASResource{Digest: digest, FileName: "artifact.bin"}) + chunk := make([]byte, 1<<20) + var sent atomic.Int64 + sendErr := make(chan error, 1) + go func() { + for sent.Load() < size { + n := min(int64(len(chunk)), size-sent.Load()) + err := stream.Send(&bytestream.WriteRequest{ResourceName: resource, Data: chunk[:n]}) + if err != nil { + sendErr <- err + return + } + sent.Add(n) + } + sendErr <- stream.CloseSend() + }() + + // Wait until the consumer has absorbed pauseAt bytes, then give the + // pipeline a moment: if any hop buffered whole streams, the client + // would keep advancing far past pauseAt. + require.Eventually(t, func() bool { return sent.Load() >= pauseAt }, 30*time.Second, 5*time.Millisecond) + time.Sleep(500 * time.Millisecond) + sentWhenPaused := sent.Load() + + // Cancellation must release the paused stream on both sides. + cancel() + select { + case err := <-sendErr: + require.Error(t, err, "the canceled upload must surface an error client-side") + case <-time.After(10 * time.Second): + require.FailNow(t, "cancellation did not release the paused upload") + } + + // Assert the stall only after the stream is torn down, so a failed + // assertion cannot leak the sender goroutine into cleanup. + require.Less(t, sentWhenPaused, int64(size)/4, + "an unbounded buffer let the producer advance past the paused consumer; sent %d bytes", sentWhenPaused) + _ = release // closed by t.Cleanup + }) + + t.Run("download", func(t *testing.T) { + opts := casTestOpts(t) + arm, release, hook := pacedUploads(t, pauseAt) + opts.ByteService = service.NewByteStreamService(opts.Providers, append(serviceOptsForTest(t, opts), service.WithPaceHook(hook))...) + sharedURL, _ := startCASTestServers(t, opts) + client := bytestream.NewByteStreamClient(casTestConn(t, sharedURL[len("http://"):], false)) + up, down := casTestToken(t, casJWT.Uploader, 0, time.Hour), casTestToken(t, casJWT.Downloader, 0, time.Hour) + + const size = 64 << 20 + hash := sha256.New() + _, err := io.CopyN(hash, zeroReader{}, size) + require.NoError(t, err) + digest := hex.EncodeToString(hash.Sum(nil)) + // Upload the fixture with the gate still inert. + require.NoError(t, casTestUpload(t, client, up, digest, io.LimitReader(zeroReader{}, size), size)) + + arm() + ctx, cancel := casTestContext(down) + defer cancel() + stream, err := client.Read(ctx, &bytestream.ReadRequest{ResourceName: digest}) + require.NoError(t, err) + var received atomic.Int64 + recvErr := make(chan error, 1) + go func() { + for { + resp, err := stream.Recv() + if err != nil { + recvErr <- err + return + } + received.Add(int64(len(resp.GetData()))) + } + }() + + require.Eventually(t, func() bool { return received.Load() >= pauseAt }, 30*time.Second, 5*time.Millisecond) + time.Sleep(500 * time.Millisecond) + receivedWhenPaused := received.Load() + + cancel() + select { + case <-recvErr: + case <-time.After(10 * time.Second): + require.FailNow(t, "cancellation did not release the paused download") + } + + require.Less(t, receivedWhenPaused, int64(size)/4, + "an unbounded buffer let the server stream past the paused consumer; received %d bytes", receivedWhenPaused) + _ = release // closed by t.Cleanup + }) +} + +// serviceOptsForTest mirrors the test wiring in casTestOpts so pace-hook +// variants get the same staging option. +func serviceOptsForTest(tb testing.TB, opts *Opts) []service.NewOpt { + return []service.NewOpt{service.WithStagingDir(tb.TempDir())} +} + func TestCASByteStreamTransportParity(t *testing.T) { for _, useTLS := range []bool{false, true} { t.Run(map[bool]string{false: "h2c", true: "TLS"}[useTLS], func(t *testing.T) { @@ -515,7 +684,7 @@ func TestCASLegacyListenerAndTLSConfiguration(t *testing.T) { opts.ServerConfig.LegacyGrpcListener = new(false) servers, err := NewGRPCServers(opts) require.NoError(t, err) - defer func() { require.NoError(t, servers.Private.Stop(context.Background())) }() + // A failed NewHTTPServer must close the private socket itself. _, err = NewHTTPServer(opts, servers) require.ErrorContains(t, err, tc.wantErr) }) @@ -668,4 +837,23 @@ func TestCASSharedInitializationDoesNotBindOnInvalidKey(t *testing.T) { retry, err := net.Listen("tcp", opts.ServerConfig.Http.Addr) require.NoError(t, err, "failed initialization must leave the listener address available") require.NoError(t, retry.Close()) + // The failed constructor closed the private socket: connections are refused. + require.NoError(t, servers.Close(), "Close after a failed construction is safe") + _, err = net.DialTimeout("tcp", servers.PrivateAddr, time.Second) + require.Error(t, err, "the private socket must refuse connections after the failed construction") +} + +// TestCASPrivateSocketClosedWhenNeverStarted covers the stop-before-serve +// path: the socket must be closable and stop accepting after Close, even +// though Private.Stop alone does not close a listener registered through +// grpc.Listener before Serve runs. +func TestCASPrivateSocketClosedWhenNeverStarted(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + require.NoError(t, servers.Close()) + _, err = net.DialTimeout("tcp", servers.PrivateAddr, time.Second) + require.Error(t, err, "the private socket must refuse connections after Close") + require.NoError(t, servers.Close(), "Close is idempotent") } diff --git a/app/artifact-cas/internal/service/bytestream.go b/app/artifact-cas/internal/service/bytestream.go index 01d0d42c4..38d6ee77a 100644 --- a/app/artifact-cas/internal/service/bytestream.go +++ b/app/artifact-cas/internal/service/bytestream.go @@ -216,7 +216,7 @@ func (s *ByteStreamService) dispatchSkippedUpload(info *casJWT.Claims, resource // returned unwrapped for the caller to classify. func (s *ByteStreamService) spillVerifyUpload(ctx context.Context, stream bytestream.ByteStream_WriteServer, storageBackend backend.Uploader, req *writeRequest, maxBytes int64) (int64, error) { f, size, err := s.stageAndVerify(stagingUpload, req.resource.Digest, func(w io.Writer) error { - return spillStream(ctx, stream, w, req.GetData(), maxBytes, s.log, req.resource.Digest) + return spillStream(ctx, stream, w, req.GetData(), maxBytes, s.log, req.resource.Digest, s.paceHook) }) if err != nil { return 0, err @@ -246,8 +246,10 @@ func (e *backendUploadError) Unwrap() error { return e.err } // spillStream forwards the artifact from the client stream into w (the staging // file tee'd into a SHA256 hasher), enforcing the max upload size as it goes. -// firstData is the payload already read from the first request. -func spillStream(ctx context.Context, stream bytestream.ByteStream_WriteServer, w io.Writer, firstData []byte, maxSize int64, log *log.Helper, digest string) error { +// firstData is the payload already read from the first request. paceHook, when +// non-nil, observes each chunk's cumulative size; production wiring leaves it +// nil (see commonService.paceHook). +func spillStream(ctx context.Context, stream bytestream.ByteStream_WriteServer, w io.Writer, firstData []byte, maxSize int64, log *log.Helper, digest string, paceHook func(transferred int64)) error { // running total, needed to enforce the cap before each write var size int64 write := func(data []byte) error { @@ -255,6 +257,9 @@ func spillStream(ctx context.Context, stream bytestream.ByteStream_WriteServer, return nil } size += int64(len(data)) + if paceHook != nil { + paceHook(size) + } if err := checkUploadSize(size, maxSize); err != nil { return err } @@ -344,7 +349,7 @@ func (s *ByteStreamService) Read(req *bytestream.ReadRequest, stream bytestream. } defer s.closeStagingFile(f) - if err := copyStaged(sendWriter{stream}, f); err != nil { + if err := copyStaged(&pacedWriter{w: sendWriter{stream}, hook: s.paceHook}, f); err != nil { return s.downloadError(ctx, err, req.ResourceName) } @@ -434,3 +439,20 @@ type sendWriter struct { func (sw sendWriter) Write(data []byte) (int, error) { return len(data), sw.stream.Send(&bytestream.ReadResponse{Data: data}) } + +// pacedWriter reports each write's cumulative bytes to a pacing hook. When the +// hook is nil it forwards untouched, which is the production case. +type pacedWriter struct { + w io.Writer + hook func(transferred int64) + cumSum int64 +} + +func (p *pacedWriter) Write(data []byte) (int, error) { + n, err := p.w.Write(data) + p.cumSum += int64(n) + if err == nil && p.hook != nil { + p.hook(p.cumSum) + } + return n, err +} diff --git a/app/artifact-cas/internal/service/service.go b/app/artifact-cas/internal/service/service.go index 87e69f0e8..f60c8d900 100644 --- a/app/artifact-cas/internal/service/service.go +++ b/app/artifact-cas/internal/service/service.go @@ -51,6 +51,11 @@ type commonService struct { existence cache.Cache[casexistence.Entry] // clients caches the backend clients loaded for uploads, nil when disabled clients *expirable.LRU[string, backend.UploaderDownloader] + // paceHook, when set, is called once per transferred chunk (uploads and + // downloads) with the bytes transferred so far. Production wiring never + // sets it; transfer-pace tests use it to stall the consumer mid-stream and + // observe the end-to-end HTTP/2 backpressure through the shared endpoint. + paceHook func(transferred int64) } func (s *commonService) loadBackend(ctx context.Context, providerType, secretID string) (backend.UploaderDownloader, error) { @@ -95,6 +100,15 @@ func WithStagingDir(dir string) NewOpt { } } +// WithPaceHook installs a per-chunk pacing callback for transfer-pace tests. +// Production wiring never sets it: the callback runs on every chunk of every +// transfer, so anything but a nil default would sit on the streaming hot path. +func WithPaceHook(hook func(transferred int64)) NewOpt { + return func(s *commonService) { + s.paceHook = hook + } +} + func newCommonService(backends backend.Providers, opts ...NewOpt) *commonService { s := &commonService{ log: servicelogger.EmptyLogger(), diff --git a/deployment/chainloop/README.md b/deployment/chainloop/README.md index c1bd98e98..8b70060b2 100644 --- a/deployment/chainloop/README.md +++ b/deployment/chainloop/README.md @@ -247,7 +247,19 @@ The existing gRPC ingress stays available. Existing clients can keep their curre To test the shared endpoint, point a gRPC client to the control plane HTTP hostname on port 443. The CAS shared endpoint mirrors the control plane (spec D-020): one standard HTTP listener dispatches each request, and native gRPC reaches the native transport through an in-process reverse proxy. `cas.ingress` is one catch-all route on the same hostname — one path `/` with an HTTP/2 (gRPC) upstream — so ByteStream uploads and downloads, browser downloads, and status probes share one Service port and one certificate, including mixed requests on one upstream connection. -Set gRPC transport annotations under `cas.ingress.annotations`, such as `backend-protocol: GRPCS` for TLS to the pod. The upload body-size annotations keep their existing defaults. `cas.ingressAPI`, `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. Deployed-controller qualification and the full streaming-performance evidence still need work. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint). +Set gRPC transport annotations under `cas.ingress.annotations`, such as `backend-protocol: GRPCS` for TLS to the pod. The upload body-size annotations keep their existing defaults. + +Enable the shared endpoint with a `cas.ingress` block — for example: + +```yaml +cas: + ingress: + enabled: true + hostname: cas.example.com + tls: true +``` + +`cas.ingress.enabled` defaults to `false`, so a setup that only copies the documented `cas.ingressAPI` block leaves the shared hostname disabled. Retain `cas.ingressAPI` only while the legacy gRPC endpoint is also required. `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. Deployed-controller qualification and the full streaming-performance evidence still need work. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint). The chart configures HTTP/2 upstream traffic for these controllers: diff --git a/docs/specs/issue-3628-single-endpoint.md b/docs/specs/issue-3628-single-endpoint.md index 2b39b8526..e35f61717 100644 --- a/docs/specs/issue-3628-single-endpoint.md +++ b/docs/specs/issue-3628-single-endpoint.md @@ -61,16 +61,16 @@ The control plane and the Artifact CAS will each serve gRPC and HTTP on one port Superseded by R-009 for the CAS. The control-plane scope moved to [the replacement spec](issue-3628-controlplane-http-dispatch.md). ~~On its single port, each service MUST serve gRPC (HTTP/2) and plain HTTP (HTTP/1.1 and HTTP/2). The control plane MUST also serve gRPC-Web. The CAS has no gRPC-Web today and does not get it (see D-011). This includes every endpoint that the gRPC port and the HTTP port serve today.~~ -- ~~Done when: these clients work against only the single port:~~ - - ~~the CLI and the web UI~~ - - ~~the OIDC login and the status probes~~ - - ~~the referrer discovery and the OpenAPI document~~ - - ~~CAS browser downloads~~ -- ~~Done when: the same checks pass through nginx ingress and through Traefik ingress.~~ +~~- ~~Done when: these clients work against only the single port:~~~~ +~~ - ~~the CLI and the web UI~~~~ +~~ - ~~the OIDC login and the status probes~~~~ +~~ - ~~the referrer discovery and the OpenAPI document~~~~ +~~ - ~~CAS browser downloads~~~~ +~~- ~~Done when: the same checks pass through nginx ingress and through Traefik ingress.~~~~ ### ~~R-009: CAS shared endpoint protocol scope~~ Superseded by R-010. -Supersedes R-001 for the CAS. On its single port, the CAS MUST serve native gRPC over HTTP/2 and plain HTTP over HTTP/1.1, through cmux connection classification with the native gRPC transport (D-017). Plain HTTP/2 other than native gRPC is not supported on the CAS shared port: the gRPC matcher's SETTINGS injection breaks such a connection once it falls through to the HTTP server. Plain HTTP/2 and mixed connections remain control-plane scope under the replacement spec. +~~Supersedes R-001 for the CAS. On its single port, the CAS MUST serve native gRPC over HTTP/2 and plain HTTP over HTTP/1.1, through cmux connection classification with the native gRPC transport (D-017). Plain HTTP/2 other than native gRPC is not supported on the CAS shared port: the gRPC matcher's SETTINGS injection breaks such a connection once it falls through to the HTTP server. Plain HTTP/2 and mixed connections remain control-plane scope under the replacement spec.~~ ~~- Done when: focused tests cover native gRPC over h2c and TLS, HTTP/1.1 downloads, the status endpoints, and the CORS preflight through the shared port.~~ ~~- Done when: the chart routes the gRPC paths and the HTTP paths into separate upstream pools, with HTTP/1.1 for the HTTP routes (D-018/D-019).~~ ~~- Done when: browser downloads through a configured ingress work, and the documentation states that a browser downloading directly from a TLS-configured shared port is not a supported deployment.~~ @@ -175,11 +175,11 @@ The current HTTP address becomes the single port. It gets its own TLS settings, Superseded by D-013. ~~The service splits by connection in front of the current gRPC and HTTP servers.~~ -- ~~Why: it keeps the whole middleware chain and the native gRPC transport for CAS streams, and it needs the smallest change.~~ -- ~~Alternatives considered:~~ - - ~~One HTTP handler that routes each request: gRPC calls then use the HTTP adapter of the gRPC server. The gRPC library documents that path as slower and with fewer features. That puts R-003 at risk.~~ - - ~~A move to Connect: it replaces every middleware and route, which is a large migration for the same result.~~ -- ~~Source: drafting~~ +~~- ~~Why: it keeps the whole middleware chain and the native gRPC transport for CAS streams, and it needs the smallest change.~~~~ +~~- ~~Alternatives considered:~~~~ +~~ - ~~One HTTP handler that routes each request: gRPC calls then use the HTTP adapter of the gRPC server. The gRPC library documents that path as slower and with fewer features. That puts R-003 at risk.~~~~ +~~ - ~~A move to Connect: it replaces every middleware and route, which is a large migration for the same result.~~~~ +~~- ~~Source: drafting~~~~ ### D-003: Deprecation default The legacy gRPC port is on by default, then off by default, then deleted. @@ -200,10 +200,10 @@ The chart uses one Service and one hostname per service. Only where the controll Superseded by D-015. ~~The services solve it in the server.~~ -- ~~Why: it works with any ingress, any proxy, and outside Kubernetes.~~ -- ~~Alternatives considered:~~ - - ~~A chart-level proxy or Gateway API routes (#2642): they need extra infrastructure and leave two ports in the service.~~ -- ~~Source: drafting~~ +~~- ~~Why: it works with any ingress, any proxy, and outside Kubernetes.~~~~ +~~- ~~Alternatives considered:~~~~ +~~ - ~~A chart-level proxy or Gateway API routes (#2642): they need extra infrastructure and leave two ports in the service.~~~~ +~~- ~~Source: drafting~~~~ ### D-006: Metrics and profiler ports The metrics and profiler ports stay separate. @@ -235,10 +235,10 @@ The Helm chart change stays in this spec, as milestone 2. Superseded by D-014. ~~The single port routes each connection once. It supports only clients and proxies that keep gRPC and plain HTTP on separate HTTP/2 connections.~~ -- ~~Why: a connection that the native gRPC server owns cannot pass single requests to the HTTP server. Native gRPC transport for CAS streams (R-003) needs the connection split of D-001.~~ -- ~~Alternatives considered:~~ - - ~~Route each request on one HTTP/2 server: it supports every proxy. But all gRPC calls then use the slower HTTP adapter of the gRPC server, which puts R-003 at risk.~~ -- ~~Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927)~~ +~~- ~~Why: a connection that the native gRPC server owns cannot pass single requests to the HTTP server. Native gRPC transport for CAS streams (R-003) needs the connection split of D-001.~~~~ +~~- ~~Alternatives considered:~~~~ +~~ - ~~Route each request on one HTTP/2 server: it supports every proxy. But all gRPC calls then use the slower HTTP adapter of the gRPC server, which puts R-003 at risk.~~~~ +~~- ~~Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927)~~~~ ### D-011: gRPC-Web only on the control plane The single port serves gRPC-Web on the control plane only. The CAS serves gRPC and HTTP. @@ -262,13 +262,13 @@ Drops D-002. The current HTTP port becomes the single port, not the gRPC port. T Superseded by D-016. ~~Supersedes D-001. The service splits by connection in front of the current gRPC and HTTP servers, with the cmux library as the connection classifier. The library owns the connection buffering and the listener plumbing. The service owns only a small matcher and the HTTP request dispatcher. The matcher sends the server's HTTP/2 settings frame to clients that wait for it. It reads the first request's content type through the standard HTTP/2 library. The matcher closes the connection when its first request is not native gRPC. The dispatcher carries the gRPC-Web wrapper and the native-gRPC fallback.~~ -- ~~Why: it keeps the whole middleware chain and the native gRPC transport for CAS streams. It also takes HTTP/2 parsing out of the code this project maintains. Argo CD and etcd run the same library split in production.~~ -- ~~Why: the gRPC library documents that its HTTP adapter may differ in performance and features from the native transport. It does not document the adapter as slower, and this project has not measured a difference. The native transport stays the normal path for gRPC connections, so R-003 does not rest on the adapter.~~ -- ~~Alternatives considered:~~ - - ~~The custom connection parser of the first implementation: it works, but the project then maintains its own HTTP/2 frame parsing, replay, and resource bounds.~~ - - ~~One HTTP handler that routes each request: every gRPC call, on every connection, then uses the HTTP adapter of the gRPC server.~~ - - ~~A move to Connect: it requires migrating the generated server interfaces and the interceptors. Ordinary HTTP routes can stay, but the migration is larger than the classifier swap for the same single port.~~ -- ~~Source: [PR comment](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6266145446)~~ +~~- ~~Why: it keeps the whole middleware chain and the native gRPC transport for CAS streams. It also takes HTTP/2 parsing out of the code this project maintains. Argo CD and etcd run the same library split in production.~~~~ +~~- ~~Why: the gRPC library documents that its HTTP adapter may differ in performance and features from the native transport. It does not document the adapter as slower, and this project has not measured a difference. The native transport stays the normal path for gRPC connections, so R-003 does not rest on the adapter.~~~~ +~~- ~~Alternatives considered:~~~~ +~~ - ~~The custom connection parser of the first implementation: it works, but the project then maintains its own HTTP/2 frame parsing, replay, and resource bounds.~~~~ +~~ - ~~One HTTP handler that routes each request: every gRPC call, on every connection, then uses the HTTP adapter of the gRPC server.~~~~ +~~ - ~~A move to Connect: it requires migrating the generated server interfaces and the interceptors. Ordinary HTTP routes can stay, but the migration is larger than the classifier swap for the same single port.~~~~ +~~- ~~Source: [PR comment](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6266145446)~~~~ ### D-014: Separate upstream connections, one-way fallback Supersedes D-010. The single port routes each connection once. It supports clients and proxies that keep native gRPC and plain HTTP on separate upstream HTTP/2 connections. Supported ingress configurations use separate upstream pools for the two kinds of traffic. The chart documents and tests them for nginx and Traefik. A connection that the HTTP server owns may carry a native gRPC request later. The HTTP server then gives the request to the gRPC server through the documented fallback, on the adapter transport. The reverse is not supported: a connection that the native gRPC server owns cannot carry a plain HTTP request. Arbitrary mixed native-gRPC and plain-HTTP requests on one shared upstream connection are not required. @@ -289,16 +289,16 @@ Supersedes D-005. The services solve the multiplexing in the server process. The Superseded by D-017. ~~Supersedes D-013. The service keeps cmux connection classification, the native gRPC server, and the HTTP request dispatcher with its gRPC-Web wrapper and native-gRPC fallback. Both native gRPC and ordinary HTTP/2 clients may wait for server SETTINGS before their first request. Silence does not identify the protocol of that request.~~ -- ~~The matcher uses the standard HTTP/2 framer and HPACK decoder. It limits incoming frames to the initial 16 KiB maximum before reading their payloads, as well as limiting classification bytes, time, and concurrency.~~ -- ~~When the matcher sends an empty SETTINGS frame and the first request is HTTP, it waits within the classification bounds for that SETTINGS acknowledgement. It records the validated ACK's exact offset in cmux's replay. The HTTP listener removes only that nine-byte frame before the HTTP server reads the connection. The HTTP server owns all subsequent SETTINGS exchanges. An incomplete injected handshake is closed.~~ -- ~~Native gRPC connections still go to the existing native server with their original replay. HTTP connections that did not need an injected SETTINGS frame also retain their original replay. HTTP requests and later streams are not parsed again by the handoff adapter.~~ -- ~~Why: rejecting every non-gRPC request after the matcher sends SETTINGS also rejects valid HTTP-only clients. Forwarding the extra ACK unchanged makes net/http reject the connection after the client acknowledges the HTTP server's own SETTINGS. Removing the one ACK preserves its ownership without adding an ongoing frame parser.~~ -- ~~Evidence: focused tests cover HTTP and gRPC-Web content types over plaintext and TLS, ACKs before and after the first request, subsequent requests on the same connection, incomplete handshakes, and an oversized frame header without its payload. These tests do not establish full R-002 parity or R-003 CAS performance.~~ -- ~~Source: [implementation review](https://github.com/chainloop-dev/chainloop/pull/3647#pullrequestreview-5573391966).~~ +~~- ~~The matcher uses the standard HTTP/2 framer and HPACK decoder. It limits incoming frames to the initial 16 KiB maximum before reading their payloads, as well as limiting classification bytes, time, and concurrency.~~~~ +~~- ~~When the matcher sends an empty SETTINGS frame and the first request is HTTP, it waits within the classification bounds for that SETTINGS acknowledgement. It records the validated ACK's exact offset in cmux's replay. The HTTP listener removes only that nine-byte frame before the HTTP server reads the connection. The HTTP server owns all subsequent SETTINGS exchanges. An incomplete injected handshake is closed.~~~~ +~~- ~~Native gRPC connections still go to the existing native server with their original replay. HTTP connections that did not need an injected SETTINGS frame also retain their original replay. HTTP requests and later streams are not parsed again by the handoff adapter.~~~~ +~~- ~~Why: rejecting every non-gRPC request after the matcher sends SETTINGS also rejects valid HTTP-only clients. Forwarding the extra ACK unchanged makes net/http reject the connection after the client acknowledges the HTTP server's own SETTINGS. Removing the one ACK preserves its ownership without adding an ongoing frame parser.~~~~ +~~- ~~Evidence: focused tests cover HTTP and gRPC-Web content types over plaintext and TLS, ACKs before and after the first request, subsequent requests on the same connection, incomplete handshakes, and an oversized frame header without its payload. These tests do not establish full R-002 parity or R-003 CAS performance.~~~~ +~~- ~~Source: [implementation review](https://github.com/chainloop-dev/chainloop/pull/3647#pullrequestreview-5573391966).~~~~ ### ~~D-017: CAS shared endpoint: cmux with the native transport, and the corrected SETTINGS contract~~ Superseded by D-020. -Supersedes D-016 for the CAS endpoint. The control-plane parts of D-014 to D-016 were already superseded by [the replacement spec](issue-3628-controlplane-http-dispatch.md). +~~Supersedes D-016 for the CAS endpoint. The control-plane parts of D-014 to D-016 were already superseded by [the replacement spec](issue-3628-controlplane-http-dispatch.md).~~ ~~The CAS shared endpoint is `server.http.addr`. cmux reads the start of each new connection and hands the whole connection to one of the two existing transports: an HTTP/2 connection whose first request declares a native gRPC content type goes to the existing native gRPC listener (`grpc.Server.Serve`); every other connection goes to the existing HTTP server. ByteStream uploads and downloads therefore keep the grpc-go native transport, so R-003 does not rest on the HTTP adapter.~~ ~~- Why not the merged control-plane design, per-request `ServeHTTP` dispatch: the [PR #3663 1 GiB comparison](https://github.com/chainloop-dev/chainloop/pull/3663) measured about 1.25 GiB of combined client/server sampled peak live heap for adapter uploads, against about 28 MiB for the native transport, and download throughput 11.8% lower, already past the R-003 limit. grpc-go's handler transport reads the HTTP body in a separate goroutine into an unbounded receive buffer, so read-ahead is not bounded by the application's pace. CAS moves gigabyte streams; the control plane does not. The two services keep different transport designs on purpose.~~ @@ -310,7 +310,7 @@ Supersedes D-016 for the CAS endpoint. The control-plane parts of D-014 to D-016 ### ~~D-018: CAS chart: one hostname, path-split upstream pools, HTTP/1.1 for the HTTP routes~~ Superseded by D-020. -The chart exposes one CAS hostname and one public port, and splits the routes by protocol into separate upstream pools: the Google ByteStream, CAS, health, reflection, metadata, and channelz RPC paths go to the gRPC pool with an HTTP/2 upstream (nginx `backend-protocol: GRPC` and `grpc-services`; the upload body-size annotations move with them), and `/download/*`, `/statusz`, and `/infoz` go to the HTTP pool, which dials HTTP/1.1, not h2c. One certificate terminates at the edge. +~~The chart exposes one CAS hostname and one public port, and splits the routes by protocol into separate upstream pools: the Google ByteStream, CAS, health, reflection, metadata, and channelz RPC paths go to the gRPC pool with an HTTP/2 upstream (nginx `backend-protocol: GRPC` and `grpc-services`; the upload body-size annotations move with them), and `/download/*`, `/statusz`, and `/infoz` go to the HTTP pool, which dials HTTP/1.1, not h2c. One certificate terminates at the edge.~~ ~~- Why the path split is exact for CAS: CAS has no gRPC-Web (D-011), so its gRPC paths and its HTTP paths are disjoint. No path can carry both protocols. The control plane could not use this split because native gRPC and gRPC-Web share RPC paths; the [PR #3647 NGINX reproduction](https://github.com/chainloop-dev/chainloop/pull/3647#issuecomment-6272007730) showed the path-only split failing there with 415 on gRPC-Web. CAS does not have that failure mode.~~ ~~- Why HTTP/1.1 upstream on the HTTP pool: D-017. An HTTP/2 upstream for the HTTP routes triggers the matcher's SETTINGS injection, and the client's acknowledgement then breaks the HTTP server connection with GOAWAY.~~ ~~- Controller notes: community nginx merges same-host paths from multiple Ingress resources and sets the backend protocol per resource, which expresses the split. F5 NGINX conflicts on unrelated same-host Ingresses; use its VirtualServer resource. Traefik applies the `service.serversscheme` annotation per Service, so one Service cannot be dialed h2c for the gRPC routes and HTTP/1.1 for the HTTP routes; the Traefik split needs a second Service for the HTTP routes or a controller-specific resource. The browser download endpoint stays on the HTTP routes: it is published API ([docs](https://docs.chainloop.dev/api-reference/downloadservice/download-artifacts-from-cas)), and CAS does not drop it.~~ @@ -318,7 +318,7 @@ The chart exposes one CAS hostname and one public port, and splits the routes by ### ~~D-019: CAS ingress connection reuse and certificate ownership~~ Superseded by D-020. -Supplements D-018. Ingress-nginx shares a global keepalive cache. Separate Ingress resources alone do not isolate their connections. +~~Supplements D-018. Ingress-nginx shares a global keepalive cache. Separate Ingress resources alone do not isolate their connections.~~ ~~The CAS Service has an HTTP port and a gRPC port alias. Both target the same application listener. The HTTP routes use pod endpoints. The gRPC routes use the Service ClusterIP and the alias. Each protocol gets a different backend identity and upstream address. This keeps one Service, one hostname, and one application listener (R-008).~~ @@ -334,7 +334,7 @@ Supplements D-018. Ingress-nginx shares a global keepalive cache. Separate Ingre ### D-020: CAS shared endpoint: per-request dispatch with a standard-library reverse proxy to a private native server Supersedes D-017, D-018, and D-019. The merged control-plane design stays unchanged. -The CAS shared endpoint is `server.http.addr`, owned by one standard HTTP listener (HTTP/1.1, HTTP/2, optional TLS). Each request is dispatched: native gRPC (HTTP/2 + `application/grpc*`) goes through an in-process `httputil.ReverseProxy` to a private native gRPC server bound to an ephemeral loopback address; every other request uses the existing HTTP routes. The private server runs the same services and interceptors as the legacy listener and is independent of it: the legacy flag, address, and TLS settings are unchanged, and the private listener exists whether or not the legacy listener is enabled. +The CAS shared endpoint is `server.http.addr`, owned by one standard HTTP listener (HTTP/1.1, HTTP/2, optional TLS). Each request is dispatched: native gRPC — HTTP/2 with a content type in the `application/grpc` family, excluding `application/grpc-web*`, which CAS does not serve (D-011) — goes through an in-process `httputil.ReverseProxy` to a private native gRPC server bound to an ephemeral loopback address; every other request uses the existing HTTP routes. The private server runs the same services and interceptors as the legacy listener and is independent of it: the legacy flag, address, and TLS settings are unchanged, and the private listener exists whether or not the legacy listener is enabled. - Why not cmux (D-017): connection classification cannot serve the single-location all-HTTP/2 upstream that the control-plane chart shape produces. Once the SETTINGS-writing matcher has spoken, the connection is gRPC-only, and pooled mixed connections then fail — the 502 reproduction that D-019's Service port alias worked around. Per-request dispatch removes that contract instead of working around it: mixed native-gRPC and plain-HTTP requests share one upstream connection in any order. - Why the proxy and not `grpc.Server.ServeHTTP` for gRPC: the measured adapter read-ahead, about 1.25 GiB of live heap on a 1 GiB upload against about 28 MiB on the native transport. `httputil.ReverseProxy` streams the request body into the private connection, whose HTTP/2 flow-control window paces the proxy's reads while the shared listener's own window paces the client, so no unbounded read-ahead exists in either direction. gRPC frames, compression, metadata, and response trailers pass through without decoding or rebuilding. - Why a private loopback listener and not the legacy listener: the legacy listener carries independent address and TLS settings and is optional. An h2c proxy cannot dial a TLS-configured address, and the shared endpoint must work with the legacy listener disabled. @@ -346,7 +346,7 @@ The CAS shared endpoint is `server.http.addr`, owned by one standard HTTP listen ## Open Questions -- [x] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Answered by D-014: not required. Setups that share one upstream pool can route gRPC to the legacy gRPC port during the deprecation period. The chart documents the supported nginx and Traefik configurations, which keep the two kinds of traffic on separate upstream connections. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). +- [x] **Must the single port support proxies that share one upstream connection for gRPC and plain HTTP?** Examples: Envoy-based proxies, service meshes. Answered by D-014 for the connection-classification designs, which could not accept them. Superseded for the CAS by R-010 and D-020: the per-request dispatcher accepts mixed native-gRPC and plain-HTTP requests on one upstream HTTP/2 connection in any order, so a shared upstream pool works for the CAS shared endpoint. The control-plane replacement spec accepts mixed connections from the start. The legacy gRPC port remains available during the deprecation period. Source: [cubic review](https://github.com/chainloop-dev/chainloop/pull/3629#discussion_r4286475927). D-007 and D-008 record the answers to the questions from drafting. From 5ecf45046f64cd4ea76feb3d9a3a8001f630c4e6 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 00:59:29 +0200 Subject: [PATCH 10/19] docs(chart): give the cas.ingress TLS example a certificate source The example enabled tls without an issuer or pre-created Secret, so the rendered Ingress referenced a secret nobody creates. Add the cert-manager issuer annotation to the example and state the requirement. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Signed-off-by: Miguel Martinez Trivino --- deployment/chainloop/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/deployment/chainloop/README.md b/deployment/chainloop/README.md index 8b70060b2..f1e2e03a2 100644 --- a/deployment/chainloop/README.md +++ b/deployment/chainloop/README.md @@ -257,8 +257,12 @@ cas: enabled: true hostname: cas.example.com tls: true + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod ``` +With `tls: true`, configure a certificate source — the cert-manager issuer annotation above, or a pre-created Secret named `cas.example.com-tls` — so the rendered Ingress references a Secret that exists. + `cas.ingress.enabled` defaults to `false`, so a setup that only copies the documented `cas.ingressAPI` block leaves the shared hostname disabled. Retain `cas.ingressAPI` only while the legacy gRPC endpoint is also required. `cas.serviceAPI`, and the control plane's connection to the legacy CAS endpoint keep their existing settings. CAS adds no gRPC-Web or Connect support. Deployed-controller qualification and the full streaming-performance evidence still need work. See the [experiment instructions](../../app/artifact-cas/README.md#shared-endpoint). The chart configures HTTP/2 upstream traffic for these controllers: From 136efdcd7e73ed9f0fd6769084f30a42ef410482 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 08:34:30 +0200 Subject: [PATCH 11/19] refactor(cas): simplify shared-endpoint proxy config and fix flush comment - Build targetURL once at construction instead of per Rewrite call; add a terse comment noting SetURL does not mutate the shared URL. - Drop the explicit FlushInterval: 0 (zero value is identical behavior). - Fix the flush comment: immediate flushing for gRPC comes from ReverseProxy's streaming special case (ContentLength == -1), not from FlushInterval. - Remove the unused grpcAddr field from APIServer. Verified: gofmt clean, go vet clean, go test -race passes for all CAS packages. Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 Signed-off-by: Miguel Martinez Trivino --- app/artifact-cas/internal/server/http.go | 32 +++++++++++++----------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 8e504d5bf..44a718961 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -56,12 +56,11 @@ import ( // ordinary HTTP requests may share one upstream connection in any order, as // they do on the control plane (spec D-020). type APIServer struct { - lis net.Listener - httpSrv *http.Server - grpcSrv *grpc.Server - grpcAddr string - scheme string - workers sync.WaitGroup + lis net.Listener + httpSrv *http.Server + grpcSrv *grpc.Server + scheme string + workers sync.WaitGroup } // Start serves the shared endpoint until ctx is cancelled or a transport @@ -182,11 +181,10 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) log.NewHelper(logger).Infof("[shared] CAS endpoint (%s) listening on: %s", scheme, lis.Addr().String()) return &APIServer{ - lis: lis, - httpSrv: srv, - grpcSrv: grpcServers.Private, - grpcAddr: grpcServers.PrivateAddr, - scheme: scheme, + lis: lis, + httpSrv: srv, + grpcSrv: grpcServers.Private, + scheme: scheme, }, nil } @@ -195,12 +193,15 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) // and response bodies, forwards hop-by-hop-excluded headers (including gRPC // metadata such as the authorization token), and copies response trailers, // so the existing gRPC frames pass through without decoding or rebuilding. -// FlushInterval zero flushes each write immediately, which gRPC streaming +// gRPC responses carry no Content-Length, so ReverseProxy flushes each write +// immediately through its streaming special case, which gRPC streaming // requires. The transport speaks only unencrypted HTTP/2 (h2c prior // knowledge): the private server is gRPC-only and loopback-only. The // Rewrite restores the incoming authority after SetURL clears it, so the // native server receives the original :authority for interceptor parity. func grpcProxy(target string, logger log.Logger) *httputil.ReverseProxy { + // Fixed for the process lifetime: built once rather than per request. + targetURL := &url.URL{Scheme: "http", Host: target} transport := &h.Transport{ Protocols: new(h.Protocols), DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext, @@ -217,15 +218,16 @@ func grpcProxy(target string, logger log.Logger) *httputil.ReverseProxy { transport.Protocols.SetHTTP1(false) return &httputil.ReverseProxy{ Rewrite: func(pr *httputil.ProxyRequest) { - pr.SetURL(&url.URL{Scheme: "http", Host: target}) + // SetURL only reads the target, so one shared URL is safe across + // concurrent requests. + pr.SetURL(targetURL) // SetURL clears Out.Host for security (RFC 7230 host injection), // but gRPC carries the routing target in :authority. Restore the // incoming authority so the native server's interceptors observe // the same value they would on a direct connection. pr.Out.Host = pr.In.Host }, - Transport: transport, - FlushInterval: 0, + Transport: transport, ErrorHandler: func(w h.ResponseWriter, _ *h.Request, err error) { log.NewHelper(logger).Errorf("shared endpoint gRPC proxy error: %v", err) w.WriteHeader(h.StatusBadGateway) From 06c844e3d84c4e1632eeb286fe0bae96a1a0b5e4 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 09:41:05 +0200 Subject: [PATCH 12/19] fix(cas): satisfy lint on shared-endpoint tests Factor the repeated artifact filename into casTestFileName and drop the unused opts parameter from serviceOptsForTest. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/internal/server/http_test.go | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index b9ed8fed5..2d3681294 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -59,6 +59,7 @@ import ( const ( casAnyAddr = "127.0.0.1:0" + casTestFileName = "artifact.bin" casSharedEndpoint = "shared" casLegacyEndpoint = "legacy" casGRPCRequest = "grpc" @@ -97,7 +98,7 @@ func (b *diskBackend) Describe(_ context.Context, digest string) (*v1.CASResourc if err != nil { return nil, err } - return &v1.CASResource{Digest: digest, FileName: "artifact.bin", Size: info.Size()}, nil + return &v1.CASResource{Digest: digest, FileName: casTestFileName, Size: info.Size()}, nil } func (b *diskBackend) Exists(_ context.Context, digest string) (bool, error) { _, err := os.Stat(filepath.Join(b.dir, digest)) @@ -209,7 +210,7 @@ func casTestUpload(t testing.TB, client bytestream.ByteStreamClient, token, dige if err != nil { return err } - resource := casResourceName(t, &v1.CASResource{Digest: digest, FileName: "artifact.bin"}) + resource := casResourceName(t, &v1.CASResource{Digest: digest, FileName: casTestFileName}) chunk := make([]byte, 64<<10) for { n, readErr := payload.Read(chunk) @@ -311,7 +312,7 @@ func TestCASSharedPausedConsumerBackpressure(t *testing.T) { t.Run("upload", func(t *testing.T) { opts := casTestOpts(t) arm, release, hook := pacedUploads(t, pauseAt) - opts.ByteService = service.NewByteStreamService(opts.Providers, append(serviceOptsForTest(t, opts), service.WithPaceHook(hook))...) + opts.ByteService = service.NewByteStreamService(opts.Providers, append(serviceOptsForTest(t), service.WithPaceHook(hook))...) sharedURL, _ := startCASTestServers(t, opts) client := bytestream.NewByteStreamClient(casTestConn(t, sharedURL[len("http://"):], false)) up := casTestToken(t, casJWT.Uploader, 0, time.Hour) @@ -327,7 +328,7 @@ func TestCASSharedPausedConsumerBackpressure(t *testing.T) { defer cancel() stream, err := client.Write(ctx) require.NoError(t, err) - resource := casResourceName(t, &v1.CASResource{Digest: digest, FileName: "artifact.bin"}) + resource := casResourceName(t, &v1.CASResource{Digest: digest, FileName: casTestFileName}) chunk := make([]byte, 1<<20) var sent atomic.Int64 sendErr := make(chan error, 1) @@ -370,7 +371,7 @@ func TestCASSharedPausedConsumerBackpressure(t *testing.T) { t.Run("download", func(t *testing.T) { opts := casTestOpts(t) arm, release, hook := pacedUploads(t, pauseAt) - opts.ByteService = service.NewByteStreamService(opts.Providers, append(serviceOptsForTest(t, opts), service.WithPaceHook(hook))...) + opts.ByteService = service.NewByteStreamService(opts.Providers, append(serviceOptsForTest(t), service.WithPaceHook(hook))...) sharedURL, _ := startCASTestServers(t, opts) client := bytestream.NewByteStreamClient(casTestConn(t, sharedURL[len("http://"):], false)) up, down := casTestToken(t, casJWT.Uploader, 0, time.Hour), casTestToken(t, casJWT.Downloader, 0, time.Hour) @@ -420,7 +421,7 @@ func TestCASSharedPausedConsumerBackpressure(t *testing.T) { // serviceOptsForTest mirrors the test wiring in casTestOpts so pace-hook // variants get the same staging option. -func serviceOptsForTest(tb testing.TB, opts *Opts) []service.NewOpt { +func serviceOptsForTest(tb testing.TB) []service.NewOpt { return []service.NewOpt{service.WithStagingDir(tb.TempDir())} } From 61aa68ac3e576342571baea9a53ecdbffcf205a0 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 10:03:29 +0200 Subject: [PATCH 13/19] chore(cas): skip iac-misconfiguration entropy check on sample configs Both config.devel.yaml and samples/config.yaml are development/sample configs with local placeholder values (public-key paths, dev tokens). The PR-validation iac-misconfiguration policy flags them as Base64 High Entropy String errors. Annotate the shared-endpoint blocks with checkov:skip=CKV_SECRET_6, matching the existing precedent in deployment/chainloop/values.yaml. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/configs/config.devel.yaml | 1 + app/artifact-cas/configs/samples/config.yaml | 1 + 2 files changed, 2 insertions(+) diff --git a/app/artifact-cas/configs/config.devel.yaml b/app/artifact-cas/configs/config.devel.yaml index 61364d7de..748fb78e0 100644 --- a/app/artifact-cas/configs/config.devel.yaml +++ b/app/artifact-cas/configs/config.devel.yaml @@ -4,6 +4,7 @@ # [root]/devel/compose.yml server: # Keep 9001 available to compare native gRPC with the shared 8001 endpoint. + # checkov:skip=CKV_SECRET_6: development/sample config with local placeholder values, no real credentials legacy_grpc_listener: true http: # Shared endpoint: HTTP/1.1, HTTP/2, and native gRPC. Each request is diff --git a/app/artifact-cas/configs/samples/config.yaml b/app/artifact-cas/configs/samples/config.yaml index 2af490377..f242ea975 100644 --- a/app/artifact-cas/configs/samples/config.yaml +++ b/app/artifact-cas/configs/samples/config.yaml @@ -4,6 +4,7 @@ auth: server: # Retain the native transport while evaluating the shared HTTP endpoint. + # checkov:skip=CKV_SECRET_6: sample config with local placeholder values, no real credentials legacy_grpc_listener: true http: addr: 0.0.0.0:8001 From 8664eec242ca2ebbd755f1e1d49dfe15ee55f1c1 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 12:15:21 +0200 Subject: [PATCH 14/19] chore(cas): attach CKV_SECRET_6 skips to the flagged values, not legacy_grpc_listener The suppression must sit immediately before the placeholder values it covers (the dev vault token and the commented TLS examples), not on the unrelated legacy_grpc_listener field, or Checkov still reports the lines. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/configs/config.devel.yaml | 3 ++- app/artifact-cas/configs/samples/config.yaml | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/app/artifact-cas/configs/config.devel.yaml b/app/artifact-cas/configs/config.devel.yaml index 748fb78e0..c2f4e3441 100644 --- a/app/artifact-cas/configs/config.devel.yaml +++ b/app/artifact-cas/configs/config.devel.yaml @@ -4,7 +4,6 @@ # [root]/devel/compose.yml server: # Keep 9001 available to compare native gRPC with the shared 8001 endpoint. - # checkov:skip=CKV_SECRET_6: development/sample config with local placeholder values, no real credentials legacy_grpc_listener: true http: # Shared endpoint: HTTP/1.1, HTTP/2, and native gRPC. Each request is @@ -22,6 +21,7 @@ server: # Some unary RPCs are slow, so we need to increase the timeout # For example, Azure Blob Storage describe takes more than 1 second to respond sometimes timeout: 5s + # checkov:skip=CKV_SECRET_6: commented-out local dev TLS example paths, no real credentials # tls_config: # certificate: "../../devel/devkeys/selfsigned/cas.crt" # private_key: "../../devel/devkeys/selfsigned/cas.key" @@ -34,6 +34,7 @@ credentials_service: # Remember to run vault via docker compose up vault: address: ${VAULT_ADDRESS:http://0.0.0.0:8200} + # checkov:skip=CKV_SECRET_6: development vault dev-mode token, not a real credential token: ${VAULT_TOKEN:notasecret} observability: diff --git a/app/artifact-cas/configs/samples/config.yaml b/app/artifact-cas/configs/samples/config.yaml index f242ea975..725d907a8 100644 --- a/app/artifact-cas/configs/samples/config.yaml +++ b/app/artifact-cas/configs/samples/config.yaml @@ -4,7 +4,6 @@ auth: server: # Retain the native transport while evaluating the shared HTTP endpoint. - # checkov:skip=CKV_SECRET_6: sample config with local placeholder values, no real credentials legacy_grpc_listener: true http: addr: 0.0.0.0:8001 @@ -13,12 +12,14 @@ server: # request; native gRPC reaches the native transport through an in-process # reverse proxy (spec D-020). # Optional TLS for both transports on port 8001. Independent of grpc.tls_config. + # checkov:skip=CKV_SECRET_6: commented-out local dev TLS example paths, no real credentials # tls_config: # certificate: "./configs/tls/shared.crt" # private_key: "./configs/tls/shared.key" grpc: addr: 0.0.0.0:9001 timeout: 1s + # checkov:skip=CKV_SECRET_6: local dev TLS example paths, no real credentials tls_config: certificate: "./configs/tls/server.crt" private_key: "./configs/tls/server.key" @@ -31,6 +32,7 @@ credentials_service: # Remember to run vault via docker compose up vault: address: ${VAULT_ADDRESS:http://0.0.0.0:8200} + # checkov:skip=CKV_SECRET_6: development vault dev-mode token, not a real credential token: ${VAULT_TOKEN:notasecret} # aws_secret_manager: From 6e1032e6d56d59a09197dad60c040d2bac3acbc9 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 12:31:01 +0200 Subject: [PATCH 15/19] fix(cas): address review round on metrics TLS, and fail the endpoint if the private gRPC transport dies server.http_metrics.tls_config was never applied by the metrics server; rejecting it broke existing deployments that set it. Ignore it with a warning and stay plaintext, preserving the previous behavior. APIServer.Start filtered ErrClosed/ErrServerStopped unconditionally, so a private gRPC transport that failed on its own was swallowed and the app kept serving HTTP while proxied gRPC returned 502. Track an explicit stopping flag: the ordinary-stop sentinel is benign only when it comes from Stop. A spontaneous failure now propagates so the whole endpoint stops. TestCASSharedGRPCFailureStopsEndpoint pins this. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/cmd/wire_gen.go | 2 +- app/artifact-cas/internal/server/http.go | 24 ++++++++++++----- app/artifact-cas/internal/server/http_test.go | 26 +++++++++++++++++++ .../internal/server/httpmetrics.go | 11 ++++---- .../internal/server/httpmetrics_test.go | 13 +++++----- 5 files changed, 57 insertions(+), 19 deletions(-) diff --git a/app/artifact-cas/cmd/wire_gen.go b/app/artifact-cas/cmd/wire_gen.go index 87c2c4050..5553b6f05 100644 --- a/app/artifact-cas/cmd/wire_gen.go +++ b/app/artifact-cas/cmd/wire_gen.go @@ -77,7 +77,7 @@ func wireApp(bootstrap *conf.Bootstrap, confServer *conf.Server, auth *conf.Auth cleanup() return nil, nil, err } - httpMetricsServer, err := server.NewHTTPMetricsServer(confServer) + httpMetricsServer, err := server.NewHTTPMetricsServer(confServer, logger) if err != nil { cleanup2() cleanup() diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 44a718961..6e20cc529 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -26,6 +26,7 @@ import ( "net/url" "strings" "sync" + "sync/atomic" "time" api "github.com/chainloop-dev/chainloop/app/artifact-cas/api/cas/v1" @@ -56,16 +57,20 @@ import ( // ordinary HTTP requests may share one upstream connection in any order, as // they do on the control plane (spec D-020). type APIServer struct { - lis net.Listener - httpSrv *http.Server - grpcSrv *grpc.Server - scheme string - workers sync.WaitGroup + lis net.Listener + httpSrv *http.Server + grpcSrv *grpc.Server + scheme string + workers sync.WaitGroup + stopping atomic.Bool } // Start serves the shared endpoint until ctx is cancelled or a transport // fails. The HTTP server serves every connection; the private native gRPC -// server serves the connections the proxy opens to it. +// server serves the connections the proxy opens to it. A genuine failure of +// either transport — including the private gRPC listener failing — is +// surfaced so the whole endpoint stops, rather than serving HTTP while +// proxied gRPC answers 502. func (s *APIServer) Start(ctx context.Context) error { errCh := make(chan error, 2) s.workers.Go(func() { errCh <- s.httpSrv.Start(ctx) }) @@ -74,7 +79,11 @@ func (s *APIServer) Start(ctx context.Context) error { case <-ctx.Done(): return nil case err := <-errCh: - if isOrdinaryStopError(err) { + // The ordinary-stop sentinel (ErrClosed / ErrServerStopped) is benign + // only when it comes from our own Stop. The same error from a + // spontaneously failed transport must propagate, or the app would keep + // serving HTTP after the private gRPC server is gone. + if s.stopping.Load() && isOrdinaryStopError(err) { return nil } return err @@ -85,6 +94,7 @@ func (s *APIServer) Start(ctx context.Context) error { // stream, because those live on HTTP connections — then stops the private // native gRPC server so its in-flight RPCs finish. func (s *APIServer) Stop(ctx context.Context) error { + s.stopping.Store(true) httpErr := s.httpSrv.Stop(ctx) grpcErr := s.grpcSrv.Stop(ctx) s.workers.Wait() diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index 2d3681294..da510c8b3 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -822,6 +822,32 @@ func TestCASSharedShutdownDrainsUpload(t *testing.T) { } } +// TestCASSharedGRPCFailureStopsEndpoint: if the private gRPC transport fails, +// the shared endpoint must not keep serving HTTP while proxied gRPC answers +// 502. Start must surface the failure so the app stops. +func TestCASSharedGRPCFailureStopsEndpoint(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + // Simulate the private gRPC transport failing: close its listener before + // the servers start. grpc-go's Serve on the closed listener returns an + // error that Start must not swallow. + require.NoError(t, servers.Close()) + startErr := make(chan error, 1) + go func() { startErr <- api.Start(context.Background()) }() + select { + case err := <-startErr: + require.Error(t, err, "a failed private gRPC transport must fail the shared endpoint") + case <-time.After(10 * time.Second): + require.FailNow(t, "Start kept serving after the private gRPC transport failed") + } + // Best-effort teardown of the HTTP server that started on the shared port. + _ = api.Stop(context.Background()) +} + func TestCASSharedInitializationDoesNotBindOnInvalidKey(t *testing.T) { opts := casTestOpts(t) opts.ServerConfig.LegacyGrpcListener = new(false) diff --git a/app/artifact-cas/internal/server/httpmetrics.go b/app/artifact-cas/internal/server/httpmetrics.go index 03bd1b72c..841554684 100644 --- a/app/artifact-cas/internal/server/httpmetrics.go +++ b/app/artifact-cas/internal/server/httpmetrics.go @@ -16,9 +16,8 @@ package server import ( - "errors" - "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/conf" + "github.com/go-kratos/kratos/v2/log" "github.com/go-kratos/kratos/v2/middleware/recovery" "github.com/go-kratos/kratos/v2/transport/http" "github.com/prometheus/client_golang/prometheus/promhttp" @@ -29,10 +28,12 @@ type HTTPMetricsServer struct { *http.Server } -// NewHTTPMetricsServer exposes the metrics endpoint -func NewHTTPMetricsServer(c *conf.Server) (*HTTPMetricsServer, error) { +// NewHTTPMetricsServer exposes the metrics endpoint. server.http_metrics never +// applied TLS: a configured tls_config is ignored with a warning rather than +// rejected, so existing deployments that set it keep working. +func NewHTTPMetricsServer(c *conf.Server, logger log.Logger) (*HTTPMetricsServer, error) { if c.GetHttpMetrics().GetTlsConfig() != nil { - return nil, errors.New("server.http_metrics.tls_config is unsupported; configure shared endpoint TLS under server.http.tls_config") + log.NewHelper(logger).Warn("server.http_metrics.tls_config is ignored: metrics stay plaintext; configure shared endpoint TLS under server.http.tls_config") } var opts = []http.ServerOption{ http.Middleware( diff --git a/app/artifact-cas/internal/server/httpmetrics_test.go b/app/artifact-cas/internal/server/httpmetrics_test.go index 73065786b..9c6efa4d4 100644 --- a/app/artifact-cas/internal/server/httpmetrics_test.go +++ b/app/artifact-cas/internal/server/httpmetrics_test.go @@ -15,13 +15,16 @@ package server import ( + "io" "testing" "github.com/chainloop-dev/chainloop/app/artifact-cas/internal/conf" + "github.com/go-kratos/kratos/v2/log" "github.com/stretchr/testify/require" ) func TestCASMetricsTLSConfiguration(t *testing.T) { + logger := log.NewStdLogger(io.Discard) for _, tc := range []struct { name string tls *conf.Server_TLS @@ -31,12 +34,10 @@ func TestCASMetricsTLSConfiguration(t *testing.T) { {name: "TLS config", tls: &conf.Server_TLS{Certificate: "cert.pem", PrivateKey: "key.pem"}}, } { t.Run(tc.name, func(t *testing.T) { - _, err := NewHTTPMetricsServer(&conf.Server{HttpMetrics: &conf.Server_HTTP{Addr: casAnyAddr, TlsConfig: tc.tls}}) - if tc.tls != nil { - require.ErrorContains(t, err, "server.http_metrics.tls_config is unsupported") - } else { - require.NoError(t, err) - } + // A configured but unsupported tls_config is ignored with a warning, + // so existing deployments that set it keep working. + _, err := NewHTTPMetricsServer(&conf.Server{HttpMetrics: &conf.Server_HTTP{Addr: casAnyAddr, TlsConfig: tc.tls}}, logger) + require.NoError(t, err) }) } } From f07f7dd5ba4561bfc0389f7a72e3e5a60afec76f Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 13:15:44 +0200 Subject: [PATCH 16/19] fix(cas): log a deprecation warning when the legacy gRPC listener is enabled Match the merged control-plane change (#3647): when server.legacy_grpc_listener is on, log that the gRPC-only listener is deprecated and point at the shared HTTP endpoint and the flag. This is the R-006 startup warning. TestCASLegacyListenerDeprecationWarning pins it firing when enabled and staying quiet when disabled. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/internal/server/http.go | 3 ++ app/artifact-cas/internal/server/http_test.go | 29 +++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 6e20cc529..286cf46e6 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -189,6 +189,9 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) }) log.NewHelper(logger).Infof("[shared] CAS endpoint (%s) listening on: %s", scheme, lis.Addr().String()) + if legacyGRPCListenerEnabled(c) { + log.NewHelper(logger).Warnw("msg", fmt.Sprintf("the gRPC-only listener on %q is deprecated; use the shared HTTP endpoint and set server.legacy_grpc_listener to false when all clients have moved", c.GetGrpc().GetAddr())) + } return &APIServer{ lis: lis, diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index da510c8b3..cc31feb05 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -31,6 +31,7 @@ import ( "net/http/httptest" "os" "path/filepath" + "strings" "sync" "sync/atomic" "testing" @@ -646,6 +647,34 @@ func TestCASAuthenticationAndUploadErrors(t *testing.T) { } } +// TestCASLegacyListenerDeprecationWarning covers the R-006 startup warning: +// the shared endpoint logs a deprecation warning when the legacy gRPC +// listener is enabled, matching the control plane. +func TestCASLegacyListenerDeprecationWarning(t *testing.T) { + for _, tc := range []struct { + name string + enabled *bool + wantLog bool + }{ + {name: "default on", wantLog: true}, + {name: "explicit on", enabled: new(true), wantLog: true}, + {name: "explicit off", enabled: new(false), wantLog: false}, + } { + t.Run(tc.name, func(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = tc.enabled + var buf bytes.Buffer + opts.Logger = log.NewStdLogger(&buf) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + require.Equal(t, tc.wantLog, strings.Contains(buf.String(), "the gRPC-only listener"), "logs: %s", buf.String()) + require.NoError(t, api.Stop(context.Background())) + }) + } +} + func TestCASLegacyListenerAndTLSConfiguration(t *testing.T) { for _, tc := range []struct { name string From c4a0a2db0b48ef79ad4ae23f524f5092dffbd8ab Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 13:21:25 +0200 Subject: [PATCH 17/19] fix(cas): address proxy review round - pool sizing, stream termination docs, dispatch comments Cap the proxy upstream pool at 10 connections: the target is one loopback host and HTTP/2 multiplexes many streams per connection, so the cap only bounds idle loopback FDs between requests, not throughput. Document the idle-timeout and ExpectContinue rationale on the transport. Document that the ErrorHandler's headers-only 502 is finalized by net/http with END_STREAM when the handler returns (covered by TestCASProxyErrorTerminatesHTTP2Stream against a dead upstream), that the Start errCh is buffered to the worker count so a second transport error never blocks, and that native gRPC requires HTTP/2 at the dispatch gate. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/internal/server/http.go | 24 +++++++++++++-- app/artifact-cas/internal/server/http_test.go | 30 +++++++++++++++++++ 2 files changed, 52 insertions(+), 2 deletions(-) diff --git a/app/artifact-cas/internal/server/http.go b/app/artifact-cas/internal/server/http.go index 286cf46e6..1c48c2aec 100644 --- a/app/artifact-cas/internal/server/http.go +++ b/app/artifact-cas/internal/server/http.go @@ -72,6 +72,9 @@ type APIServer struct { // surfaced so the whole endpoint stops, rather than serving HTTP while // proxied gRPC answers 502. func (s *APIServer) Start(ctx context.Context) error { + // Buffered to the worker count: both sends complete into the buffer even + // when Start returns after the first error, so the second transport's + // report never blocks and the goroutine always finishes. errCh := make(chan error, 2) s.workers.Go(func() { errCh <- s.httpSrv.Start(ctx) }) s.workers.Go(func() { errCh <- s.grpcSrv.Start(ctx) }) @@ -181,6 +184,9 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) // with the control plane. proxy := grpcProxy(grpcServers.PrivateAddr, logger) srv.Handler = h.HandlerFunc(func(w h.ResponseWriter, r *h.Request) { + // Native gRPC requires HTTP/2: an HTTP/1.1 request with an + // application/grpc content type cannot reach the private gRPC server, + // so only HTTP/2 requests take the proxy branch. if r.ProtoMajor == 2 && isNativeGRPCContentType(r.Header.Get("Content-Type")) { proxy.ServeHTTP(w, r) return @@ -212,14 +218,24 @@ func NewHTTPServer(settings *Opts, grpcServers *GRPCServers) (*APIServer, error) // knowledge): the private server is gRPC-only and loopback-only. The // Rewrite restores the incoming authority after SetURL clears it, so the // native server receives the original :authority for interceptor parity. +// +// The upstream pool targets a single loopback host and HTTP/2 multiplexes +// many streams onto one connection, so a small connection cap is plenty; the +// cap exists to bound idle loopback FDs held between requests, not to shape +// throughput. Idle connections are reaped after the 90-second idle timeout so +// a drained transfer does not pin the pooled connection for the process +// lifetime; Stop drains HTTP first, so idle upstream sockets only persist +// briefly past the last request. ExpectContinueTimeout is a no-op for gRPC +// (clients never send Expect: 100-continue) and does not hold a pool slot +// beyond the dial. func grpcProxy(target string, logger log.Logger) *httputil.ReverseProxy { // Fixed for the process lifetime: built once rather than per request. targetURL := &url.URL{Scheme: "http", Host: target} transport := &h.Transport{ Protocols: new(h.Protocols), DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext, - MaxIdleConns: 100, - MaxIdleConnsPerHost: 100, + MaxIdleConns: 10, + MaxIdleConnsPerHost: 10, IdleConnTimeout: 90 * time.Second, ExpectContinueTimeout: time.Second, // Byte-exact pass-through: gRPC carries its own per-message content @@ -241,6 +257,10 @@ func grpcProxy(target string, logger log.Logger) *httputil.ReverseProxy { pr.Out.Host = pr.In.Host }, Transport: transport, + // On an upstream failure the handler must only write the status; when + // it returns, net/http finalizes the HTTP/2 response with END_STREAM, + // so the client gets a clean 502 instead of a stream left open. The + // regression test TestCASProxyErrorTerminatesHTTP2Stream covers this. ErrorHandler: func(w h.ResponseWriter, _ *h.Request, err error) { log.NewHelper(logger).Errorf("shared endpoint gRPC proxy error: %v", err) w.WriteHeader(h.StatusBadGateway) diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index cc31feb05..f044af43f 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -279,6 +279,36 @@ func TestCASProxyPreservesAuthority(t *testing.T) { require.NoError(t, err) } +// TestCASProxyErrorTerminatesHTTP2Stream pins the ErrorHandler contract: on an +// upstream failure the proxy writes only the status header, and net/http +// finalizes the HTTP/2 response with END_STREAM when the handler returns, so +// the client gets a clean 502 instead of a stream left open. +func TestCASProxyErrorTerminatesHTTP2Stream(t *testing.T) { + proxy := grpcProxy("127.0.0.1:1", log.NewStdLogger(io.Discard)) // nothing listening + srv := httptest.NewUnstartedServer(proxy) + srv.EnableHTTP2 = true + srv.StartTLS() + defer srv.Close() + + tr := &http.Transport{Protocols: new(http.Protocols), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} //nolint:gosec // local test certificate + tr.Protocols.SetHTTP2(true) + defer tr.CloseIdleConnections() + client := &http.Client{Transport: tr, Timeout: 5 * time.Second} + + req, err := http.NewRequest(http.MethodPost, srv.URL+"/cas.v1.StatusService/Statusz", bytes.NewReader([]byte{0, 0, 0, 0, 0})) + require.NoError(t, err) + req.Header.Set("Content-Type", "application/grpc") + req.Header.Set("Te", "trailers") + resp, err := client.Do(req) + require.NoError(t, err) + // Reading the body to completion must terminate, not wait for more frames. + _, err = io.Copy(io.Discard, resp.Body) + require.NoError(t, resp.Body.Close()) + require.NoError(t, err) + require.Equal(t, http.StatusBadGateway, resp.StatusCode) + require.Equal(t, 2, resp.ProtoMajor) +} + // pacedUploads gates a ByteStream service so each direction can pause its // consumer at a fixed transferred byte count. The hook stays inert until arm // is closed (so a setup upload through the same service is not paused), and From 84f4c8b8ea5b40383d1d67d28d1b18e019f723cb Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 13:29:09 +0200 Subject: [PATCH 18/19] test(cas): prove dual-transport failure leaks no goroutine Cover the errCh contract empirically: when both transports fail before Start reads the first error, the second transport's report is still buffered (the channel capacity equals the worker count), so its goroutine finishes and Stop's workers.Wait() does not hang. The channel is not closed after the first read: a late writer would panic on send-to-closed. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/internal/server/http_test.go | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/app/artifact-cas/internal/server/http_test.go b/app/artifact-cas/internal/server/http_test.go index f044af43f..c99ed550b 100644 --- a/app/artifact-cas/internal/server/http_test.go +++ b/app/artifact-cas/internal/server/http_test.go @@ -829,6 +829,42 @@ func TestCASSharedConcurrentUploadsMemoryBounded(t *testing.T) { "four concurrent 128 MiB uploads must not buffer whole streams; peak heap %d bytes", peak) } +// TestCASSharedDualTransportFailureDoesNotLeak proves the errCh contract: when +// both transports fail before Start reads the first error, the second +// transport's report is still buffered (capacity equals the worker count), so +// its goroutine finishes and is not leaked. Stop's workers.Wait() would hang +// if the second send blocked. +func TestCASSharedDualTransportFailureDoesNotLeak(t *testing.T) { + opts := casTestOpts(t) + opts.ServerConfig.LegacyGrpcListener = new(false) + servers, err := NewGRPCServers(opts) + require.NoError(t, err) + api, err := NewHTTPServer(opts, servers) + require.NoError(t, err) + // Fail both transports before Start: close the shared HTTP listener and + // the private gRPC listener. + require.NoError(t, api.lis.Close()) + require.NoError(t, servers.Close()) + + done := make(chan error, 1) + go func() { done <- api.Start(context.Background()) }() + select { + case err := <-done: + require.Error(t, err, "Start must fail when both transports fail") + case <-time.After(10 * time.Second): + require.FailNow(t, "Start did not return on dual-transport failure") + } + // The second writer must not be stuck: Stop would otherwise block forever + // in workers.Wait(). + stopped := make(chan struct{}) + go func() { _ = api.Stop(context.Background()); close(stopped) }() + select { + case <-stopped: + case <-time.After(10 * time.Second): + require.FailNow(t, "Stop hung: a transport goroutine is still blocked on errCh") + } +} + func TestCASSharedShutdownDrainsUpload(t *testing.T) { opts := casTestOpts(t) opts.ServerConfig.LegacyGrpcListener = new(false) From 573f776ff6131cf834512ed8347a71121aed242e Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Tue, 20 Oct 2026 14:12:21 +0200 Subject: [PATCH 19/19] chore(cas): place CKV_SECRET_6 skips inline on the flagged config lines The SARIF findings point at the dev credential-path prefix and a commented-out example secret, not the TLS/vault lines the earlier annotations guarded. Move the suppressions inline onto those exact lines (secret_prefix in both files, and the commented secret_key in the sample), matching the end-of-line pattern already used in deployment/chainloop/values.yaml. Refs #3628. Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: ses_eaa37d42cffeMTXKfQSHj8wUG0 --- app/artifact-cas/configs/config.devel.yaml | 4 +--- app/artifact-cas/configs/samples/config.yaml | 7 ++----- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/app/artifact-cas/configs/config.devel.yaml b/app/artifact-cas/configs/config.devel.yaml index c2f4e3441..83863e30d 100644 --- a/app/artifact-cas/configs/config.devel.yaml +++ b/app/artifact-cas/configs/config.devel.yaml @@ -21,7 +21,6 @@ server: # Some unary RPCs are slow, so we need to increase the timeout # For example, Azure Blob Storage describe takes more than 1 second to respond sometimes timeout: 5s - # checkov:skip=CKV_SECRET_6: commented-out local dev TLS example paths, no real credentials # tls_config: # certificate: "../../devel/devkeys/selfsigned/cas.crt" # private_key: "../../devel/devkeys/selfsigned/cas.key" @@ -30,11 +29,10 @@ server: credentials_service: # we will check that we can read there - secret_prefix: chainloop-devel + secret_prefix: chainloop-devel # checkov:skip=CKV_SECRET_6: development-only credential-path prefix, not a credential # Remember to run vault via docker compose up vault: address: ${VAULT_ADDRESS:http://0.0.0.0:8200} - # checkov:skip=CKV_SECRET_6: development vault dev-mode token, not a real credential token: ${VAULT_TOKEN:notasecret} observability: diff --git a/app/artifact-cas/configs/samples/config.yaml b/app/artifact-cas/configs/samples/config.yaml index 725d907a8..7807a93fc 100644 --- a/app/artifact-cas/configs/samples/config.yaml +++ b/app/artifact-cas/configs/samples/config.yaml @@ -12,14 +12,12 @@ server: # request; native gRPC reaches the native transport through an in-process # reverse proxy (spec D-020). # Optional TLS for both transports on port 8001. Independent of grpc.tls_config. - # checkov:skip=CKV_SECRET_6: commented-out local dev TLS example paths, no real credentials # tls_config: # certificate: "./configs/tls/shared.crt" # private_key: "./configs/tls/shared.key" grpc: addr: 0.0.0.0:9001 timeout: 1s - # checkov:skip=CKV_SECRET_6: local dev TLS example paths, no real credentials tls_config: certificate: "./configs/tls/server.crt" private_key: "./configs/tls/server.key" @@ -28,17 +26,16 @@ server: credentials_service: # We use the prefix to check that we can read from it on initialization - secret_prefix: chainloop-devel + secret_prefix: chainloop-devel # checkov:skip=CKV_SECRET_6: development-only credential-path prefix, not a credential # Remember to run vault via docker compose up vault: address: ${VAULT_ADDRESS:http://0.0.0.0:8200} - # checkov:skip=CKV_SECRET_6: development vault dev-mode token, not a real credential token: ${VAULT_TOKEN:notasecret} # aws_secret_manager: # creds: # access_key: not-a-key - # secret_key: not-a-secret + # secret_key: not-a-secret # checkov:skip=CKV_SECRET_6: commented-out example value, not a credential # region: us-east-1 # gcp_secret_manager: