diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cdcf6c2..430cb6d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **The playground opts a broadcast into ABR, and plays ABR broadcasts adaptively (`playground`).** The publish board's ABR box announces the broadcast's opt-in beneath `/.abr/` beside it, as the HLS scenario does beneath `/.hls/`. A broadcast whose video tracks share an alternate group is played adaptively: the viewer switches renditions at group boundaries, stepping down when frames fall behind their media time to what the rate measured since then carries, and up one rendition after a calm stretch. A step up keeps the old rendition until its last group before the switch has arrived; a step down drops it at once. The subscribe board picks a rendition by hand or automatically, and shows the rendition playing, the receive rate and the lag. - **`PEERS` names are resolved again while the relay runs, so peers behind a name join and leave without a restart; an entry without a port is an SRV name (`_moqt._udp.hubs.example.com`) (`internal/relay`).** A `PEERS` entry that is neither `host:port` nor a name without a port now stops the relay at startup, and a host that doesn't resolve is retried instead of being dialed by name. - **The relay re-reads its certificates while it runs (`internal/relay`).** About once a minute it reads `CERT_FILE`/`KEY_FILE`, `PEER_CERT_FILE`/`PEER_KEY_FILE` and `CA_FILE` and compares them with what it holds; what changed is taken from the next TLS handshake on, so cert-manager, a Vault or step-ca agent, or any job that writes the files needs no restart. A public certificate is taken once it parses and matches its key; a peer certificate and the CA together, once they pass the startup checks (issued by the CA, the peering name, valid now). Anything else is logged, once per distinct error rather than at every read, and the certificates in use are kept; renewal tools should replace the files atomically (write, then rename). Sessions already established are untouched. The CA is now checked at each handshake on both sides: as the dialed relay (`GetConfigForClient`) and as the dialer (`VerifyConnection` against the current CA, with the same checks crypto/tls made). The relay's key-set and usage requests are signed with the renewed peer certificate (`reqsign.Signer.Rotate`), and `qumo dns` re-reads its `PEER_CERT_FILE`/`PEER_KEY_FILE` on the same one-minute cadence and signs its feed reads with a renewed pair (`internal/certreload`, shared by both). New metrics: `qumo_relay_cert_reloads_total{cert,result}` and `qumo_relay_cert_not_after_seconds{cert}`, and `qumo_dns_cert_reloads_total{cert,result}` and `qumo_dns_cert_not_after_seconds{cert}`; alert on the `not_after` gauges. A CA is rotated by putting the new one beside the old in `CA_FILE` first ([TLS & mTLS](docs/site/content/en/docs/deployment/tls.md#certificate-renewal)). +- **A relay reports how long each publishing session kept broadcasts opted into HLS and ABR (`internal/relay`, `usage.OptInSeconds`).** A publishing session's `usage` and `session_close` records now carry `gateway.opt_in_seconds.abr` and `gateway.opt_in_seconds.hls`: the whole seconds its announcements beneath `/.abr/` and `/.hls/` lasted, added up over its broadcasts, from when the relay received each until it ended, the session closed or a revoked ticket ended it. Only an opt-in the session's grant covers reaches the relay, so the time is billed to the key that holds the opt-in, and a control plane can charge for ABR or HLS by the time it was used. `qumo abr` itself never needs to know whose broadcast it transcodes. A session that opted nothing into an egress carries no metric for it, so a receiver that doesn't bill opt-ins sees records as before. ### Changed diff --git a/docs/site/content/en/docs/cli/abr.md b/docs/site/content/en/docs/cli/abr.md index d98a20c4..6091d845 100644 --- a/docs/site/content/en/docs/cli/abr.md +++ b/docs/site/content/en/docs/cli/abr.md @@ -43,6 +43,11 @@ is refused within half a minute, and one on air ends at the next 30 s check of live sessions. HLS is held the same way (`"hls"`), and a key whose entry lists no `optIns` holds none. +A relay with `QUMO_USAGE_URL` reports how long each opt-in lasted, in the +publishing session's usage records under the tenant's key +(`gateway.opt_in_seconds.abr`, see [usage reports]({{< relref "../configuration#usage-reports" >}})), +so the control plane can bill ABR by the time it was used. + The tenant's app signs the opt-in's token with the same key as the broadcast's: `/.abr/acme/live/cam1` beside `/acme/live/cam1`, presented on the publisher's session as a ticket, since a session's credentials all come from diff --git a/docs/site/content/en/docs/configuration.md b/docs/site/content/en/docs/configuration.md index 29dbfffb..5a8dfed3 100644 --- a/docs/site/content/en/docs/configuration.md +++ b/docs/site/content/en/docs/configuration.md @@ -255,7 +255,8 @@ With `QUMO_USAGE_URL` set, the relay POSTs a JSON array of records to it every 1 {"type": "usage", "session_id": "…", "kid": "…", "metrics": {"gateway.ingress_bytes": 10162, "gateway.egress_bytes": 9223}, "ts": "…"}, {"type": "session_close", "session_id": "…", "kid": "…", - "metrics": {"gateway.ingress_bytes": 10438, "gateway.egress_bytes": 9165}, "reason": "closed", "ts": "…"}, + "metrics": {"gateway.ingress_bytes": 10438, "gateway.egress_bytes": 9165, + "gateway.opt_in_seconds.abr": 312}, "reason": "closed", "ts": "…"}, {"type": "viewers", "run": "…", "kid": "…", "metrics": {"gateway.ingress_bytes": 120, "gateway.egress_bytes": 880412}, "ts": "…"}] ``` @@ -264,6 +265,8 @@ Byte counts are cumulative and from the relay's point of view (what it received **Sessions that publish a broadcast are reported one by one,** as the first three records: an open, usage every 30 s, and a close. A session publishes when its token has a `publish` pattern, or from when it is granted a ticket that has one: its open comes then, and what it moved as a viewer up to its last 30 s check stays in its key's viewer total, the rest counting as its usage. +**A publishing session's records also carry its opt-in time:** how long it kept broadcasts opted into each egress, `gateway.opt_in_seconds.abr` for [`qumo abr`](../cli/abr/) and `gateway.opt_in_seconds.hls` for [`qumo hls`](../cli/hls/), in whole seconds, added up over its broadcasts (two broadcasts opted into ABR for a minute each are 120). It is the time each announcement of the session beneath `/.abr/` or `/.hls/` lasted on this relay, from when the relay received it until it ended, the session closed or a revoked ticket ended it, so a control plane can bill an opt-in by the time it was used. Only an opt-in the session's grant covers reaches the relay, so an opt-in a key doesn't hold is never counted. An egress the session opted nothing into has no metric. A broadcast opted in through [RTMP or RTSP ingest](../cli/rtmp/#opting-streams-into-hls-and-abr) counts on the ingest's session, not the publisher's. + **Every other session is a viewer, and viewers are reported together.** Viewers outnumber publishers by orders of magnitude, and a record per viewer would make the receiver's load grow with the audience. Their bytes are added up per key into one running total since the relay started, sent as a `viewers` record naming the key and the relay's `run`. The `run` is new each time the relay starts, so a restart begins new totals rather than lowering old ones. There is no open or close for viewers, and a total is sent only when it has changed. Records go out in batches of at most 500. A failed send, a 401, 403, 408, 413 or 429 included, keeps the records and retries them (logged once, then again when it recovers); only a batch the receiver can't read (400 or 422) is dropped. When the relay shuts down, it waits up to 5 s for its last sessions to record their end before the final send. diff --git a/internal/auth/auth.go b/internal/auth/auth.go index 010991cb..1ba0df66 100644 --- a/internal/auth/auth.go +++ b/internal/auth/auth.go @@ -13,6 +13,9 @@ import ( "errors" "fmt" "net/http" + "time" + + "github.com/qumo-dev/qumo/token" ) // maxBody bounds how much of a reply's body is drained. @@ -73,6 +76,10 @@ type Request struct { // Bytes is the session's cumulative byte totals, set on revalidate and // end. Bytes Bytes + // OptIns is how long the session has kept broadcasts opted into each + // egress, added up over its broadcasts, set on revalidate and end + // (usage.OptInSeconds). An egress it opted nothing into is absent. + OptIns map[token.OptIn]time.Duration // Reason is why the session ended, set on end. Reason string } diff --git a/internal/auth/usage.go b/internal/auth/usage.go index 3832ff0d..18cc9795 100644 --- a/internal/auth/usage.go +++ b/internal/auth/usage.go @@ -8,6 +8,7 @@ import ( "github.com/qumo-dev/qumo/internal/meter" "github.com/qumo-dev/qumo/internal/reqsign" + "github.com/qumo-dev/qumo/token" "github.com/qumo-dev/qumo/usage" ) @@ -103,10 +104,10 @@ func (r *usageReporter) publishing(id string) { r.meter.Open(id, s.kid) } -// revalidated records a session's cumulative bytes at a revalidate and, when -// live is set, that the session is still live. A session it doesn't know is -// left unknown. -func (r *usageReporter) revalidated(id string, b Bytes, live bool) { +// revalidated records a session's cumulative bytes and opt-in time at a +// revalidate and, when live is set, that the session is still live. A +// session it doesn't know is left unknown. +func (r *usageReporter) revalidated(id string, b Bytes, optIns map[token.OptIn]time.Duration, live bool) { r.mu.Lock() defer r.mu.Unlock() s, ok := r.sessions[id] @@ -116,18 +117,17 @@ func (r *usageReporter) revalidated(id string, b Bytes, live bool) { if live { s.seen = r.now() } - switch { - case !s.publishes: + if !s.publishes { s = r.countViewer(s, b) - case b.Sent > s.counted.Sent || b.Received > s.counted.Received: - r.meter.Report(id, s.kid, usageMetrics(b.since(s.counted))) + } else if m := publisherMetrics(b.since(s.counted), optIns); carriesUsage(m) { + r.meter.Report(id, s.kid, m) } r.sessions[id] = s } // close records a session's end with its final totals and forgets it. It // reports false for a session it doesn't know. -func (r *usageReporter) close(id string, b Bytes, reason string) bool { +func (r *usageReporter) close(id string, b Bytes, optIns map[token.OptIn]time.Duration, reason string) bool { r.mu.Lock() defer r.mu.Unlock() r.closed[id] = r.now() @@ -140,7 +140,7 @@ func (r *usageReporter) close(id string, b Bytes, reason string) bool { r.countViewer(s, b) return true } - r.meter.Close(id, s.kid, usageMetrics(b.since(s.counted)), reason) + r.meter.Close(id, s.kid, publisherMetrics(b.since(s.counted), optIns), reason) return true } @@ -166,6 +166,29 @@ func usageMetrics(b Bytes) map[string]int64 { return map[string]int64{usage.IngressBytes: clampInt64(b.Received), usage.EgressBytes: clampInt64(b.Sent)} } +// publisherMetrics is a publishing session's metrics: its bytes, and how +// long it kept broadcasts opted into each egress, in whole seconds. +func publisherMetrics(b Bytes, optIns map[token.OptIn]time.Duration) map[string]int64 { + m := usageMetrics(b) + for o, d := range optIns { + if secs := int64(d / time.Second); secs > 0 { + m[usage.OptInSeconds(string(o))] = secs + } + } + return m +} + +// carriesUsage reports whether m holds any usage: a session that moved +// nothing and kept nothing opted in for a whole second has none to report. +func carriesUsage(m map[string]int64) bool { + for _, v := range m { + if v > 0 { + return true + } + } + return false +} + func clampInt64(v uint64) int64 { if v > math.MaxInt64 { return math.MaxInt64 diff --git a/internal/auth/usage_viewers_test.go b/internal/auth/usage_viewers_test.go index 79ee16e7..ac7459d3 100644 --- a/internal/auth/usage_viewers_test.go +++ b/internal/auth/usage_viewers_test.go @@ -4,7 +4,9 @@ import ( "context" "net/http" "testing" + "time" + "github.com/qumo-dev/qumo/token" "github.com/qumo-dev/qumo/usage" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -25,9 +27,9 @@ func TestUsageReporter_ViewersAreTotalled(t *testing.T) { r.open(id, viewerSession("k1")) } r.open("other", viewerSession("k2")) - r.revalidated("v1", Bytes{Sent: 100, Received: 10}, true) - r.revalidated("v2", Bytes{Sent: 200, Received: 20}, true) - r.revalidated("other", Bytes{Sent: 7}, true) + r.revalidated("v1", Bytes{Sent: 100, Received: 10}, nil, true) + r.revalidated("v2", Bytes{Sent: 200, Received: 20}, nil, true) + r.revalidated("other", Bytes{Sent: 7}, nil, true) require.NoError(t, r.meter.Flush(context.Background())) first := sink.received()[0] @@ -49,11 +51,11 @@ func TestUsageReporter_ViewersAreTotalled(t *testing.T) { }) t.Run("the total grows by what each session moved since it was last counted", func(t *testing.T) { - r.revalidated("v1", Bytes{Sent: 150, Received: 10}, true) // +50 - r.revalidated("v1", Bytes{Sent: 120, Received: 10}, true) // a late, lower report adds nothing - assert.True(t, r.close("v2", Bytes{Sent: 260, Received: 25}, "closed")) // +60, +5 - r.revalidated("v2", Bytes{Sent: 999}, true) // after its close: ignored - assert.True(t, r.close("v3", Bytes{}, "closed")) // moved nothing + r.revalidated("v1", Bytes{Sent: 150, Received: 10}, nil, true) // +50 + r.revalidated("v1", Bytes{Sent: 120, Received: 10}, nil, true) // a late, lower report adds nothing + assert.True(t, r.close("v2", Bytes{Sent: 260, Received: 25}, nil, "closed")) // +60, +5 + r.revalidated("v2", Bytes{Sent: 999}, nil, true) // after its close: ignored + assert.True(t, r.close("v3", Bytes{}, nil, "closed")) // moved nothing require.NoError(t, r.meter.Flush(context.Background())) batches := sink.received() @@ -67,10 +69,10 @@ func TestUsageReporter_ViewerTotalSurvivesAFailedSend(t *testing.T) { sink := &fakeUsageSink{statuses: []int{http.StatusServiceUnavailable, http.StatusOK}} r := newTestUsage(t, sink) r.open("v1", viewerSession("k1")) - r.revalidated("v1", Bytes{Sent: 100}, true) + r.revalidated("v1", Bytes{Sent: 100}, nil, true) assert.Error(t, r.meter.Flush(context.Background())) - r.revalidated("v1", Bytes{Sent: 160}, true) + r.revalidated("v1", Bytes{Sent: 160}, nil, true) require.NoError(t, r.meter.Flush(context.Background())) batches := sink.received() @@ -84,7 +86,7 @@ func TestUsageReporter_PublishersStayPerSession(t *testing.T) { r := newTestUsage(t, sink) r.open("p1", usageSession{kid: "k1", publishes: true}) r.open("p2", usageSession{kid: "k1", publishes: true}) - r.revalidated("p1", Bytes{Received: 50}, true) + r.revalidated("p1", Bytes{Received: 50}, nil, true) require.NoError(t, r.meter.Flush(context.Background())) var got []string @@ -107,7 +109,7 @@ func TestUsageReporter_Publishing(t *testing.T) { r.publishing("v1") r.publishing("p1") r.publishing("unknown") - r.revalidated("v1", Bytes{Received: 40}, true) + r.revalidated("v1", Bytes{Received: 40}, nil, true) require.NoError(t, r.meter.Flush(context.Background())) var got []string @@ -132,3 +134,58 @@ func TestBytes_Since(t *testing.T) { }) } } + +// A publishing session's usage carries how long it kept broadcasts opted +// into each egress, in whole seconds, and is reported for it even when the +// session moved no bytes; its close carries the final time. A viewer's +// total carries none. +func TestUsageReporter_OptInSeconds(t *testing.T) { + sink := &fakeUsageSink{} + r := newTestUsage(t, sink) + r.open("p1", usageSession{kid: "k1", publishes: true}) + r.open("v1", viewerSession("k1")) + abr := func(d time.Duration) map[token.OptIn]time.Duration { + return map[token.OptIn]time.Duration{token.OptInABR: d} + } + + r.revalidated("p1", Bytes{Received: 100}, abr(1500*time.Millisecond), true) + r.revalidated("v1", Bytes{Sent: 10}, abr(time.Hour), true) + require.NoError(t, r.meter.Flush(context.Background())) + first := byType(sink.received()[0]) + assert.Equal(t, map[string]int64{ + usage.IngressBytes: 100, usage.EgressBytes: 0, usage.OptInSeconds("abr"): 1, + }, first[usage.Usage].Metrics, "whole seconds") + assert.NotContains(t, first[usage.Viewers].Metrics, usage.OptInSeconds("abr"), "a viewer opts nothing in") + + r.open("p2", usageSession{kid: "k1", publishes: true}) + r.revalidated("p2", Bytes{}, abr(400*time.Millisecond), true) + require.NoError(t, r.meter.Flush(context.Background())) + require.Len(t, sink.received(), 2) + assert.NotContains(t, byType(sink.received()[1]), usage.Usage, + "no bytes and under a second opted in is nothing to report") + + r.revalidated("p2", Bytes{}, abr(2*time.Second), true) + require.NoError(t, r.meter.Flush(context.Background())) + require.Len(t, sink.received(), 3) + assert.Equal(t, int64(2), byType(sink.received()[2])[usage.Usage].Metrics[usage.OptInSeconds("abr")], + "opt-in time without bytes is reported") + + assert.True(t, r.close("p1", Bytes{Received: 120}, map[token.OptIn]time.Duration{ + token.OptInABR: 3 * time.Second, token.OptInHLS: 500 * time.Millisecond, + }, usage.ReasonClosed)) + require.NoError(t, r.meter.Flush(context.Background())) + closed := byType(sink.received()[3])[usage.SessionClose] + assert.Equal(t, map[string]int64{ + usage.IngressBytes: 120, usage.EgressBytes: 0, usage.OptInSeconds("abr"): 3, + }, closed.Metrics, "under a second of HLS is left out") + assert.NoError(t, closed.Validate()) +} + +// byType returns batch's records by type, the last of each. +func byType(batch []usage.Record) map[usage.Type]usage.Record { + out := map[usage.Type]usage.Record{} + for _, rec := range batch { + out[rec.Type] = rec + } + return out +} diff --git a/internal/auth/verifier.go b/internal/auth/verifier.go index 0d29e9db..b7726521 100644 --- a/internal/auth/verifier.go +++ b/internal/auth/verifier.go @@ -146,7 +146,7 @@ func (v *Verifier) Authorize(_ context.Context, req Request) (*Grant, error) { // A refused session is ending; one that couldn't be checked // lives on, and the relay asks again. _, refused := errors.AsType[RefusedError](err) - v.usage.revalidated(req.ID, req.Bytes, !refused) + v.usage.revalidated(req.ID, req.Bytes, req.OptIns, !refused) } } return g, err @@ -157,7 +157,7 @@ func (v *Verifier) Authorize(_ context.Context, req Request) (*Grant, error) { // counted once a ticket binds it to a key (Bind). func (v *Verifier) anonymous(req Request) *Grant { if v.usage != nil && req.Event == EventRevalidate { - v.usage.revalidated(req.ID, req.Bytes, true) + v.usage.revalidated(req.ID, req.Bytes, req.OptIns, true) } return &Grant{revalidate: revalidateEvery, anonymous: true} } @@ -302,11 +302,11 @@ func publishesBroadcast(g token.Grant) bool { return len(g.Publish) > 0 } -// End reports a session's end with its final byte totals. +// End reports a session's end with its final byte totals and opt-in time. func (v *Verifier) End(_ context.Context, req Request) error { if v.usage == nil { return nil } - v.usage.close(req.ID, req.Bytes, req.Reason) + v.usage.close(req.ID, req.Bytes, req.OptIns, req.Reason) return nil } diff --git a/internal/auth/verifier_test.go b/internal/auth/verifier_test.go index 605f2d2e..c7782481 100644 --- a/internal/auth/verifier_test.go +++ b/internal/auth/verifier_test.go @@ -405,8 +405,8 @@ func TestUsageReporter(t *testing.T) { sink := &fakeUsageSink{} r := newTestUsage(t, sink) r.open("s1", testUsageSession) - r.revalidated("s1", Bytes{Sent: 1, Received: 10}, true) - r.revalidated("s1", Bytes{Sent: 2, Received: 20}, true) + r.revalidated("s1", Bytes{Sent: 1, Received: 10}, nil, true) + r.revalidated("s1", Bytes{Sent: 2, Received: 20}, nil, true) require.NoError(t, r.meter.Flush(context.Background())) batches := sink.received() require.Len(t, batches, 1) @@ -419,9 +419,9 @@ func TestUsageReporter(t *testing.T) { sink := &fakeUsageSink{statuses: []int{http.StatusBadGateway, http.StatusOK}} r := newTestUsage(t, sink) r.open("s1", testUsageSession) - r.revalidated("s1", Bytes{Received: 10}, true) + r.revalidated("s1", Bytes{Received: 10}, nil, true) assert.Error(t, r.meter.Flush(context.Background())) - r.revalidated("s1", Bytes{Received: 20}, true) + r.revalidated("s1", Bytes{Received: 20}, nil, true) require.NoError(t, r.meter.Flush(context.Background())) retry := sink.received()[1] require.Len(t, retry, 2) @@ -435,8 +435,8 @@ func TestUsageReporter(t *testing.T) { t.Run("an unknown session's end and usage are ignored", func(t *testing.T) { sink := &fakeUsageSink{} r := newTestUsage(t, sink) - assert.False(t, r.close("nope", Bytes{Sent: 1}, "")) - r.revalidated("nope", Bytes{Sent: 1}, true) + assert.False(t, r.close("nope", Bytes{Sent: 1}, nil, "")) + r.revalidated("nope", Bytes{Sent: 1}, nil, true) require.NoError(t, r.meter.Flush(context.Background())) assert.Empty(t, sink.received(), "nothing pending, nothing sent") }) @@ -450,13 +450,13 @@ func TestUsageReporter(t *testing.T) { r.open("live", testUsageSession) now = now.Add(sessionGrace + time.Second) - r.revalidated("refused", Bytes{Sent: 1}, false) - r.revalidated("live", Bytes{}, true) + r.revalidated("refused", Bytes{Sent: 1}, nil, false) + r.revalidated("live", Bytes{}, nil, true) r.forgetStale() - assert.False(t, r.close("gone", Bytes{}, ""), "not revalidated within the grace") - assert.False(t, r.close("refused", Bytes{}, ""), "a refused revalidate doesn't keep it") - assert.True(t, r.close("live", Bytes{}, ""), "revalidated: still remembered") + assert.False(t, r.close("gone", Bytes{}, nil, ""), "not revalidated within the grace") + assert.False(t, r.close("refused", Bytes{}, nil, ""), "a refused revalidate doesn't keep it") + assert.True(t, r.close("live", Bytes{}, nil, ""), "revalidated: still remembered") }) } @@ -556,7 +556,7 @@ func TestVerifier_RevalidateAfterEnd(t *testing.T) { types = append(types, rec.Type) } assert.Equal(t, []usage.Type{usage.SessionOpen, usage.SessionClose}, types) - assert.False(t, v.usage.close(late.ID, Bytes{}, ""), "the session isn't known again") + assert.False(t, v.usage.close(late.ID, Bytes{}, nil, ""), "the session isn't known again") } // writeKeySet writes a key set file holding keys and returns its path. @@ -972,7 +972,7 @@ func TestUsageReporter_OpenAfterClose(t *testing.T) { now := time.Now() r.now = func() time.Time { return now } - r.close("s1", Bytes{}, "closed") + r.close("s1", Bytes{}, nil, "closed") r.open("s1", usageSession{kid: "k1", publishes: true}) r.publishing("s1") diff --git a/internal/ingest/auth.go b/internal/ingest/auth.go index 239fbcb2..4e9ab881 100644 --- a/internal/ingest/auth.go +++ b/internal/ingest/auth.go @@ -52,21 +52,10 @@ func rtmpTarget(app, streamKey string) (moqt.BroadcastPath, []string) { // and media there would answer an egress that expects a redirect. var errOptInPath = errors.New("the path lies beneath an opt-in directory, where the server opts streams in: push to the broadcast's own path") -// optInDirectory returns the opt-in directory path lies beneath, such as -// /.abr/ for /.abr/live/cam, and false when it lies beneath none. -func optInDirectory(path moqt.BroadcastPath) (string, bool) { - for _, o := range token.OptIns() { - if dir := o.Directory(); strings.HasPrefix(string(path), dir) { - return dir, true - } - } - return "", false -} - // checkPushPath refuses a push to path when the server keeps it for opt-ins // (errOptInPath). func checkPushPath(path moqt.BroadcastPath) error { - if _, ok := optInDirectory(path); ok { + if _, ok := optin.Of(path); ok { return fmt.Errorf("%s: %w", path, errOptInPath) } return nil diff --git a/internal/ingest/cmd.go b/internal/ingest/cmd.go index 80de060b..87583c9f 100644 --- a/internal/ingest/cmd.go +++ b/internal/ingest/cmd.go @@ -290,8 +290,8 @@ func publishAuthorizer(v *auth.Verifier, beneath string) PublishAuthorizer { } return func(credential string, path moqt.BroadcastPath) (time.Time, error) { broadcast := path - if dir, ok := optInDirectory(path); ok { - b, named := optin.Broadcast(string(path), dir) + if o, ok := optin.Of(path); ok { + b, named := optin.Broadcast(string(path), o.Directory()) if !named { // Cleaned, it could name a broadcast outside the // directory the relay grants its exact path in. diff --git a/internal/integration/optin_usage_test.go b/internal/integration/optin_usage_test.go new file mode 100644 index 00000000..ce5f3571 --- /dev/null +++ b/internal/integration/optin_usage_test.go @@ -0,0 +1,135 @@ +//go:build integration + +package integration + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync" + "testing" + "time" + + "github.com/qumo-dev/gomoqt/moqt" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/qumo-dev/qumo/internal/auth" + "github.com/qumo-dev/qumo/internal/optin" + "github.com/qumo-dev/qumo/internal/relay" + "github.com/qumo-dev/qumo/token" + "github.com/qumo-dev/qumo/usage" +) + +// usageSink records the usage records POSTed to it. +type usageSink struct { + mu sync.Mutex + records []usage.Record +} + +func (s *usageSink) ServeHTTP(w http.ResponseWriter, r *http.Request) { + var batch []usage.Record + if err := json.NewDecoder(r.Body).Decode(&batch); err != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + s.mu.Lock() + defer s.mu.Unlock() + s.records = append(s.records, batch...) +} + +func (s *usageSink) all() []usage.Record { + s.mu.Lock() + defer s.mu.Unlock() + return append([]usage.Record(nil), s.records...) +} + +// TestRelay_ReportsOptInSeconds runs a paid tenant's publisher against a +// relay that reports usage: it publishes a broadcast and, on a ticket its key +// holds the ABR opt-in for, opts it into ABR. The close record of its session +// carries how long the opt-in lasted (usage.OptInSeconds), under the +// tenant's key, so a control plane can bill ABR by the time it was used. +func TestRelay_ReportsOptInSeconds(t *testing.T) { + const src = moqt.BroadcastPath("/live/metered") + optInPath := optin.Path(token.ABRDirectory, src) + + tenant, err := token.GenerateKey("/live/") + require.NoError(t, err) + tenantPub := tenant.Public() + tenantPub.OptIns = []token.OptIn{token.OptInABR} + set, err := token.MarshalKeySet(tenantPub) + require.NoError(t, err) + keys := filepath.Join(t.TempDir(), "keys.json") + require.NoError(t, os.WriteFile(keys, set, 0o600)) + sink := &usageSink{} + usageSrv := httptest.NewServer(sink) + t.Cleanup(usageSrv.Close) + verifier, err := auth.NewVerifier(auth.VerifierConfig{Keys: keys, UsageURL: usageSrv.URL}) + require.NoError(t, err) + runCtx, stopRun := context.WithCancel(context.Background()) + runDone := make(chan struct{}) + go func() { + defer close(runDone) + verifier.Run(runCtx) + }() + t.Cleanup(func() { stopRun(); <-runDone }) + + ended := make(chan auth.Request, 4) + addr, srv := startAuthRelay(t, verifier.Authorize, nil, func(s *relay.Server) { + s.Tickets = verifier + s.End = func(ctx context.Context, req auth.Request) error { + err := verifier.End(ctx, req) + ended <- req + return err + } + }) + sign := func(g token.Grant) string { + tok, err := token.Sign(tenant, g, time.Minute) + require.NoError(t, err) + return tok + } + + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + pubMux := moqt.NewTrackMux(0) + pubMux.PublishFunc(ctx, src, func(tw *moqt.TrackWriter) { <-tw.Context().Done() }) + // Native QUIC: see TestABR_PaidOptIn. + pub := dialOver(t, nativeURL(addr)+"/?jwt="+sign(token.Grant{Root: "/live/", Publish: []string{"/metered"}}), nil, pubMux) + actx, acancel := context.WithTimeout(ctx, 3*time.Second) + defer acancel() + _, err = pub.Auth(actx, []byte(sign(token.Grant{Root: "/.abr/live/", Publish: []string{"/metered"}}))) + require.NoError(t, err, "the key holds ABR") + pubMux.Publish(ctx, optInPath, optin.Handler(src)) + require.Eventually(t, routed(srv, optInPath), 5*time.Second, 25*time.Millisecond) + + // Opted in for a little over two seconds, then the publisher leaves. + time.Sleep(2200 * time.Millisecond) + require.NoError(t, pub.CloseWithError(moqt.NoError, "done")) + var end auth.Request + select { + case end = <-ended: + case <-time.After(5 * time.Second): + t.Fatal("the relay did not report the session's end") + } + assert.GreaterOrEqual(t, end.OptIns[token.OptInABR], 2*time.Second) + assert.NotContains(t, end.OptIns, token.OptInHLS, "nothing was opted into HLS") + + // The verifier sends what is pending once more when it stops. + stopRun() + <-runDone + var closed *usage.Record + for _, rec := range sink.all() { + if rec.Type == usage.SessionClose && rec.SessionID == end.ID { + closed = &rec + } + } + require.NotNil(t, closed, "no session_close for the publisher's session in %v", sink.all()) + assert.Equal(t, tenant.ID, closed.KID) + secs := closed.Metrics[usage.OptInSeconds("abr")] + assert.GreaterOrEqual(t, secs, int64(2)) + assert.LessOrEqual(t, secs, int64(5), "the time the opt-in lasted, not longer") + assert.NoError(t, closed.Validate()) +} diff --git a/internal/optin/optin.go b/internal/optin/optin.go index ae8b3241..a2db4d88 100644 --- a/internal/optin/optin.go +++ b/internal/optin/optin.go @@ -11,6 +11,8 @@ import ( "strings" "github.com/qumo-dev/gomoqt/moqt" + + "github.com/qumo-dev/qumo/token" ) // Handler serves the opt-in of broadcast, announced as broadcast's path @@ -25,6 +27,18 @@ func Handler(broadcast moqt.BroadcastPath) moqt.TrackHandler { }) } +// Of returns the egress p opts a broadcast into, when p lies beneath an +// opt-in directory: token.OptInABR for /.abr/live/cam1. It reports false for +// a path beneath none. +func Of(p moqt.BroadcastPath) (token.OptIn, bool) { + for _, o := range token.OptIns() { + if strings.HasPrefix(string(p), o.Directory()) { + return o, true + } + } + return "", false +} + // Path returns the opt-in of broadcast beneath dir, an opt-in directory: // Path("/.abr/", "/live/cam1") is "/.abr/live/cam1". func Path(dir string, broadcast moqt.BroadcastPath) moqt.BroadcastPath { diff --git a/internal/optin/optin_test.go b/internal/optin/optin_test.go index d63c69bd..3cc66b98 100644 --- a/internal/optin/optin_test.go +++ b/internal/optin/optin_test.go @@ -42,6 +42,26 @@ func TestBroadcast(t *testing.T) { } } +func TestOf(t *testing.T) { + tests := map[moqt.BroadcastPath]struct { + want token.OptIn + wantOK bool + }{ + "/.abr/live/cam1": {want: token.OptInABR, wantOK: true}, + "/.hls/live/cam1": {want: token.OptInHLS, wantOK: true}, + "/live/cam1": {}, + "/.abrx/cam1": {}, + "/live/.abr/cam1": {}, + } + for p, tt := range tests { + t.Run(string(p), func(t *testing.T) { + got, ok := Of(p) + assert.Equal(t, tt.wantOK, ok) + assert.Equal(t, tt.want, got) + }) + } +} + func TestHasDotSegment(t *testing.T) { tests := map[string]struct { path string diff --git a/internal/relay/admit.go b/internal/relay/admit.go index 2627b4d1..afbbe852 100644 --- a/internal/relay/admit.go +++ b/internal/relay/admit.go @@ -698,15 +698,17 @@ func (s *Server) admitClient(ctx context.Context, req auth.Request) (g *auth.Gra } // reportEnd sends the end event of a checked session admitted by req, with -// its final byte totals and why it ended. ctx is the session's: done by now, -// so the report keeps its values but not its cancellation. A failed report is -// recorded and otherwise ignored; End bounds its own wait. -func (s *Server) reportEnd(ctx context.Context, req auth.Request, stats moqt.SessionStats, reason string) { +// its final byte totals, its opt-in time (optIns, nil for none) and why it +// ended. ctx is the session's: done by now, so the report keeps its values +// but not its cancellation. A failed report is recorded and otherwise +// ignored; End bounds its own wait. +func (s *Server) reportEnd(ctx context.Context, req auth.Request, stats moqt.SessionStats, optIns *optInClock, reason string) { if s.End == nil { return } req.Event = auth.EventEnd req.Bytes = sessionBytes(stats) + req.OptIns = optIns.totals() req.Reason = reason if err := s.End(context.WithoutCancel(ctx), req); err != nil { metricAuthRequests.WithLabelValues(auth.EventEnd, authError).Inc() diff --git a/internal/relay/admit_test.go b/internal/relay/admit_test.go index 9711b688..13c19656 100644 --- a/internal/relay/admit_test.go +++ b/internal/relay/admit_test.go @@ -193,7 +193,7 @@ func TestServer_ReportEnd(t *testing.T) { connect := auth.Request{ID: "00ff", Event: auth.EventConnect, Path: "/acme", Query: "jwt=h.p.s"} delta := counterDelta(t, func() { - srv.reportEnd(context.Background(), connect, stats, endDropped) + srv.reportEnd(context.Background(), connect, stats, nil, endDropped) }, metricAuthRequests.WithLabelValues(auth.EventEnd, tt.wantResult)) assert.Equal(t, 1.0, delta) @@ -210,7 +210,7 @@ func TestServer_ReportEnd(t *testing.T) { // reports nothing. func TestServer_ReportEnd_Off(t *testing.T) { delta := counterDelta(t, func() { - (&Server{}).reportEnd(context.Background(), auth.Request{ID: "00ff"}, moqt.SessionStats{}, endClosed) + (&Server{}).reportEnd(context.Background(), auth.Request{ID: "00ff"}, moqt.SessionStats{}, nil, endClosed) }, metricAuthRequests.WithLabelValues(auth.EventEnd, authOK)) assert.Zero(t, delta) diff --git a/internal/relay/lease.go b/internal/relay/lease.go index ed2166d4..a63c70fd 100644 --- a/internal/relay/lease.go +++ b/internal/relay/lease.go @@ -59,6 +59,9 @@ type lease struct { sess leasedSession authorize authorizeFunc req auth.Request + // optIns is the session's opt-in time, sent with each revalidate; nil + // for none. + optIns *optInClock mu sync.Mutex timer *time.Timer @@ -72,9 +75,10 @@ type lease struct { } // startLease starts the lease of a session admitted by req, to be revalidated -// every cadence. It returns nil for a zero cadence, which never revalidates; -// otherwise the caller stops it when the session ends first. -func startLease(ctx context.Context, sess leasedSession, authorize authorizeFunc, req auth.Request, cadence time.Duration) *lease { +// every cadence, reporting the opt-in time optIns counts (nil for none). It +// returns nil for a zero cadence, which never revalidates; otherwise the +// caller stops it when the session ends first. +func startLease(ctx context.Context, sess leasedSession, authorize authorizeFunc, req auth.Request, cadence time.Duration, optIns *optInClock) *lease { if cadence <= 0 { return nil } @@ -83,6 +87,7 @@ func startLease(ctx context.Context, sess leasedSession, authorize authorizeFunc sess: sess, authorize: authorize, req: req, + optIns: optIns, cadence: cadence, } l.mu.Lock() @@ -167,6 +172,7 @@ func (l *lease) revalidate() (*auth.Grant, error) { req := l.req req.Event = auth.EventRevalidate req.Bytes = sessionBytes(l.sess.Stats()) + req.OptIns = l.optIns.totals() g, err := l.authorize(l.ctx, req) metricAuthRequests.WithLabelValues(auth.EventRevalidate, authOutcome(g, err)).Inc() return g, err diff --git a/internal/relay/lease_credential_test.go b/internal/relay/lease_credential_test.go index d88ba27b..73b303ad 100644 --- a/internal/relay/lease_credential_test.go +++ b/internal/relay/lease_credential_test.go @@ -42,7 +42,7 @@ func verifiedSession(t *testing.T, v *auth.Verifier, key token.SigningKey) (*fak g, err := v.Authorize(t.Context(), req) require.NoError(t, err) sess := &fakeLeasedSession{} - l := startLease(t.Context(), sess, v.Authorize, req, g.Revalidate()) + l := startLease(t.Context(), sess, v.Authorize, req, g.Revalidate(), nil) require.NotNil(t, l, "every verified session is re-checked") return sess, l } diff --git a/internal/relay/lease_test.go b/internal/relay/lease_test.go index 7fbd7c43..668fd201 100644 --- a/internal/relay/lease_test.go +++ b/internal/relay/lease_test.go @@ -22,7 +22,7 @@ import ( var leaseRequest = auth.Request{ID: "00ff", Event: auth.EventConnect, Path: "/acme", Query: "jwt=h.p.s"} func TestStartLease_NothingToDo(t *testing.T) { - assert.Nil(t, startLease(context.Background(), &fakeLeasedSession{}, (&fakeAuth{}).authorize, leaseRequest, 0)) + assert.Nil(t, startLease(context.Background(), &fakeLeasedSession{}, (&fakeAuth{}).authorize, leaseRequest, 0, nil)) } // TestLease_RevalidateEnds verifies a refused revalidate ends @@ -41,7 +41,7 @@ func TestLease_RevalidateEnds(t *testing.T) { server := &fakeAuth{err: tt.err} sess := &fakeLeasedSession{} sess.setStats(1500, 300) - l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second) + l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second, nil) defer l.stop() time.Sleep(30*time.Second - time.Nanosecond) @@ -67,7 +67,7 @@ func TestLease_RevalidateTakesCadence(t *testing.T) { // The reply's grant differs from connect's; only the cadence matters. server := &fakeAuth{grant: testGrant(t, "other/**", "", time.Minute)} sess := &fakeLeasedSession{} - l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second) + l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second, nil) defer l.stop() time.Sleep(150 * time.Second) @@ -89,7 +89,7 @@ func TestLease_RevalidateUnavailable(t *testing.T) { t.Run(name, func(t *testing.T) { synctest.Test(t, func(t *testing.T) { sess := &fakeLeasedSession{} - l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second) + l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second, nil) defer l.stop() time.Sleep(time.Hour) @@ -103,7 +103,7 @@ func TestLease_RevalidateUnavailable(t *testing.T) { t.Run("retries back off", func(t *testing.T) { synctest.Test(t, func(t *testing.T) { server := &fakeAuth{err: errors.New("503")} - l := startLease(t.Context(), &fakeLeasedSession{}, server.authorize, leaseRequest, 30*time.Second) + l := startLease(t.Context(), &fakeLeasedSession{}, server.authorize, leaseRequest, 30*time.Second, nil) defer l.stop() // Retries follow the first attempt at 30 s with jittered delays of @@ -123,7 +123,7 @@ func TestLease_Stop(t *testing.T) { synctest.Test(t, func(t *testing.T) { server := &fakeAuth{} sess := &fakeLeasedSession{} - l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second) + l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second, nil) time.Sleep(10 * time.Second) l.stop() @@ -162,7 +162,7 @@ func TestLease_RevalidateReportsBytes(t *testing.T) { synctest.Test(t, func(t *testing.T) { server := &fakeAuth{grant: testGrant(t, "", "**", 30*time.Second)} sess := &fakeLeasedSession{} - l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second) + l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second, nil) defer l.stop() sess.setStats(100, 10) @@ -219,7 +219,7 @@ func TestLease_RevalidateWithoutCadenceStops(t *testing.T) { synctest.Test(t, func(t *testing.T) { server := &fakeAuth{grant: testGrant(t, "", "**", 0)} sess := &fakeLeasedSession{} - l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second) + l := startLease(t.Context(), sess, server.authorize, leaseRequest, 30*time.Second, nil) defer l.stop() time.Sleep(time.Hour) diff --git a/internal/relay/optin_time.go b/internal/relay/optin_time.go new file mode 100644 index 00000000..4e4834af --- /dev/null +++ b/internal/relay/optin_time.go @@ -0,0 +1,89 @@ +package relay + +import ( + "sync" + "time" + + "github.com/qumo-dev/gomoqt/moqt" + + "github.com/qumo-dev/qumo/internal/optin" + "github.com/qumo-dev/qumo/token" +) + +// optInClock adds up how long a session keeps broadcasts opted into each +// egress: how long each announcement of its beneath an opt-in directory +// (/.abr/live/cam) lasts, summed over its announcements, so two broadcasts +// opted into ABR for a minute each are two minutes. The session's usage +// reports carry it (auth.Request.OptIns), for a control plane to bill an +// opt-in by the time it was used. A nil clock counts nothing. +// +// It counts what the relay received from the session: an announcement the +// session's grant does not cover never reaches it, so an opt-in the +// session's key does not hold is never counted. +type optInClock struct { + mu sync.Mutex + tallies map[token.OptIn]*optInTally +} + +// optInTally is one egress's time: done, from announcements that ended, +// plus live announcements since since, the last time live changed. +type optInTally struct { + done time.Duration + live int + since time.Time +} + +func newOptInClock() *optInClock { + return &optInClock{tallies: map[token.OptIn]*optInTally{}} +} + +// track counts ann, an announcement the session made, until it ends, when it +// opts a broadcast into an egress; any other is not counted, nor one that +// ended already (moqt runs AfterFunc for it later, on another goroutine, +// which would count the time until then). +func (c *optInClock) track(ann *moqt.Announcement) { + if c == nil || !ann.IsActive() { + return + } + o, ok := optin.Of(ann.BroadcastPath()) + if !ok { + return + } + c.change(o, +1) + ann.AfterFunc(func() { c.change(o, -1) }) +} + +// change adds delta to o's live announcements, counting the time they were +// live until now. +func (c *optInClock) change(o token.OptIn, delta int) { + c.mu.Lock() + defer c.mu.Unlock() + now := time.Now() + t := c.tallies[o] + if t == nil { + t = &optInTally{} + c.tallies[o] = t + } + t.done += time.Duration(t.live) * now.Sub(t.since) + t.live += delta + t.since = now +} + +// totals returns how long the session has kept broadcasts opted into each +// egress so far, live announcements counted up to now, or nil for none. +func (c *optInClock) totals() map[token.OptIn]time.Duration { + if c == nil { + return nil + } + c.mu.Lock() + defer c.mu.Unlock() + if len(c.tallies) == 0 { + return nil + } + now := time.Now() + out := make(map[token.OptIn]time.Duration, len(c.tallies)) + for o, t := range c.tallies { + out[o] = t.done + time.Duration(t.live)*now.Sub(t.since) + } + return out +} diff --git a/internal/relay/optin_time_test.go b/internal/relay/optin_time_test.go new file mode 100644 index 00000000..9fcf3786 --- /dev/null +++ b/internal/relay/optin_time_test.go @@ -0,0 +1,74 @@ +package relay + +import ( + "context" + "testing" + "testing/synctest" + "time" + + "github.com/qumo-dev/gomoqt/moqt" + "github.com/stretchr/testify/assert" + + "github.com/qumo-dev/qumo/token" +) + +// announce returns a live announcement of path and the function that ends it. +func announce(t *testing.T, path moqt.BroadcastPath) (*moqt.Announcement, func()) { + t.Helper() + ann, end := moqt.NewAnnouncement(context.Background(), path) + t.Cleanup(end) + return ann, end +} + +func TestOptInClock(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + c := newOptInClock() + assert.Nil(t, c.totals(), "nothing opted in yet") + + cam1, endCam1 := announce(t, "/.abr/live/cam1") + cam2, endCam2 := announce(t, "/.abr/live/cam2") + hls, _ := announce(t, "/.hls/live/cam1") + plain, _ := announce(t, "/live/cam1") + c.track(cam1) + c.track(plain) + time.Sleep(10 * time.Second) + c.track(cam2) + c.track(hls) + time.Sleep(5 * time.Second) + + assert.Equal(t, map[token.OptIn]time.Duration{ + token.OptInABR: 20 * time.Second, // cam1 15 s, cam2 5 s + token.OptInHLS: 5 * time.Second, + }, c.totals(), "live announcements count up to now; a broadcast's own announcement is not an opt-in") + + endCam1() + time.Sleep(10 * time.Second) + endCam2() + time.Sleep(time.Hour) + + assert.Equal(t, map[token.OptIn]time.Duration{ + token.OptInABR: 30 * time.Second, // cam1 0–15 s, cam2 10–25 s + token.OptInHLS: time.Hour + 15*time.Second, + }, c.totals(), "an ended announcement stops counting") + }) +} + +func TestOptInClock_EndedAnnouncement(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + c := newOptInClock() + ann, end := announce(t, "/.abr/live/cam1") + end() + + c.track(ann) + time.Sleep(time.Minute) + + assert.Nil(t, c.totals(), "an announcement that ended before it was tracked counts nothing") + }) +} + +func TestOptInClock_Nil(t *testing.T) { + var c *optInClock + ann, _ := announce(t, "/.abr/live/cam1") + c.track(ann) + assert.Nil(t, c.totals(), "an unchecked session counts nothing") +} diff --git a/internal/relay/server.go b/internal/relay/server.go index 5aac4784..c33ecc5f 100644 --- a/internal/relay/server.go +++ b/internal/relay/server.go @@ -275,7 +275,7 @@ func (s *Server) serveUpgrade(w http.ResponseWriter, r *http.Request, upgrader h // so the session counted at connect is closed. if g != nil && !a.served.Load() { s.sessionEnds.add() - s.reportEnd(r.Context(), req, moqt.SessionStats{}, endUpgradeFailed) + s.reportEnd(r.Context(), req, moqt.SessionStats{}, nil, endUpgradeFailed) s.sessionEnds.done() } } @@ -809,7 +809,11 @@ func (s *Server) serveSession(sess *moqt.Session) { a := admissionFrom(sess.Context()) checked := a != nil && a.grant != nil var l *lease + // optIns counts the opt-in time of a checked session, whose usage is + // reported; nil counts none. + var optIns *optInClock if checked { + optIns = newOptInClock() a.served.Store(true) // Registered first so that it runs last: after the close below, // with the session's final byte totals and close cause. Counted in @@ -817,7 +821,7 @@ func (s *Server) serveSession(sess *moqt.Session) { s.sessionEnds.add() defer func() { defer s.sessionEnds.done() - s.reportEnd(sess.Context(), a.req, sess.Stats(), sessionEndReason(a, l, context.Cause(sess.Context()))) + s.reportEnd(sess.Context(), a.req, sess.Stats(), optIns, sessionEndReason(a, l, context.Cause(sess.Context()))) }() } // Registered before the close below, so that it runs after it, with the @@ -836,7 +840,7 @@ func (s *Server) serveSession(sess *moqt.Session) { }() defer sess.CloseWithError(moqt.NoError, moqt.NoError.String()) if checked { - if l = startLease(sess.Context(), sess, s.Authorize, a.req, a.grant.Revalidate()); l != nil { + if l = startLease(sess.Context(), sess, s.Authorize, a.req, a.grant.Revalidate(), optIns); l != nil { defer l.stop() } if a.grant.Anonymous() { @@ -874,6 +878,8 @@ func (s *Server) serveSession(sess *moqt.Session) { return } + optIns.track(ann) + slog.Debug("relay: received announcement", "node", s.Config.NodeID, "broadcast_path", ann.BroadcastPath(), diff --git a/usage/usage.go b/usage/usage.go index acc52f60..d434a585 100644 --- a/usage/usage.go +++ b/usage/usage.go @@ -9,6 +9,10 @@ // outnumber publishers by orders of magnitude, so a record per viewer would // make a receiver's load grow with the audience. // +// A publishing session's records also carry how long it kept broadcasts +// opted into each egress (OptInSeconds), so that a control plane can bill an +// option such as ABR by the time it was used. +// // Byte counts are cumulative and from the relay's point of view (what it // received is ingress), so a receiver can take a resent record without // counting it twice. A relay that restarts begins a new Run, whose viewer @@ -46,6 +50,16 @@ const ( EgressBytes = "gateway.egress_bytes" ) +// OptInSeconds names the metric of how long a publishing session kept +// broadcasts opted into the egress named optIn ("hls", "abr"), in whole +// seconds, added up over its broadcasts: two broadcasts opted into ABR for +// a minute each are 120. A session that opted none in carries none. +// +// OptInSeconds("abr") is "gateway.opt_in_seconds.abr" +func OptInSeconds(optIn string) string { + return "gateway.opt_in_seconds." + optIn +} + // Why a publishing session ended, the Reason of its SessionClose. const ( // ReasonRefused is a session the relay ended because a re-check refused diff --git a/usage/usage_test.go b/usage/usage_test.go index 56b41b7b..6f266f6e 100644 --- a/usage/usage_test.go +++ b/usage/usage_test.go @@ -46,3 +46,8 @@ func TestRecord_Validate(t *testing.T) { }) } } + +func TestOptInSeconds(t *testing.T) { + assert.Equal(t, "gateway.opt_in_seconds.abr", OptInSeconds("abr")) + assert.Equal(t, "gateway.opt_in_seconds.hls", OptInSeconds("hls")) +}