Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **The playground opts a broadcast into ABR, and plays ABR broadcasts adaptively (`playground`).** The publish board's ABR box announces the broadcast's opt-in beneath `/.abr/` beside it, as the HLS scenario does beneath `/.hls/`. A broadcast whose video tracks share an alternate group is played adaptively: the viewer switches renditions at group boundaries, stepping down when frames fall behind their media time to what the rate measured since then carries, and up one rendition after a calm stretch. A step up keeps the old rendition until its last group before the switch has arrived; a step down drops it at once. The subscribe board picks a rendition by hand or automatically, and shows the rendition playing, the receive rate and the lag.
- **`PEERS` names are resolved again while the relay runs, so peers behind a name join and leave without a restart; an entry without a port is an SRV name (`_moqt._udp.hubs.example.com`) (`internal/relay`).** A `PEERS` entry that is neither `host:port` nor a name without a port now stops the relay at startup, and a host that doesn't resolve is retried instead of being dialed by name.
- **The relay re-reads its certificates while it runs (`internal/relay`).** About once a minute it reads `CERT_FILE`/`KEY_FILE`, `PEER_CERT_FILE`/`PEER_KEY_FILE` and `CA_FILE` and compares them with what it holds; what changed is taken from the next TLS handshake on, so cert-manager, a Vault or step-ca agent, or any job that writes the files needs no restart. A public certificate is taken once it parses and matches its key; a peer certificate and the CA together, once they pass the startup checks (issued by the CA, the peering name, valid now). Anything else is logged, once per distinct error rather than at every read, and the certificates in use are kept; renewal tools should replace the files atomically (write, then rename). Sessions already established are untouched. The CA is now checked at each handshake on both sides: as the dialed relay (`GetConfigForClient`) and as the dialer (`VerifyConnection` against the current CA, with the same checks crypto/tls made). The relay's key-set and usage requests are signed with the renewed peer certificate (`reqsign.Signer.Rotate`), and `qumo dns` re-reads its `PEER_CERT_FILE`/`PEER_KEY_FILE` on the same one-minute cadence and signs its feed reads with a renewed pair (`internal/certreload`, shared by both). New metrics: `qumo_relay_cert_reloads_total{cert,result}` and `qumo_relay_cert_not_after_seconds{cert}`, and `qumo_dns_cert_reloads_total{cert,result}` and `qumo_dns_cert_not_after_seconds{cert}`; alert on the `not_after` gauges. A CA is rotated by putting the new one beside the old in `CA_FILE` first ([TLS & mTLS](docs/site/content/en/docs/deployment/tls.md#certificate-renewal)).
- **A relay reports how long each publishing session kept broadcasts opted into HLS and ABR (`internal/relay`, `usage.OptInSeconds`).** A publishing session's `usage` and `session_close` records now carry `gateway.opt_in_seconds.abr` and `gateway.opt_in_seconds.hls`: the whole seconds its announcements beneath `/.abr/` and `/.hls/` lasted, added up over its broadcasts, from when the relay received each until it ended, the session closed or a revoked ticket ended it. Only an opt-in the session's grant covers reaches the relay, so the time is billed to the key that holds the opt-in, and a control plane can charge for ABR or HLS by the time it was used. `qumo abr` itself never needs to know whose broadcast it transcodes. A session that opted nothing into an egress carries no metric for it, so a receiver that doesn't bill opt-ins sees records as before.

### Changed

Expand Down
5 changes: 5 additions & 0 deletions docs/site/content/en/docs/cli/abr.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ is refused within half a minute, and one on air ends at the next 30 s check of
live sessions. HLS is held the same way (`"hls"`), and a key whose entry lists
no `optIns` holds none.

A relay with `QUMO_USAGE_URL` reports how long each opt-in lasted, in the
publishing session's usage records under the tenant's key
(`gateway.opt_in_seconds.abr`, see [usage reports]({{< relref "../configuration#usage-reports" >}})),
so the control plane can bill ABR by the time it was used.

The tenant's app signs the opt-in's token with the same key as the
broadcast's: `/.abr/acme/live/cam1` beside `/acme/live/cam1`, presented on the
publisher's session as a ticket, since a session's credentials all come from
Expand Down
5 changes: 4 additions & 1 deletion docs/site/content/en/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,8 @@ With `QUMO_USAGE_URL` set, the relay POSTs a JSON array of records to it every 1
{"type": "usage", "session_id": "…", "kid": "…",
"metrics": {"gateway.ingress_bytes": 10162, "gateway.egress_bytes": 9223}, "ts": "…"},
{"type": "session_close", "session_id": "…", "kid": "…",
"metrics": {"gateway.ingress_bytes": 10438, "gateway.egress_bytes": 9165}, "reason": "closed", "ts": "…"},
"metrics": {"gateway.ingress_bytes": 10438, "gateway.egress_bytes": 9165,
"gateway.opt_in_seconds.abr": 312}, "reason": "closed", "ts": "…"},
{"type": "viewers", "run": "…", "kid": "…",
"metrics": {"gateway.ingress_bytes": 120, "gateway.egress_bytes": 880412}, "ts": "…"}]
```
Expand All @@ -264,6 +265,8 @@ Byte counts are cumulative and from the relay's point of view (what it received

**Sessions that publish a broadcast are reported one by one,** as the first three records: an open, usage every 30 s, and a close. A session publishes when its token has a `publish` pattern, or from when it is granted a ticket that has one: its open comes then, and what it moved as a viewer up to its last 30 s check stays in its key's viewer total, the rest counting as its usage.

**A publishing session's records also carry its opt-in time:** how long it kept broadcasts opted into each egress, `gateway.opt_in_seconds.abr` for [`qumo abr`](../cli/abr/) and `gateway.opt_in_seconds.hls` for [`qumo hls`](../cli/hls/), in whole seconds, added up over its broadcasts (two broadcasts opted into ABR for a minute each are 120). It is the time each announcement of the session beneath `/.abr/` or `/.hls/` lasted on this relay, from when the relay received it until it ended, the session closed or a revoked ticket ended it, so a control plane can bill an opt-in by the time it was used. Only an opt-in the session's grant covers reaches the relay, so an opt-in a key doesn't hold is never counted. An egress the session opted nothing into has no metric. A broadcast opted in through [RTMP or RTSP ingest](../cli/rtmp/#opting-streams-into-hls-and-abr) counts on the ingest's session, not the publisher's.

**Every other session is a viewer, and viewers are reported together.** Viewers outnumber publishers by orders of magnitude, and a record per viewer would make the receiver's load grow with the audience. Their bytes are added up per key into one running total since the relay started, sent as a `viewers` record naming the key and the relay's `run`. The `run` is new each time the relay starts, so a restart begins new totals rather than lowering old ones. There is no open or close for viewers, and a total is sent only when it has changed.

Records go out in batches of at most 500. A failed send, a 401, 403, 408, 413 or 429 included, keeps the records and retries them (logged once, then again when it recovers); only a batch the receiver can't read (400 or 422) is dropped. When the relay shuts down, it waits up to 5 s for its last sessions to record their end before the final send.
Expand Down
7 changes: 7 additions & 0 deletions internal/auth/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ import (
"errors"
"fmt"
"net/http"
"time"

"github.com/qumo-dev/qumo/token"
)

// maxBody bounds how much of a reply's body is drained.
Expand Down Expand Up @@ -73,6 +76,10 @@ type Request struct {
// Bytes is the session's cumulative byte totals, set on revalidate and
// end.
Bytes Bytes
// OptIns is how long the session has kept broadcasts opted into each
// egress, added up over its broadcasts, set on revalidate and end
// (usage.OptInSeconds). An egress it opted nothing into is absent.
OptIns map[token.OptIn]time.Duration
// Reason is why the session ended, set on end.
Reason string
}
Expand Down
43 changes: 33 additions & 10 deletions internal/auth/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (

"github.com/qumo-dev/qumo/internal/meter"
"github.com/qumo-dev/qumo/internal/reqsign"
"github.com/qumo-dev/qumo/token"
"github.com/qumo-dev/qumo/usage"
)

Expand Down Expand Up @@ -103,10 +104,10 @@ func (r *usageReporter) publishing(id string) {
r.meter.Open(id, s.kid)
}

// revalidated records a session's cumulative bytes at a revalidate and, when
// live is set, that the session is still live. A session it doesn't know is
// left unknown.
func (r *usageReporter) revalidated(id string, b Bytes, live bool) {
// revalidated records a session's cumulative bytes and opt-in time at a
// revalidate and, when live is set, that the session is still live. A
// session it doesn't know is left unknown.
func (r *usageReporter) revalidated(id string, b Bytes, optIns map[token.OptIn]time.Duration, live bool) {
r.mu.Lock()
defer r.mu.Unlock()
s, ok := r.sessions[id]
Expand All @@ -116,18 +117,17 @@ func (r *usageReporter) revalidated(id string, b Bytes, live bool) {
if live {
s.seen = r.now()
}
switch {
case !s.publishes:
if !s.publishes {
s = r.countViewer(s, b)
case b.Sent > s.counted.Sent || b.Received > s.counted.Received:
r.meter.Report(id, s.kid, usageMetrics(b.since(s.counted)))
} else if m := publisherMetrics(b.since(s.counted), optIns); carriesUsage(m) {
r.meter.Report(id, s.kid, m)
}
r.sessions[id] = s
}

// close records a session's end with its final totals and forgets it. It
// reports false for a session it doesn't know.
func (r *usageReporter) close(id string, b Bytes, reason string) bool {
func (r *usageReporter) close(id string, b Bytes, optIns map[token.OptIn]time.Duration, reason string) bool {
r.mu.Lock()
defer r.mu.Unlock()
r.closed[id] = r.now()
Expand All @@ -140,7 +140,7 @@ func (r *usageReporter) close(id string, b Bytes, reason string) bool {
r.countViewer(s, b)
return true
}
r.meter.Close(id, s.kid, usageMetrics(b.since(s.counted)), reason)
r.meter.Close(id, s.kid, publisherMetrics(b.since(s.counted), optIns), reason)
return true
}

Expand All @@ -166,6 +166,29 @@ func usageMetrics(b Bytes) map[string]int64 {
return map[string]int64{usage.IngressBytes: clampInt64(b.Received), usage.EgressBytes: clampInt64(b.Sent)}
}

// publisherMetrics is a publishing session's metrics: its bytes, and how
// long it kept broadcasts opted into each egress, in whole seconds.
func publisherMetrics(b Bytes, optIns map[token.OptIn]time.Duration) map[string]int64 {
m := usageMetrics(b)
for o, d := range optIns {
if secs := int64(d / time.Second); secs > 0 {
m[usage.OptInSeconds(string(o))] = secs
}
}
return m
}

// carriesUsage reports whether m holds any usage: a session that moved
// nothing and kept nothing opted in for a whole second has none to report.
func carriesUsage(m map[string]int64) bool {
for _, v := range m {
if v > 0 {
return true
}
}
return false
}

func clampInt64(v uint64) int64 {
if v > math.MaxInt64 {
return math.MaxInt64
Expand Down
81 changes: 69 additions & 12 deletions internal/auth/usage_viewers_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ import (
"context"
"net/http"
"testing"
"time"

"github.com/qumo-dev/qumo/token"
"github.com/qumo-dev/qumo/usage"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand All @@ -25,9 +27,9 @@ func TestUsageReporter_ViewersAreTotalled(t *testing.T) {
r.open(id, viewerSession("k1"))
}
r.open("other", viewerSession("k2"))
r.revalidated("v1", Bytes{Sent: 100, Received: 10}, true)
r.revalidated("v2", Bytes{Sent: 200, Received: 20}, true)
r.revalidated("other", Bytes{Sent: 7}, true)
r.revalidated("v1", Bytes{Sent: 100, Received: 10}, nil, true)
r.revalidated("v2", Bytes{Sent: 200, Received: 20}, nil, true)
r.revalidated("other", Bytes{Sent: 7}, nil, true)
require.NoError(t, r.meter.Flush(context.Background()))

first := sink.received()[0]
Expand All @@ -49,11 +51,11 @@ func TestUsageReporter_ViewersAreTotalled(t *testing.T) {
})

t.Run("the total grows by what each session moved since it was last counted", func(t *testing.T) {
r.revalidated("v1", Bytes{Sent: 150, Received: 10}, true) // +50
r.revalidated("v1", Bytes{Sent: 120, Received: 10}, true) // a late, lower report adds nothing
assert.True(t, r.close("v2", Bytes{Sent: 260, Received: 25}, "closed")) // +60, +5
r.revalidated("v2", Bytes{Sent: 999}, true) // after its close: ignored
assert.True(t, r.close("v3", Bytes{}, "closed")) // moved nothing
r.revalidated("v1", Bytes{Sent: 150, Received: 10}, nil, true) // +50
r.revalidated("v1", Bytes{Sent: 120, Received: 10}, nil, true) // a late, lower report adds nothing
assert.True(t, r.close("v2", Bytes{Sent: 260, Received: 25}, nil, "closed")) // +60, +5
r.revalidated("v2", Bytes{Sent: 999}, nil, true) // after its close: ignored
assert.True(t, r.close("v3", Bytes{}, nil, "closed")) // moved nothing
require.NoError(t, r.meter.Flush(context.Background()))

batches := sink.received()
Expand All @@ -67,10 +69,10 @@ func TestUsageReporter_ViewerTotalSurvivesAFailedSend(t *testing.T) {
sink := &fakeUsageSink{statuses: []int{http.StatusServiceUnavailable, http.StatusOK}}
r := newTestUsage(t, sink)
r.open("v1", viewerSession("k1"))
r.revalidated("v1", Bytes{Sent: 100}, true)
r.revalidated("v1", Bytes{Sent: 100}, nil, true)

assert.Error(t, r.meter.Flush(context.Background()))
r.revalidated("v1", Bytes{Sent: 160}, true)
r.revalidated("v1", Bytes{Sent: 160}, nil, true)
require.NoError(t, r.meter.Flush(context.Background()))

batches := sink.received()
Expand All @@ -84,7 +86,7 @@ func TestUsageReporter_PublishersStayPerSession(t *testing.T) {
r := newTestUsage(t, sink)
r.open("p1", usageSession{kid: "k1", publishes: true})
r.open("p2", usageSession{kid: "k1", publishes: true})
r.revalidated("p1", Bytes{Received: 50}, true)
r.revalidated("p1", Bytes{Received: 50}, nil, true)
require.NoError(t, r.meter.Flush(context.Background()))

var got []string
Expand All @@ -107,7 +109,7 @@ func TestUsageReporter_Publishing(t *testing.T) {
r.publishing("v1")
r.publishing("p1")
r.publishing("unknown")
r.revalidated("v1", Bytes{Received: 40}, true)
r.revalidated("v1", Bytes{Received: 40}, nil, true)
require.NoError(t, r.meter.Flush(context.Background()))

var got []string
Expand All @@ -132,3 +134,58 @@ func TestBytes_Since(t *testing.T) {
})
}
}

// A publishing session's usage carries how long it kept broadcasts opted
// into each egress, in whole seconds, and is reported for it even when the
// session moved no bytes; its close carries the final time. A viewer's
// total carries none.
func TestUsageReporter_OptInSeconds(t *testing.T) {
sink := &fakeUsageSink{}
r := newTestUsage(t, sink)
r.open("p1", usageSession{kid: "k1", publishes: true})
r.open("v1", viewerSession("k1"))
abr := func(d time.Duration) map[token.OptIn]time.Duration {
return map[token.OptIn]time.Duration{token.OptInABR: d}
}

r.revalidated("p1", Bytes{Received: 100}, abr(1500*time.Millisecond), true)
r.revalidated("v1", Bytes{Sent: 10}, abr(time.Hour), true)
require.NoError(t, r.meter.Flush(context.Background()))
first := byType(sink.received()[0])
assert.Equal(t, map[string]int64{
usage.IngressBytes: 100, usage.EgressBytes: 0, usage.OptInSeconds("abr"): 1,
}, first[usage.Usage].Metrics, "whole seconds")
assert.NotContains(t, first[usage.Viewers].Metrics, usage.OptInSeconds("abr"), "a viewer opts nothing in")

r.open("p2", usageSession{kid: "k1", publishes: true})
r.revalidated("p2", Bytes{}, abr(400*time.Millisecond), true)
require.NoError(t, r.meter.Flush(context.Background()))
require.Len(t, sink.received(), 2)
assert.NotContains(t, byType(sink.received()[1]), usage.Usage,
"no bytes and under a second opted in is nothing to report")

r.revalidated("p2", Bytes{}, abr(2*time.Second), true)
require.NoError(t, r.meter.Flush(context.Background()))
require.Len(t, sink.received(), 3)
assert.Equal(t, int64(2), byType(sink.received()[2])[usage.Usage].Metrics[usage.OptInSeconds("abr")],
"opt-in time without bytes is reported")

assert.True(t, r.close("p1", Bytes{Received: 120}, map[token.OptIn]time.Duration{
token.OptInABR: 3 * time.Second, token.OptInHLS: 500 * time.Millisecond,
}, usage.ReasonClosed))
require.NoError(t, r.meter.Flush(context.Background()))
closed := byType(sink.received()[3])[usage.SessionClose]
assert.Equal(t, map[string]int64{
usage.IngressBytes: 120, usage.EgressBytes: 0, usage.OptInSeconds("abr"): 3,
}, closed.Metrics, "under a second of HLS is left out")
assert.NoError(t, closed.Validate())
}

// byType returns batch's records by type, the last of each.
func byType(batch []usage.Record) map[usage.Type]usage.Record {
out := map[usage.Type]usage.Record{}
for _, rec := range batch {
out[rec.Type] = rec
}
return out
}
8 changes: 4 additions & 4 deletions internal/auth/verifier.go
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ func (v *Verifier) Authorize(_ context.Context, req Request) (*Grant, error) {
// A refused session is ending; one that couldn't be checked
// lives on, and the relay asks again.
_, refused := errors.AsType[RefusedError](err)
v.usage.revalidated(req.ID, req.Bytes, !refused)
v.usage.revalidated(req.ID, req.Bytes, req.OptIns, !refused)
}
}
return g, err
Expand All @@ -157,7 +157,7 @@ func (v *Verifier) Authorize(_ context.Context, req Request) (*Grant, error) {
// counted once a ticket binds it to a key (Bind).
func (v *Verifier) anonymous(req Request) *Grant {
if v.usage != nil && req.Event == EventRevalidate {
v.usage.revalidated(req.ID, req.Bytes, true)
v.usage.revalidated(req.ID, req.Bytes, req.OptIns, true)
}
return &Grant{revalidate: revalidateEvery, anonymous: true}
}
Expand Down Expand Up @@ -302,11 +302,11 @@ func publishesBroadcast(g token.Grant) bool {
return len(g.Publish) > 0
}

// End reports a session's end with its final byte totals.
// End reports a session's end with its final byte totals and opt-in time.
func (v *Verifier) End(_ context.Context, req Request) error {
if v.usage == nil {
return nil
}
v.usage.close(req.ID, req.Bytes, req.Reason)
v.usage.close(req.ID, req.Bytes, req.OptIns, req.Reason)
return nil
}
Loading
Loading