Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **The playground opts a broadcast into ABR, and plays ABR broadcasts adaptively (`playground`).** The publish board's ABR box announces the broadcast's opt-in beneath `/.abr/` beside it, as the HLS scenario does beneath `/.hls/`. A broadcast whose video tracks share an alternate group is played adaptively: the viewer switches renditions at group boundaries, stepping down when frames fall behind their media time to what the rate measured since then carries, and up one rendition after a calm stretch. A step up keeps the old rendition until its last group before the switch has arrived; a step down drops it at once. The subscribe board picks a rendition by hand or automatically, and shows the rendition playing, the receive rate and the lag.
- **`PEERS` names are resolved again while the relay runs, so peers behind a name join and leave without a restart; an entry without a port is an SRV name (`_moqt._udp.hubs.example.com`) (`internal/relay`).** A `PEERS` entry that is neither `host:port` nor a name without a port now stops the relay at startup, and a host that doesn't resolve is retried instead of being dialed by name.
- **The relay re-reads its certificates while it runs (`internal/relay`).** About once a minute it reads `CERT_FILE`/`KEY_FILE`, `PEER_CERT_FILE`/`PEER_KEY_FILE` and `CA_FILE` and compares them with what it holds; what changed is taken from the next TLS handshake on, so cert-manager, a Vault or step-ca agent, or any job that writes the files needs no restart. A public certificate is taken once it parses and matches its key; a peer certificate and the CA together, once they pass the startup checks (issued by the CA, the peering name, valid now). Anything else is logged, once per distinct error rather than at every read, and the certificates in use are kept; renewal tools should replace the files atomically (write, then rename). Sessions already established are untouched. The CA is now checked at each handshake on both sides: as the dialed relay (`GetConfigForClient`) and as the dialer (`VerifyConnection` against the current CA, with the same checks crypto/tls made). The relay's key-set and usage requests are signed with the renewed peer certificate (`reqsign.Signer.Rotate`), and `qumo dns` re-reads its `PEER_CERT_FILE`/`PEER_KEY_FILE` on the same one-minute cadence and signs its feed reads with a renewed pair (`internal/certreload`, shared by both). New metrics: `qumo_relay_cert_reloads_total{cert,result}` and `qumo_relay_cert_not_after_seconds{cert}`, and `qumo_dns_cert_reloads_total{cert,result}` and `qumo_dns_cert_not_after_seconds{cert}`; alert on the `not_after` gauges. A CA is rotated by putting the new one beside the old in `CA_FILE` first ([TLS & mTLS](docs/site/content/en/docs/deployment/tls.md#certificate-renewal)).
- **A relay reports how long each publishing session kept broadcasts opted into HLS and ABR (`internal/relay`, `usage.OptInSeconds`).** A publishing session's `usage` and `session_close` records now carry `gateway.opt_in_seconds.abr` and `gateway.opt_in_seconds.hls`: the whole seconds its announcements beneath `/.abr/` and `/.hls/` lasted, added up over its broadcasts, from when the relay received each until it ended, the session closed or a revoked ticket ended it. Only an opt-in the session's grant covers reaches the relay, so the time is billed to the key that holds the opt-in, and a control plane can charge for ABR or HLS by the time it was used. `qumo abr` itself never needs to know whose broadcast it transcodes. A session that opted nothing into an egress carries no metric for it, so a receiver that doesn't bill opt-ins sees records as before.
- **RTMP and RTSP ingest report each pushed stream's usage under its publisher's key (`internal/ingest`, `QUMO_USAGE_URL`).** With `QUMO_USAGE_URL` and `QUMO_AUTH_KEYS`, `qumo rtmp` and `qumo rtsp-push` send each stream to the usage URL as a relay sends a publishing session (`internal/meter`): a `session_open`, a `usage` record every 30 s and a `session_close`, under the kid of the token that granted the push, carrying the media the ingest took in (`usage.IngestMediaBytes`, `ingest.media_bytes`: the payload of every frame) and how long it kept the broadcast opted into each egress (`gateway.opt_in_seconds.*`). The relay sees only the ingest's own session, under the ingest's key, so a stream pushed over RTMP or RTSP is billed to its publisher here, as one published over MoQ is billed on the relay. `ingest.PublishAuthorizer` now returns a `Permission`, the lapse and the kid of the key that granted it. The opt-in clock moved from the relay to `internal/optin` (`optin.Clock`), which both use.

### Changed

- **Usage reporting is a package of its own, for every server to use (`internal/meter`).** What sends usage records (`usage.Record`) to `QUMO_USAGE_URL` is no longer the relay's: `meter.Reporter` takes a server's sessions (`Open`, `Report` with cumulative metrics, `Close`) and running totals per key (`Total`), and keeps them in memory, coalesces each session's usage, batches by 500, retries through an outage, signs the requests when given a signer and sends what is left at shutdown, as the relay did. The relay reports through it, with what it decides itself (a publisher by itself, viewers in one total per key) kept in `internal/auth`. The records the relay sends are unchanged.

### Fixed

Expand Down
5 changes: 5 additions & 0 deletions docs/site/content/en/docs/cli/abr.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ is refused within half a minute, and one on air ends at the next 30 s check of
live sessions. HLS is held the same way (`"hls"`), and a key whose entry lists
no `optIns` holds none.

A relay with `QUMO_USAGE_URL` reports how long each opt-in lasted, in the
publishing session's usage records under the tenant's key
(`gateway.opt_in_seconds.abr`, see [usage reports]({{< relref "../configuration#usage-reports" >}})),
so the control plane can bill ABR by the time it was used.

The tenant's app signs the opt-in's token with the same key as the
broadcast's: `/.abr/acme/live/cam1` beside `/acme/live/cam1`, presented on the
publisher's session as a ticket, since a session's credentials all come from
Expand Down
14 changes: 13 additions & 1 deletion docs/site/content/en/docs/cli/rtmp.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,26 @@ listener checks no subscriber, so with auth on, viewers go through the relay.
$ RELAY_URL=moqt://relay:4433 RELAY_SIGNING_KEY=ingest-key.jwk RELAY_PUBLISH=/live/ qumo rtmp
```

## Usage

With `QUMO_USAGE_URL` (and `QUMO_AUTH_KEYS`), the server reports each pushed
stream as a relay reports a publishing session ([usage reports]({{< relref "../configuration#usage-reports" >}})),
under the key of the token that granted the push: a `session_open`, a `usage`
record every 30 s and a `session_close`, carrying the media it took in
(`ingest.media_bytes`, the payload of every frame) and how long it kept its
broadcast opted into each egress (`gateway.opt_in_seconds.abr`,
`gateway.opt_in_seconds.hls`). The relay sees the server's own session, under
the server's key, so the stream is billed here, to its publisher.

## Configuration

| Variable | Default | Description |
|---|---|---|
| `RTMP_INGEST_ADDR` | `:1935` | RTMP listen address. |
| `RTMP_SERVE_ADDR` | `:4433`; none with `RELAY_URL` | MoQT listen address for subscribers to dial directly. |
| `QUMO_AUTH_KEYS` | (unset) | The key set a publisher's credential is verified against, as the relay reads it (a file or URL). Unset accepts every publish. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL and to `QUMO_USAGE_URL`. |
| `QUMO_USAGE_URL` | (unset) | Where each stream's usage is reported, under the key of the publisher's token ([usage](#usage)). Needs `QUMO_AUTH_KEYS`. |
| `INGEST_OPT_IN` | (unset) | The egresses (`hls`, `abr`) a stream is opted into on its publisher's behalf, comma-separated: with auth on, those a second `?jwt=` may publish the opt-in of. |
| `RELAY_URL` | (unset) | The relay to publish through, dialed as a client (`https://…` or `moqt://…`); it may carry a credential as `?jwt=`. |
| `RELAY_SIGNING_KEY`, `RELAY_PUBLISH` | (unset) | Set together: a signing key (`qumo auth keygen`) the server signs a fresh relay credential with on every dial and renews on the session, granting publishing beneath the directory `RELAY_PUBLISH`. A push outside it is refused. |
Expand Down
14 changes: 13 additions & 1 deletion docs/site/content/en/docs/cli/rtsp-push.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,14 +89,26 @@ listener checks no subscriber, so with auth on, viewers go through the relay.
$ RELAY_URL=moqt://relay:4433 RELAY_SIGNING_KEY=ingest-key.jwk RELAY_PUBLISH=/live/ qumo rtsp-push
```

## Usage

With `QUMO_USAGE_URL` (and `QUMO_AUTH_KEYS`), the server reports each pushed
stream as a relay reports a publishing session ([usage reports]({{< relref "../configuration#usage-reports" >}})),
under the key of the token that granted the push: a `session_open`, a `usage`
record every 30 s and a `session_close`, carrying the media it took in
(`ingest.media_bytes`, the payload of every frame) and how long it kept its
broadcast opted into each egress (`gateway.opt_in_seconds.abr`,
`gateway.opt_in_seconds.hls`). The relay sees the server's own session, under
the server's key, so the stream is billed here, to its publisher.

## Configuration

| Variable | Default | Description |
|---|---|---|
| `RTSP_INGEST_ADDR` | `:8554` | RTSP listen address. |
| `RTSP_SERVE_ADDR` | `:4433`; none with `RELAY_URL` | MoQT listen address for subscribers to dial directly. |
| `QUMO_AUTH_KEYS` | (unset) | The key set a publisher's credential is verified against, as the relay reads it (a file or URL). Unset accepts every publish. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL and to `QUMO_USAGE_URL`. |
| `QUMO_USAGE_URL` | (unset) | Where each stream's usage is reported, under the key of the publisher's token ([usage](#usage)). Needs `QUMO_AUTH_KEYS`. |
| `INGEST_OPT_IN` | (unset) | The egresses (`hls`, `abr`) a stream is opted into on its publisher's behalf, comma-separated: with auth on, those a second `?jwt=` may publish the opt-in of. |
| `RELAY_URL` | (unset) | The relay to publish through, dialed as a client (`https://…` or `moqt://…`); it may carry a credential as `?jwt=`. |
| `RELAY_SIGNING_KEY`, `RELAY_PUBLISH` | (unset) | Set together: a signing key (`qumo auth keygen`) the server signs a fresh relay credential with on every dial and renews on the session, granting publishing beneath the directory `RELAY_PUBLISH`. A push outside it is refused. |
Expand Down
5 changes: 4 additions & 1 deletion docs/site/content/en/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,8 @@ With `QUMO_USAGE_URL` set, the relay POSTs a JSON array of records to it every 1
{"type": "usage", "session_id": "…", "kid": "…",
"metrics": {"gateway.ingress_bytes": 10162, "gateway.egress_bytes": 9223}, "ts": "…"},
{"type": "session_close", "session_id": "…", "kid": "…",
"metrics": {"gateway.ingress_bytes": 10438, "gateway.egress_bytes": 9165}, "reason": "closed", "ts": "…"},
"metrics": {"gateway.ingress_bytes": 10438, "gateway.egress_bytes": 9165,
"gateway.opt_in_seconds.abr": 312}, "reason": "closed", "ts": "…"},
{"type": "viewers", "run": "…", "kid": "…",
"metrics": {"gateway.ingress_bytes": 120, "gateway.egress_bytes": 880412}, "ts": "…"}]
```
Expand All @@ -264,6 +265,8 @@ Byte counts are cumulative and from the relay's point of view (what it received

**Sessions that publish a broadcast are reported one by one,** as the first three records: an open, usage every 30 s, and a close. A session publishes when its token has a `publish` pattern, or from when it is granted a ticket that has one: its open comes then, and what it moved as a viewer up to its last 30 s check stays in its key's viewer total, the rest counting as its usage.

**A publishing session's records also carry its opt-in time:** how long it kept broadcasts opted into each egress, `gateway.opt_in_seconds.abr` for [`qumo abr`](../cli/abr/) and `gateway.opt_in_seconds.hls` for [`qumo hls`](../cli/hls/), in whole seconds, added up over its broadcasts (two broadcasts opted into ABR for a minute each are 120). It is the time each announcement of the session beneath `/.abr/` or `/.hls/` lasted on this relay, from when the relay received it until it ended, the session closed or a revoked ticket ended it, so a control plane can bill an opt-in by the time it was used. Only an opt-in the session's grant covers reaches the relay, so an opt-in a key doesn't hold is never counted. An egress the session opted nothing into has no metric. A broadcast opted in through [RTMP or RTSP ingest](../cli/rtmp/#opting-streams-into-hls-and-abr) counts on the ingest's session, under the ingest's own key; the ingest reports the stream itself under its publisher's key ([rtmp → usage](../cli/rtmp/#usage)), so a control plane bills the publisher's key and leaves the ingest's out, as it does any key of its own.

**Every other session is a viewer, and viewers are reported together.** Viewers outnumber publishers by orders of magnitude, and a record per viewer would make the receiver's load grow with the audience. Their bytes are added up per key into one running total since the relay started, sent as a `viewers` record naming the key and the relay's `run`. The `run` is new each time the relay starts, so a restart begins new totals rather than lowering old ones. There is no open or close for viewers, and a total is sent only when it has changed.

Records go out in batches of at most 500. A failed send, a 401, 403, 408, 413 or 429 included, keeps the records and retries them (logged once, then again when it recovers); only a batch the receiver can't read (400 or 422) is dropped. When the relay shuts down, it waits up to 5 s for its last sessions to record their end before the final send.
Expand Down
29 changes: 7 additions & 22 deletions internal/auth/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ package auth
import (
"errors"
"fmt"
"net"
"net/http"
"net/url"
"time"

"github.com/qumo-dev/qumo/token"
)

// maxBody bounds how much of a reply's body is drained.
Expand Down Expand Up @@ -60,26 +61,6 @@ func RefusalStatus(err error) int {
return http.StatusServiceUnavailable
}

// checkURL accepts an https URL, or http only on a loopback host: the bearer
// token the relay sends to it must not cross a network in the clear.
func checkURL(rawURL string) error {
u, err := url.Parse(rawURL)
if err != nil {
return err
}
switch u.Scheme {
case "https":
case "http":
host := u.Hostname()
if ip := net.ParseIP(host); host != "localhost" && (ip == nil || !ip.IsLoopback()) {
return fmt.Errorf("http is allowed only for a loopback host, got %q", host)
}
default:
return fmt.Errorf("want an https URL, got scheme %q", u.Scheme)
}
return nil
}

// Request is one session event: what the relay asks its Authorize about at
// connect and revalidate, and reports to its End.
type Request struct {
Expand All @@ -95,6 +76,10 @@ type Request struct {
// Bytes is the session's cumulative byte totals, set on revalidate and
// end.
Bytes Bytes
// OptIns is how long the session has kept broadcasts opted into each
// egress, added up over its broadcasts, set on revalidate and end
// (usage.OptInSeconds). An egress it opted nothing into is absent.
OptIns map[token.OptIn]time.Duration
// Reason is why the session ended, set on end.
Reason string
}
Expand Down
24 changes: 0 additions & 24 deletions internal/auth/auth_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,30 +9,6 @@ import (
"github.com/stretchr/testify/assert"
)

func TestCheckURL(t *testing.T) {
tests := map[string]struct {
url string
wantErrText string
}{
"https": {url: "https://keys.example.com/v1/set"},
"loopback http": {url: "http://127.0.0.1:4440/"},
"localhost http": {url: "http://localhost:4440/"},
"http off loopback": {url: "http://keys.example.com/", wantErrText: "loopback"},
"another scheme": {url: "ftp://keys.example.com/", wantErrText: "https"},
}
for name, tt := range tests {
t.Run(name, func(t *testing.T) {
err := checkURL(tt.url)

if tt.wantErrText != "" {
assert.ErrorContains(t, err, tt.wantErrText)
return
}
assert.NoError(t, err)
})
}
}

func TestRefusalStatus(t *testing.T) {
tests := map[string]struct {
err error
Expand Down
2 changes: 1 addition & 1 deletion internal/auth/keysource.go
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ func (u *urlKeySource) touchCache(now time.Time) {
}

func newURLKeySource(rawURL, bearer string, signer *reqsign.Signer) (*urlKeySource, error) {
if err := checkURL(rawURL); err != nil {
if err := reqsign.CheckURL(rawURL); err != nil {
return nil, err
}
return &urlKeySource{url: rawURL, token: bearer, signer: signer, client: reqsign.NoRedirectClient(keysFetchTimeout)}, nil
Expand Down
4 changes: 2 additions & 2 deletions internal/auth/signed_requests_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ func TestVerifier_SignsItsRequests(t *testing.T) {

require.NoError(t, v.source.refresh(t.Context(), &store, time.Now()))
v.usage.open("s1", testUsageSession)
assert.Error(t, v.usage.flush(t.Context()))
require.NoError(t, v.usage.flush(t.Context()))
assert.Error(t, v.usage.meter.Flush(t.Context()))
require.NoError(t, v.usage.meter.Flush(t.Context()))

require.Len(t, keyReqs.all(), 1)
assertSigned(t, keyReqs.all()[0], "relay-token")
Expand Down
Loading
Loading