Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **`PEERS` names are resolved again while the relay runs, so peers behind a name join and leave without a restart; an entry without a port is an SRV name (`_moqt._udp.hubs.example.com`) (`internal/relay`).** A `PEERS` entry that is neither `host:port` nor a name without a port now stops the relay at startup, and a host that doesn't resolve is retried instead of being dialed by name.
- **The relay re-reads its certificates while it runs (`internal/relay`).** About once a minute it reads `CERT_FILE`/`KEY_FILE`, `PEER_CERT_FILE`/`PEER_KEY_FILE` and `CA_FILE` and compares them with what it holds; what changed is taken from the next TLS handshake on, so cert-manager, a Vault or step-ca agent, or any job that writes the files needs no restart. A public certificate is taken once it parses and matches its key; a peer certificate and the CA together, once they pass the startup checks (issued by the CA, the peering name, valid now). Anything else is logged, once per distinct error rather than at every read, and the certificates in use are kept; renewal tools should replace the files atomically (write, then rename). Sessions already established are untouched. The CA is now checked at each handshake on both sides: as the dialed relay (`GetConfigForClient`) and as the dialer (`VerifyConnection` against the current CA, with the same checks crypto/tls made). The relay's key-set and usage requests are signed with the renewed peer certificate (`reqsign.Signer.Rotate`), and `qumo dns` re-reads its `PEER_CERT_FILE`/`PEER_KEY_FILE` on the same one-minute cadence and signs its feed reads with a renewed pair (`internal/certreload`, shared by both). New metrics: `qumo_relay_cert_reloads_total{cert,result}` and `qumo_relay_cert_not_after_seconds{cert}`, and `qumo_dns_cert_reloads_total{cert,result}` and `qumo_dns_cert_not_after_seconds{cert}`; alert on the `not_after` gauges. A CA is rotated by putting the new one beside the old in `CA_FILE` first ([TLS & mTLS](docs/site/content/en/docs/deployment/tls.md#certificate-renewal)).
- **A relay reports how long each publishing session kept broadcasts opted into HLS and ABR (`internal/relay`, `usage.OptInSeconds`).** A publishing session's `usage` and `session_close` records now carry `gateway.opt_in_seconds.abr` and `gateway.opt_in_seconds.hls`: the whole seconds its announcements beneath `/.abr/` and `/.hls/` lasted, added up over its broadcasts, from when the relay received each until it ended, the session closed or a revoked ticket ended it. Only an opt-in the session's grant covers reaches the relay, so the time is billed to the key that holds the opt-in, and a control plane can charge for ABR or HLS by the time it was used. `qumo abr` itself never needs to know whose broadcast it transcodes. A session that opted nothing into an egress carries no metric for it, so a receiver that doesn't bill opt-ins sees records as before.
- **RTMP and RTSP ingest report each pushed stream's usage under its publisher's key (`internal/ingest`, `QUMO_USAGE_URL`).** With `QUMO_USAGE_URL` and `QUMO_AUTH_KEYS`, `qumo rtmp` and `qumo rtsp-push` send each stream to the usage URL as a relay sends a publishing session (`internal/meter`): a `session_open`, a `usage` record every 30 s and a `session_close`, under the kid of the token that granted the push, carrying the media the ingest took in (`usage.IngestMediaBytes`, `ingest.media_bytes`: the payload of every frame) and how long it kept the broadcast opted into each egress (`gateway.opt_in_seconds.*`). The relay sees only the ingest's own session, under the ingest's key, so a stream pushed over RTMP or RTSP is billed to its publisher here, as one published over MoQ is billed on the relay. `ingest.PublishAuthorizer` now returns a `Permission`, the lapse and the kid of the key that granted it. The opt-in clock moved from the relay to `internal/optin` (`optin.Clock`), which both use.

### Changed

Expand Down
14 changes: 13 additions & 1 deletion docs/site/content/en/docs/cli/rtmp.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,26 @@ listener checks no subscriber, so with auth on, viewers go through the relay.
$ RELAY_URL=moqt://relay:4433 RELAY_SIGNING_KEY=ingest-key.jwk RELAY_PUBLISH=/live/ qumo rtmp
```

## Usage

With `QUMO_USAGE_URL` (and `QUMO_AUTH_KEYS`), the server reports each pushed
stream as a relay reports a publishing session ([usage reports]({{< relref "../configuration#usage-reports" >}})),
under the key of the token that granted the push: a `session_open`, a `usage`
record every 30 s and a `session_close`, carrying the media it took in
(`ingest.media_bytes`, the payload of every frame) and how long it kept its
broadcast opted into each egress (`gateway.opt_in_seconds.abr`,
`gateway.opt_in_seconds.hls`). The relay sees the server's own session, under
the server's key, so the stream is billed here, to its publisher.

## Configuration

| Variable | Default | Description |
|---|---|---|
| `RTMP_INGEST_ADDR` | `:1935` | RTMP listen address. |
| `RTMP_SERVE_ADDR` | `:4433`; none with `RELAY_URL` | MoQT listen address for subscribers to dial directly. |
| `QUMO_AUTH_KEYS` | (unset) | The key set a publisher's credential is verified against, as the relay reads it (a file or URL). Unset accepts every publish. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL and to `QUMO_USAGE_URL`. |
| `QUMO_USAGE_URL` | (unset) | Where each stream's usage is reported, under the key of the publisher's token ([usage](#usage)). Needs `QUMO_AUTH_KEYS`. |
| `INGEST_OPT_IN` | (unset) | The egresses (`hls`, `abr`) a stream is opted into on its publisher's behalf, comma-separated: with auth on, those a second `?jwt=` may publish the opt-in of. |
| `RELAY_URL` | (unset) | The relay to publish through, dialed as a client (`https://…` or `moqt://…`); it may carry a credential as `?jwt=`. |
| `RELAY_SIGNING_KEY`, `RELAY_PUBLISH` | (unset) | Set together: a signing key (`qumo auth keygen`) the server signs a fresh relay credential with on every dial and renews on the session, granting publishing beneath the directory `RELAY_PUBLISH`. A push outside it is refused. |
Expand Down
14 changes: 13 additions & 1 deletion docs/site/content/en/docs/cli/rtsp-push.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,14 +89,26 @@ listener checks no subscriber, so with auth on, viewers go through the relay.
$ RELAY_URL=moqt://relay:4433 RELAY_SIGNING_KEY=ingest-key.jwk RELAY_PUBLISH=/live/ qumo rtsp-push
```

## Usage

With `QUMO_USAGE_URL` (and `QUMO_AUTH_KEYS`), the server reports each pushed
stream as a relay reports a publishing session ([usage reports]({{< relref "../configuration#usage-reports" >}})),
under the key of the token that granted the push: a `session_open`, a `usage`
record every 30 s and a `session_close`, carrying the media it took in
(`ingest.media_bytes`, the payload of every frame) and how long it kept its
broadcast opted into each egress (`gateway.opt_in_seconds.abr`,
`gateway.opt_in_seconds.hls`). The relay sees the server's own session, under
the server's key, so the stream is billed here, to its publisher.

## Configuration

| Variable | Default | Description |
|---|---|---|
| `RTSP_INGEST_ADDR` | `:8554` | RTSP listen address. |
| `RTSP_SERVE_ADDR` | `:4433`; none with `RELAY_URL` | MoQT listen address for subscribers to dial directly. |
| `QUMO_AUTH_KEYS` | (unset) | The key set a publisher's credential is verified against, as the relay reads it (a file or URL). Unset accepts every publish. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL. |
| `QUMO_AUTH_KEYS_CACHE` / `QUMO_RELAY_TOKEN` | (unset) | A file a downloaded key set is kept in, and the bearer token sent to a key-set URL and to `QUMO_USAGE_URL`. |
| `QUMO_USAGE_URL` | (unset) | Where each stream's usage is reported, under the key of the publisher's token ([usage](#usage)). Needs `QUMO_AUTH_KEYS`. |
| `INGEST_OPT_IN` | (unset) | The egresses (`hls`, `abr`) a stream is opted into on its publisher's behalf, comma-separated: with auth on, those a second `?jwt=` may publish the opt-in of. |
| `RELAY_URL` | (unset) | The relay to publish through, dialed as a client (`https://…` or `moqt://…`); it may carry a credential as `?jwt=`. |
| `RELAY_SIGNING_KEY`, `RELAY_PUBLISH` | (unset) | Set together: a signing key (`qumo auth keygen`) the server signs a fresh relay credential with on every dial and renews on the session, granting publishing beneath the directory `RELAY_PUBLISH`. A push outside it is refused. |
Expand Down
2 changes: 1 addition & 1 deletion docs/site/content/en/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ Byte counts are cumulative and from the relay's point of view (what it received

**Sessions that publish a broadcast are reported one by one,** as the first three records: an open, usage every 30 s, and a close. A session publishes when its token has a `publish` pattern, or from when it is granted a ticket that has one: its open comes then, and what it moved as a viewer up to its last 30 s check stays in its key's viewer total, the rest counting as its usage.

**A publishing session's records also carry its opt-in time:** how long it kept broadcasts opted into each egress, `gateway.opt_in_seconds.abr` for [`qumo abr`](../cli/abr/) and `gateway.opt_in_seconds.hls` for [`qumo hls`](../cli/hls/), in whole seconds, added up over its broadcasts (two broadcasts opted into ABR for a minute each are 120). It is the time each announcement of the session beneath `/.abr/` or `/.hls/` lasted on this relay, from when the relay received it until it ended, the session closed or a revoked ticket ended it, so a control plane can bill an opt-in by the time it was used. Only an opt-in the session's grant covers reaches the relay, so an opt-in a key doesn't hold is never counted. An egress the session opted nothing into has no metric. A broadcast opted in through [RTMP or RTSP ingest](../cli/rtmp/#opting-streams-into-hls-and-abr) counts on the ingest's session, not the publisher's.
**A publishing session's records also carry its opt-in time:** how long it kept broadcasts opted into each egress, `gateway.opt_in_seconds.abr` for [`qumo abr`](../cli/abr/) and `gateway.opt_in_seconds.hls` for [`qumo hls`](../cli/hls/), in whole seconds, added up over its broadcasts (two broadcasts opted into ABR for a minute each are 120). It is the time each announcement of the session beneath `/.abr/` or `/.hls/` lasted on this relay, from when the relay received it until it ended, the session closed or a revoked ticket ended it, so a control plane can bill an opt-in by the time it was used. Only an opt-in the session's grant covers reaches the relay, so an opt-in a key doesn't hold is never counted. An egress the session opted nothing into has no metric. A broadcast opted in through [RTMP or RTSP ingest](../cli/rtmp/#opting-streams-into-hls-and-abr) counts on the ingest's session, under the ingest's own key; the ingest reports the stream itself under its publisher's key ([rtmp → usage](../cli/rtmp/#usage)), so a control plane bills the publisher's key and leaves the ingest's out, as it does any key of its own.

**Every other session is a viewer, and viewers are reported together.** Viewers outnumber publishers by orders of magnitude, and a record per viewer would make the receiver's load grow with the audience. Their bytes are added up per key into one running total since the relay started, sent as a `viewers` record naming the key and the relay's `run`. The `run` is new each time the relay starts, so a restart begins new totals rather than lowering old ones. There is no open or close for viewers, and a total is sent only when it has changed.

Expand Down
7 changes: 2 additions & 5 deletions internal/auth/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"time"

"github.com/qumo-dev/qumo/internal/meter"
"github.com/qumo-dev/qumo/internal/optin"
"github.com/qumo-dev/qumo/internal/reqsign"
"github.com/qumo-dev/qumo/token"
"github.com/qumo-dev/qumo/usage"
Expand Down Expand Up @@ -170,11 +171,7 @@ func usageMetrics(b Bytes) map[string]int64 {
// long it kept broadcasts opted into each egress, in whole seconds.
func publisherMetrics(b Bytes, optIns map[token.OptIn]time.Duration) map[string]int64 {
m := usageMetrics(b)
for o, d := range optIns {
if secs := int64(d / time.Second); secs > 0 {
m[usage.OptInSeconds(string(o))] = secs
}
}
optin.AddSeconds(m, optIns)
return m
}

Expand Down
30 changes: 20 additions & 10 deletions internal/ingest/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,22 @@ import (

// PublishAuthorizer decides whether a pushed stream may publish the broadcast
// at path with credential, a ?jwt= the publisher presented ("" for none),
// and returns when that permission lapses: the stream is cut then. The zero
// time never lapses. An error refuses the stream.
// and returns the permission it grants. An error refuses the stream.
//
// A stream's opt-ins (see [optIn]) are asked for as well, with path the
// opt-in's, such as /.abr/live/cam for /live/cam.
type PublishAuthorizer func(credential string, path moqt.BroadcastPath) (lapses time.Time, err error)
type PublishAuthorizer func(credential string, path moqt.BroadcastPath) (Permission, error)

// A Permission is what a PublishAuthorizer grants a push.
type Permission struct {
// Lapses is when the permission lapses: the stream is cut then. The zero
// time never lapses.
Lapses time.Time
// KeyID is the kid of the key that signed the credential granting it,
// whose usage the push is (see [RTMPConfig.Meter]), or "" when none was
// checked.
KeyID string
}

// splitCredential splits a name such as "cam?jwt=A&jwt=B" into the name and
// the credentials its query carries as jwt, in order, none for none.
Expand Down Expand Up @@ -65,21 +75,21 @@ func checkPushPath(path moqt.BroadcastPath) error {
// credentials, in order, and returns the first permission granted. Without
// credentials it asks for a publish that presented none. When none is
// granted it returns the first refusal.
func authorizeAny(authorize PublishAuthorizer, credentials []string, path moqt.BroadcastPath) (time.Time, error) {
func authorizeAny(authorize PublishAuthorizer, credentials []string, path moqt.BroadcastPath) (Permission, error) {
if len(credentials) == 0 {
return authorize("", path)
}
var first error
for _, credential := range credentials {
lapses, err := authorize(credential, path)
p, err := authorize(credential, path)
if err == nil {
return lapses, nil
return p, nil
}
if first == nil {
first = err
}
}
return time.Time{}, first
return Permission{}, first
}

// optIn opts sess's broadcast into each of optIns, the egresses the server
Expand All @@ -91,15 +101,15 @@ func authorizeAny(authorize PublishAuthorizer, credentials []string, path moqt.B
func optIn(sess *Session, authorize PublishAuthorizer, credentials []string, optIns []token.OptIn) {
for _, o := range optIns {
path := optin.Path(o.Directory(), sess.path)
var lapses time.Time
var granted Permission
if authorize != nil {
var err error
if lapses, err = authorizeAny(authorize, credentials, path); err != nil {
if granted, err = authorizeAny(authorize, credentials, path); err != nil {
slog.Info("ingest opt-in refused", "broadcast_path", sess.path, "opt_in", o, "error", err)
continue
}
}
sess.OptIn(o, lapses)
sess.OptIn(o, granted.Lapses)
}
}

Expand Down
18 changes: 11 additions & 7 deletions internal/ingest/auth_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,11 +82,11 @@ func TestAuthorizeAny(t *testing.T) {
}
for name, tt := range tests {
t.Run(name, func(t *testing.T) {
fake := &fakePublishAuthorizer{answer: func(ask publishAsk) (time.Time, error) {
fake := &fakePublishAuthorizer{answer: func(ask publishAsk) (Permission, error) {
if err := tt.answers[ask.credential]; err != nil {
return time.Time{}, err
return Permission{}, err
}
return lapses, nil
return Permission{Lapses: lapses, KeyID: ask.credential}, nil
}}

got, err := authorizeAny(fake.authorize, tt.credentials, "/live/cam")
Expand All @@ -101,7 +101,8 @@ func TestAuthorizeAny(t *testing.T) {
return
}
require.NoError(t, err)
assert.Equal(t, lapses, got)
assert.Equal(t, Permission{Lapses: lapses, KeyID: tt.wantAsked[len(tt.wantAsked)-1]}, got,
"the permission of the credential that granted it")
})
}
}
Expand Down Expand Up @@ -169,7 +170,8 @@ func TestPublishAuthorizer(t *testing.T) {
authorize := publishAuthorizer(tt.verifier, tt.beneath)
require.NotNil(t, authorize)

lapses, err := authorize(tt.credential, tt.path)
granted, err := authorize(tt.credential, tt.path)
lapses := granted.Lapses

switch {
case tt.wantErr != nil:
Expand All @@ -181,6 +183,7 @@ func TestPublishAuthorizer(t *testing.T) {
case tt.wantLapse:
require.NoError(t, err)
assert.WithinDuration(t, time.Now().Add(time.Minute), lapses, 5*time.Second, "cut when the credential expires")
assert.Equal(t, tt.verifier == v, granted.KeyID == key.ID, "the push is the usage of the key that signed it")
case tt.verifier == nil:
require.NoError(t, err)
assert.True(t, lapses.IsZero(), "auth off never cuts")
Expand Down Expand Up @@ -219,9 +222,10 @@ func TestPublishAuthorizer_OptIn(t *testing.T) {
}

t.Run("a key that holds the opt-in", func(t *testing.T) {
lapses, err := publishAuthorizer(paid, "/live/")(sign(paidKey), "/.abr/live/cam")
granted, err := publishAuthorizer(paid, "/live/")(sign(paidKey), "/.abr/live/cam")
require.NoError(t, err)
assert.WithinDuration(t, time.Now().Add(time.Minute), lapses, 5*time.Second)
assert.WithinDuration(t, time.Now().Add(time.Minute), granted.Lapses, 5*time.Second)
assert.Equal(t, paidKey.ID, granted.KeyID)
})
t.Run("a key that does not", func(t *testing.T) {
_, err := publishAuthorizer(unpaid, "/live/")(sign(unpaidKey), "/.abr/live/cam")
Expand Down
Loading
Loading