Skip to content

egress: revise the egress policy without restarting the node #188

Description

@CMGS

Every input to the guarded-egress data plane is frozen when pool.NewManager runs, so widening an allow-list or admitting one internal prefix costs a node restart.

  • egress_internal_allow is closed over by the dialer's Control hook at sandboxd/pool/pool.go:379 (newEgressDialer(parsePrefixes(cfg.EgressInternalAllow))).
  • Pool and tenant policies are built once into m.poolEgress / m.tenantEgress at sandboxd/pool/pool.go:429-430.
  • PUT /v1/pools deliberately refuses egress (sandboxd/pool/setpools.go:25-28): "egress is config-owned; accepting it here would silently drop it".

Observed while standing up a desktop fleet: admitting a single host prefix, so the guest's browser could reach a node-side service, meant editing the config file and restarting sandboxd, which has to be scheduled against an idle pool. The restart is not destructive to the pool — Reconcile re-arms live claims with the new policy (sandboxd/pool/reconcile.go:191) — but it drops in-flight egress connections and execs, which is a heavy price for one CIDR.

The proxy side is already shaped for this. Evaluator is an interface (sandboxd/egress/policy.go:143) and Proxy holds only the interface (sandboxd/egress/proxy.go:66), so a holder whose Eval* methods read an atomic.Pointer[Policy] turns a revision into a pointer swap with no change to the proxy at all. Both maps are already read under m.mu (sandboxd/pool/egress.go:203,274), so they can be written without a new lock.

Hot-path cost: zero on the claim path. One atomic load per proxied request and per dialed connection, both below noise.

Three things do not reduce to a pointer swap:

  1. Socks5 decides whether the SOCKS5 door is bound, both at refill pre-bind (sandboxd/pool/egress.go:209) and at arm (sandboxd/pool/egress.go:174). Flipping it has to bind or close a socket on live sandboxes and invalidate pre-bound doors.
  2. Intercept only works if the guest trusts the node CA, which is an image property today — os-image/desktop: deliver the node's egress CA into the guest trust stores #187. Until the CA reaches the guest, interception cannot be toggled at runtime whatever the policy layer permits.
  3. Pool and tenant policies are not part of ClusterDigest (sandboxd/config/config.go:280-295 covers tokens, CA fingerprint, tenant names and checkpoint TTL). A claim redirected to a peer is evaluated against that peer's policy, so a node-local revision diverges across a mesh with nothing reporting it.

Proposal, in order:

A scoped SIGHUP reload first. Re-read the config file, validate it, and apply only the egress layer — egress_internal_allow, per-pool and per-tenant egress, secret references — ignoring every other field. This keeps the config file as the single source of truth, so no policy overlay over the config seed is needed (the shape pools.json has today), adds no new authorization surface, and propagates across a mesh by whatever already distributes the config.

A write API only if a control plane needs one. An endpoint that widens an allow-list at runtime is a privilege path that leaves no trace in the config file; it needs its own audit trail and an answer for mesh convergence, and it reverses a decision the code states explicitly. The holder above is what such an API would build on, so deferring it costs nothing.

Tests: a swap observed by an in-flight proxy, the socks door binding and unbinding under a live claim, and a reload that fails validation leaving the previous policy in force. Then a testbed check that a newly admitted prefix takes effect on an already-claimed sandbox without a restart.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions