Migrate ShellWatch backend to Go

[rado0x54]

Summary

Migrate the ShellWatch backend from Node.js/TypeScript (Fastify + ssh2 + Drizzle + MCP TS SDK) to Go. The SvelteKit frontend is unaffected — it's a static SPA and the HTTP/WS/MCP wire protocols would be preserved.

The primary driver is SSH library quality, specifically OpenSSH user-certificate support (*-cert-v01@openssh.com) and the webauthn-sk-* keytype family. Secondary drivers are operational: single static binary, simpler self-hosting, and aligning the backend with the language already used by agent-client/.

Design accepted (2026-07): the full architecture specification for the Go backend lives at docs/go-backend-architecture.md. It freezes every dependency choice to a single winner (see the updated table below), maps the Node backend's accidental complexity to concrete design responses, rules on parity item K (byte-based output offsets), and replaces the loose "next steps" with a six-phase, golden-gated migration plan. This issue remains the tracking issue; the spec is the source of truth for how. Implementation happens on the long-lived go-backend base branch (PRs target it; it lands in develop at cutover). Pre-implementation prep is tracked in #232.

Primary driver: SSH library

ssh2 (Node, current)

We carry a fork (rado0x54/ssh2#shellwatch) and would need to invest in extending it for the cert work tracked in #209. Concrete gaps in ssh2 today:

• No parsing of the OpenSSH certificate structure (everything after the embedded pubkey is discarded).
• authPK sends the bare pubkey blob when the keytype says -cert-v01; servers reject as malformed.
• No client API to pair a private key with a *-cert.pub.
• Cert-suffix regex doesn't include sk-* keytypes (so webauthn-sk-ecdsa-sha2-nistp256-cert-v01@openssh.com doesn't even parse).
• Upstream mscdex/ssh2 has the same gaps and is largely dormant — the fork is a permanent maintenance cost.

golang.org/x/crypto/ssh (Go, proposed)

The de-facto SSH library for the entire SSH-as-infrastructure category (Teleport, Smallstep, HashiCorp Boundary, Tailscale tsnet, gliderlabs/ssh, etc.). Specifically for our needs:

• Full client + server certificate support. ssh.Certificate parses every field (serial, keyId, validPrincipals, validAfter/before, criticalOptions, extensions, signatureKey, signature). ssh.NewCertSigner(cert, signer) produces a client signer that presents the cert correctly; ssh.CertChecker does server-side validation against a CA, principals, validity windows, and KRLs.
• Custom signers via the public ssh.Signer interface. The webauthn-sk-* keytype isn't natively recognized for signature verification upstream yet (see "Known gap" below), but on the client path — which is what ShellWatch needs first — you implement ssh.Signer returning the right Format and the webauthn extra (flags|counter|origin|clientData|extensions) in ssh.Signature.Rest. The library transmits Rest verbatim; real OpenSSH on the remote validates it because it knows the keytype. No fork required.

Reference: hullarb/ssheasy implements webauthn-sk-* SSH client auth in the browser via Go/WASM on unmodified golang.org/x/crypto v0.26.0 (no replace directive, no fork). The webauthnSigner in web/webauth.go is the pattern we'd transplant.

For certificate auth wrapping a webauthn-sk-* key (the webauthn-sk-ecdsa-sha2-nistp256-cert-v01@openssh.com keytype we want to lead with), the composition ssh.NewCertSigner(cert, webauthnSigner) should work — the cert layer is keytype-agnostic and Rest-passes through. Worth validating early in a spike.

Known gap to track (not a blocker for migration)

golang/go#71095 (stdlib crypto/webauthn proposal) is stalled — last activity March 2025, no Go-team champion, scope unresolved between minimal signature verification vs. full attestation parsing. This is the dependency for native webauthn-sk-* verification inside x/crypto/ssh (#69999).

This only bites if ShellWatch ever wants to be an SSH server that accepts webauthn-sk-* keys directly from clients (e.g., a future custom sshd for the demo box, in lieu of fronting with real OpenSSH). For the foreseeable architecture — ShellWatch as SSH client brokering to real OpenSSH servers — the gap doesn't apply.

Secondary drivers

• Single static binary deploy. No node_modules, no tsx/build pipeline for the server, no two-language container. Better fit for the self-host story.
• Concurrency model. Goroutines + channels match the multi-session, multi-actor fan-out (WS subscribers + MCP subscribers + audit log) more naturally than EventEmitters + async iterators.
• Existing in-house competence. agent-client/ is already Go (MIT-licensed). One backend language reduces context-switching and means changes that span agent + server stay in one mental model.
• Performance. Honestly not the primary motivation — Node handles the current load fine. The Go win is operational (deployment, memory footprint, startup time), not raw throughput.

Dependency mapping — decided (single winners)

All alternatives narrowed to one winner each; maintenance status verified July 2026. Full rationale + rejected alternatives in docs/go-backend-architecture.md §2.

Concern	Current (Node)	Go — decided
HTTP server	Fastify	go-chi/chi v5 (net/http-native; zero lock-in; pairs with oapi-codegen)
REST types + router	hand-written Fastify routes	oapi-codegen v2, strict-server, chi target — generated from docs/api/openapi.yaml, committed + go generate + CI drift check (see "Generate the REST surface" below)
WebSocket	@fastify/websocket (ws)	coder/websocket — gorilla/websocket went dormant again after the 2023 revival (last release 2024-06); coder is the only actively maintained option
SSH client	ssh2 (forked)	golang.org/x/crypto/ssh (no fork)
SSH agent protocol (server side, /agent-proxy)	ssh2 fork AgentProtocol	golang.org/x/crypto/ssh/agent (agent.ServeAgent + our ExtendedAgent; same module, no new dep)
MCP server	@modelcontextprotocol/sdk (TS)	github.com/modelcontextprotocol/go-sdk ≥ v1.6 (stable API guarantee; streamable HTTP + per-session state + notifications confirmed)
SQLite driver	better-sqlite3	modernc.org/sqlite (pure Go → static binary + cross-compile; PocketBase's default driver)
ORM / DB	Drizzle	sqlc (typed Go from plain SQL, zero runtime dep; queries double as persistence docs)
DB migrations	drizzle-kit	pressly/goose (library-first, go:embed + goose.Up at startup; more active than golang-migrate)
WebAuthn	@simplewebauthn/server	github.com/go-webauthn/webauthn (mature, used by Teleport; protocol pkg exposes the raw COSE/authenticator data we need for webauthn-sk-* derivation; v0.x — pin)
Web Push	web-push	SherClockHolmes/webpush-go (row was missing here; replaceable behind the notification-channel seam)
Config validation	zod	hand-rolled Validate() in internal/config — zod's cross-field refinements don't map to struct tags; schema is small; zero deps (go-playground/validator rejected: thin maintainership, tag DSL)
YAML loader	yaml	goccy/go-yaml — ⚠️ correction: gopkg.in/yaml.v3 was archived in 2025 and is unmaintained
Logging	pino/Fastify	log/slog (stdlib; settled consensus)
Testing	Vitest	stdlib testing + testify v1
In-process SSH server (tests)	ssh2.Server	gliderlabs/ssh (dormant but stable; test-only dep, eyes open; Tailscale/Charm derivatives exist as fallback)
Hydra admin client	hand-written fetch wrapper	hand-written on net/http (~12 endpoints; generated Ory SDK rejected as overweight)
Static frontend serving	@fastify/static	go:embed of the SvelteKit build (single self-contained binary; -static-dir dev override)
Custom SSH server (future, optional)	n/a	gliderlabs/ssh (unchanged, still future)

What stays unchanged

• SvelteKit frontend. Static SPA, talks to the backend via REST + WS + MCP. Wire protocols preserved → frontend ships unchanged.
• xterm.js. Browser-only.
• SQLite schema. SQL is portable; migrations move to goose (embedded, run at startup) but the schema itself is untouched.
• Architecture concepts. TerminalManager, AgentSession, transport abstraction, OutputBuffer, pending-action store, audit log — these are designs, not code; they translate directly.
• HTTP/WS/MCP API surface. No breaking changes to clients (web UI, MCP agents, agent-client binary).
• Auth (Ory Hydra). OAuth2/OIDC stays delegated to Hydra exactly as today (Replace OAuth shim + API keys with Ory Hydra as the single auth authority #217) — an unchanged external service. It is deliberately not a dependency-swap line item: the Go backend keeps the same role (Hydra's passkey login/consent provider + bearer introspection), so the src/hydra/ glue is a faithful port, not a redesign.
• agent-client/. Already Go, no change.

Parity foundation (done — #225)

The pre-migration prep is complete, so the rewrite starts with a language-agnostic parity oracle rather than a blank slate:

• Frozen wire contract. docs/api/openapi.yaml (REST, 45 paths + discovery/DCR), docs/api/websocket-protocol.md, docs/api/mcp-tools.md — the source of truth every Go handler must reproduce, validated in CI (pnpm api:lint). Known intentional inconsistencies are catalogued (A–J) as "converge-or-preserve" decisions for the port.
• Golden characterization fixtures. src/test/integration/__goldens__/*.json capture normalized real responses across REST, discovery, MCP tools, WS frames, audit pagination, and the WebAuthn ceremony envelopes. The Go server replays the same requests, applies the same normalization, and diffs against the same files — a direct parity check in either language.
• WebAuthn behavior pinned. A fake authenticator (test(webauthn): add fake-authenticator integration tests for the WebAuthn ceremonies #162/test(webauthn): fake authenticator + ceremony-level integration tests (#162) #228) drives real attestation/assertion crypto through every ceremony endpoint, and the ceremony response shapes are goldened (test(golden): WebAuthn ceremony response-envelope parity fixtures #229). De-risks the go-webauthn port — the exact wire behavior (UV enforcement, counter, OpenSSH webauthn-sk-* derivation) is captured.
• Full integration coverage of every REST/WS surface (test(integration): close #225 item 3 — full route coverage (WS control, actions, push, accounts) #230), so no path ships to Go uncharacterized.

Net: the acceptance gate for the rewrite is "the Go server passes the frozen contract + reproduces the goldens," not a subjective feature-parity review.

Generate the REST surface (don't hand-transcribe it)

Now that docs/api/openapi.yaml is frozen, generate the Go REST scaffold from it instead of hand-writing 45 routes:

• Tool: oapi-codegen. From the spec it emits request/response types, a chi ServerInterface (handler signatures to implement), route registration, and optional spec-driven request-validation middleware. Aligns with the chi choice above.
• Payoff: the request/response structs match the contract by construction — a whole class of drift (the kind we hit with the step-up enum and the label/host gap) can't recur. The Go work collapses to "implement the generated interface + wire in the business logic."
• Anti-drift loop: commit the generated code and run codegen under go generate; CI fails if the output drifts from the spec — the Go-side equivalent of pnpm api:lint.

Honest caveats:

• REST only. WS (/ws) and MCP (/mcp) aren't expressible in OpenAPI — those stay hand-written against websocket-protocol.md / mcp-tools.md, backed by their goldens.
• Loosely-modeled bodies stay loose. The WebAuthn ceremony options (additionalProperties: true) and credential: {} generate as map[string]any / any — fine, since they're @simplewebauthn / go-webauthn passthroughs anyway.
• Scaffold, not logic. Handler bodies (SSH brokering, Hydra calls, store access) are still hand-written; codegen only fixes the surface.
• Reproduces the A–J inconsistencies as-is — which is exactly what parity wants; converge later as a deliberate code+spec change, not a silent codegen "fix."

This makes "generate + green the REST surface against its goldens" a concrete, early implementation slice — a natural first PR after the de-risk spike.

Risks / open questions

• MCP Go SDK. The official github.com/modelcontextprotocol/go-sdk provides the streamable-HTTP transport, so this is no longer considered a top schedule risk. Remaining validation is behavioral, not existential: confirm per-client stateful sessions + debounced notifications match today's output against the MCP goldens (test(golden): cross-language parity-oracle characterization tests (#225 item 2) #227).
• Hydra provider port. Since Replace OAuth shim + API keys with Ory Hydra as the single auth authority #217, ShellWatch implements no OAuth server — it's Hydra's passkey login/consent provider + bearer introspection (src/hydra/). The Go work re-implements that provider surface + a Hydra admin-API client; the discovery docs, /api/hydra/* routes, and their goldens already pin the contract. Simpler than adopting fosite, and there's no bespoke token machinery to port.
• Test-helper rewrite. The in-process integration harness (ssh2 server, Fastify app, MCP client, WS client) must be re-built on the Go toolchain — but Pre-Go-migration: freeze wire contract + parity test infrastructure #225 turned this from a liability into an asset: the golden fixtures + frozen contract are language-agnostic, so the Go tests assert against the same normalized JSON the Node server produces, not hand-ported expectations. Rebuild expected net simpler (gliderlabs/ssh + httptest).
• The 71095 stall noted above — track but don't block on.
• Team velocity during transition. The hosted product would freeze on new features for the duration of the rewrite. Need to scope and time-box. Decided (2026-07): the freeze is on. New features pause for the duration; bugfixes land on develop as usual and are mirrored on go-backend once the affected surface is ported (Phase 3+).

Next steps (implementation not yet scheduled)

The architecture spec (docs/go-backend-architecture.md) supersedes the earlier loose list — its §9 defines the full six-phase plan, each phase gated on its golden-fixture subset. The entry point when work starts:

1. Phase 0 — de-risk spike. x/crypto/ssh client doing a passkey-backed login via a custom ssh.Signer against a real OpenSSH host; ssh.NewCertSigner(cert, webauthnSigner) against an sshd with TrustedUserCAKeys using a webauthn-sk-ecdsa-sha2-nistp256-cert-v01@openssh.com cert from a test CA; MCP Go SDK serving one tool to the unchanged SvelteKit client.
2. Phase 1 — skeleton. Go module at repo root, config, store + goose, oapi-codegen pipeline + CI drift check, embedded static serving — and the golden replay harness proven against the Node server first (validates the harness and the WS-chunking normalization assumption before any Go handler exists).
3. Phases 2–6. Auth plane → terminal core → signing + agents → periphery → side-by-side soak and cutover (spec §9). The feature-parity gate is objective: the frozen contract (docs/api/) + goldens (Pre-Go-migration: freeze wire contract + parity test infrastructure #225) are the acceptance criteria, surface by surface. Contract convergence (A–J) is a deliberate post-cutover track.

Related

• Support SSH user certificates (client-side cert auth + CA issuance) #209 — Support SSH user certificates (this issue removes the ssh2-fork-extension path as the way to deliver Support SSH user certificates (client-side cert auth + CA issuance) #209)
• Replace OAuth shim + API keys with Ory Hydra as the single auth authority #217 — Replace OAuth shim + API keys with Ory Hydra (auth is now delegated; the Go backend is Hydra's login/consent provider + introspection client, not an OAuth server)
• Pre-Go-migration: freeze wire contract + parity test infrastructure #225 — Pre-migration prep: frozen wire contract + golden parity oracle (the acceptance gate for this rewrite) — done
• Go rewrite prep: operationIds, agent-proxy wire contract, Go CI job, spike environment (#210) #232 — Pre-implementation prep: operationIds for codegen, /agent-proxy wire contract, Go CI job, Phase 0 spike environment

[rado0x54]

This is not planned in the immediate future. We just want to highlight the tradeoffs and help in the architectural decision process.

[rado0x54]

Phase 0 (de-risk spike) complete — all three goals validated on the go-backend branch (spike/go/, commit 8e0fd42), against real OpenSSH 10.3:

1. ✅ Custom ssh.Signer producing webauthn-sk-ecdsa signatures on unmodified x/crypto/ssh v0.53 — fake-authenticator assertion (Go port of test(webauthn): add fake-authenticator integration tests for the WebAuthn ceremonies #162/test(webauthn): fake authenticator + ceremony-level integration tests (#162) #228's construction), Signature.Blob = mpint R‖S, .Rest = flags‖counter‖origin‖clientDataJSON‖extensions per PROTOCOL.u2f.
2. ✅ ssh.NewCertSigner(cert, webauthnSigner) authenticated against sshd with TrustedUserCAKeys — sshd log: Accepted publickey … ECDSA-SK-CERT … ID spike-webauthn … CA ED25519. Notably, no custom ssh.PublicKey type was needed: ssh-keygen canonicalizes the cert to sk-ecdsa-…-cert-v01 (natively parsed), so the signer's pubkey is just cert.Key and only the signature format is custom. The ssh2 fork's job is reproducible with zero forks — the core premise of this issue holds.
3. ✅ MCP go-sdk v1.6.1 streamable-HTTP server + client round-trip (one tool, per-session state).

Findings recorded in spike/README.md, the sharpest one: sshd's webauthn check byte-compares a literal clientDataJSON prefix ({"type":…,"challenge":…,"origin":…) rather than parsing JSON — key order is load-bearing (Go's json.Marshal(map) sorts keys and can never match). Challenge is base64url-no-pad, matching browser output.

Next per the spec (§9): Phase 1 — skeleton (root go.mod, config, store + goose, oapi-codegen pipeline — the CI gate from #232 already runs — embedded static serving, and the golden replay harness proven against the Node server first).

[rado0x54]

Phase 1 (skeleton) complete on go-backend (88f9280, 987f57c):

• Root module github.com/rado0x54/shellwatch (Go 1.26) coexisting with the Node tree; spike/go stays its own module.
• internal/api generated by pinned oapi-codegen v2.7.1 from the frozen spec (committed, go generate + CI drift gate — the two remaining Go rewrite prep: operationIds, agent-proxy wire contract, Go CI job, spike environment (#210) #232 item-3 sub-tasks are now done: full Go build/vet/test job + committed-code drift check replace the pre-module smoke script).
• internal/config: faithful port of the zod schema + loader (defaults, refinements, keyDirectory/SPA-redirect derivations, [user@]host[:port] parsing — bug-compatible IPv6 behavior included, tested).
• internal/store: modernc SQLite (WAL, FK on, single-writer), goose baseline migration captured byte-for-byte from the Node schema after drizzle 0000–0010, sqlc query pipeline proven (tests cover migrate-idempotency, defaults, FK + CHECK enforcement). Toolchain finding: sqlc's query rewriter miscomputes offsets on multi-byte characters in .sql files — keep them ASCII (noted in the queries file).
• cmd/shellwatch: chi skeleton serving GET /health and GET /api/version (golden-matching bodies, verified live), embedded SPA via go:embed with -static-dir dev override, config load + DB migrate on boot, signal-driven ordered shutdown.
• Golden harness proven against the Node server first, as the plan requires: internal/golden ports the golden.ts normalization; (1) idempotency holds over all committed fixtures, (2) live replay of the six stateless goldens (health, both 401s, three discovery docs) against the real Node test app (fake Hydra, pinned externalUrl) — bit-identical through the Go normalization.

Next: Phase 2 — auth plane (Hydra providers + mediated DCR + discovery, bearer gate, WebAuthn ceremonies + Go fake authenticator, step-up, invite; gated on the discovery/DCR/WebAuthn goldens).

[rado0x54]

Phase 2 (auth plane) complete on go-backend (through 8a2ff3f), CI green. Delivered in four verified slices, each gated on the frozen goldens:

• Slice 1 — foundation: bearer gate (exempt lists, path→scope routing, /ws subprotocol token, {error,code}), Hydra introspection resolver (access-token-only, fail-closed, positive-only cache), discovery docs, ephemeral.Store[K,V] + injectable clock (the W4 fix — no module-level singletons), write-behind last-used flusher (W7 — auth hot path never writes). 7 stateless goldens.
• Slice 2 — WebAuthn ceremonies: go-webauthn-backed registration + assertion verification, byte-identical COSE→OpenSSH derivation, step-up store/gate, store.Credentials as sole owner of the table (W8) with the self-register account+credential atom in a real transaction (W9), and the Go fake authenticator (real P-256 crypto). self-register, in-account register, step-up-verify goldens.
• Slice 3 — Hydra providers + DCR: login provider options/verify → acceptLoginRequest, mediated DCR policy (redirect allowlist, scope subset, offline_access), ensureSpaClient, Go fake Hydra. Broke a package cycle by moving Principal/Resolver into auth. login-verify golden.
• Slice 4 — passkey-invite: injected invite store (supersede + atomic consume-if-token-matches), mint/redeem (pending_confirmation) + step-up-gated confirm, OpenSSH SHA256 fingerprint derivation. invite-mint, invite-redeem goldens.

13 goldens now pass on the Go server (7 stateless + 6 WebAuthn), all CI-enforced on every push.

Next: Phase 3 — terminal core (Manager/Session/byte-based OutputBuffer, SSH transport on the Phase-0-proven signers, REST sessions/endpoints/keys, WS hub), gated on the REST + WS goldens.