Repository navigation
Add optional Runtime history and Dashboard durable time ranges #38
Description
Activity
Phase 4 qualification update (PR #30, commit 8de7f12): public E2B Runtime evidence shows OTLP Collector fan-out, with ordinary operational metrics sent to Mimir and high-cardinality team_id + sandbox_id Runtime history sent to ClickHouse; authenticated reads enforce both identities, validate ranges, calculate query steps, and use bounded retention. Dify independently supports optional/no-op OTLP enablement but not the Runtime-history query boundary. Decision: keep a provider-neutral, bounded exporter handoff; use an optional operator Collector; keep a separate backend-neutral runtimehistory query adapter; treat ClickHouse as the first high-cardinality reference backend; do not use a shared Prometheus label filter as the tenant security boundary or let Web query telemetry backends directly. Implemented now: default-disabled sanitized async handoff, bounded queue/drop behavior, per-export timeout, panic isolation, deadline-aware shutdown, and tests proving provider receipts/raw provider text cannot reach export. Still intentionally unimplemented: concrete OTLP transport, ClickHouse/runtimehistory adapter, public history API, durable Web ranges, retention config, and exporter coverage telemetry. Verification: focused and race tests, go vet, complete Agents API + agents-client Go tests, and independent blind re-review all pass. Full make check remains locally gated by the required dedicated PostgreSQL URL; GitHub core-check is the full gate.
Phase 4A update on PR #30: commit fabcba4 adds the optional, default-disabled server-only OTLP/HTTP protobuf transport for validated Runtime observations. Configuration is strict and file-based; credentials/native provider identifiers/raw errors remain excluded. CPU/capacity/memory points now fail closed unless the sample carries the provider-qualified compute started_at incarnation fence; unfenced samples export coverage and read duration only. Focused tests, runtimeobs race tests, full Agents API + agents-client tests, go vet, server build, and independent blind re-review passed. Local make check still stops at the documented dedicated PostgreSQL URL prerequisite. Remaining Phase 4 work is unchanged: deployment-wide sampler, exporter coverage metrics, backend-neutral history query adapter, ClickHouse reference backend, tenant-scoped history API/client, and durable Dashboard ranges.
Phase 4 sampling update pushed to PR #30 in commit a1513ff. This adds optional execution-owner singleton background sampling (sample_interval_seconds 5..300), bounded deployment-wide managed-Session keyset scans, per-provider deadlines, lease-loss cancellation and pre-export ownership fencing, plus on_read/periodic OTLP classification. No Runtime lifecycle mutation or product database dependency is introduced. Verification: focused Runtime/store/server tests passed; runtimeobs race tests passed; full Agents API and agents-client Go tests passed; go vet, sqlc regeneration check, and server build passed; independent blind re-review reported no findings. Local make check still stops at the documented missing PARSAR_AGENTS_API_TEST_DATABASE_URL prerequisite; GitHub core-check remains authoritative.
Phase 4 query-boundary update pushed to PR #30 in commit 7169eab. Added backend-neutral internal/runtimehistory types/service plus a tenant-scoped Core store resolver. Reader calls now occur only after authenticated tenant/Session/Environment resolution; query ranges, server-selected resolution, total point budget, allocation+started_at incarnation fences, half-open bucket bounds, null/zero semantics, coverage, safe provider labels and numeric values are validated. OTLP exports now include Core resolved/observed nanosecond attributes so lower-precision generic metric tables can correlate one observation. No production Reader, public API, Web Durable range, product DB table or lifecycle behavior is enabled yet. Focused and full Agents API/client tests, vet and server build passed; independent blind re-review reported no findings. Local make check still stops at missing PARSAR_AGENTS_API_TEST_DATABASE_URL.
Runtime observability phase update (commit
2d48083, PR #30):- Added authenticated public capability discovery:
GET /v1/agents/runtime-history/capabilities. - Added bounded Session-scoped history query:
GET /v1/agents/sessions/{session_id}/runtime-historywith inclusivestart, exclusiveend, and boundedmax_points. - Tenant identity is derived only from authentication. The caller supplies only the Session ID; Session ownership is resolved before any Reader call.
- Added fail-closed validation for request bounds, retention, canonical identities, response scope/range echoes, half-open ordered buckets, collection and aggregate limits, safe JSON numbers, temporal consistency, and lossless allocation incarnation fences.
- Added strict
packages/agents-clientmethods and projection. Malformed, cross-scope, mixed-Environment, stale, oversized, duplicate, or temporally inconsistent responses are rejected as invalid upstream data. - Updated generated OpenAPI and contract/architecture documentation. OpenAPI generation now preserves response collection limits, metric enums, and the provider-type pattern.
- Token history is intentionally absent; no token series is synthesized from Runtime samples.
Verification:
- Agents API, Go client, and contract tests passed.
- Agents client: 332 tests passed.
- Web: 592 unit tests passed; production build and typecheck passed.
- Playwright acceptance: 75/75 passed.
go vet, SQLC consistency, and Agents API binary build passed.- Independent blind review completed with no remaining findings.
make checkstill stops only at the repository prerequisite requiringPARSAR_AGENTS_API_TEST_DATABASE_URLfor a dedicated test PostgreSQL database.
Explicit remaining boundary: no production Runtime history Reader is configured and the Dashboard Durable source UI is not enabled yet. Current Dashboard trends remain Live/browser-local. The next slice is a real durable backend/Reader plus acceptance proving background collection, persistence, retention, incarnation separation, and tenant isolation before enabling
Live / Durablein Web.- Added authenticated public capability discovery:
Sequencing update: Durable Runtime history remains tracked here, but persistence-layer implementation is intentionally deferred until the Live Dashboard effect is accepted. PR #30 now focuses on truthful browser-local time-series behavior and is deployed for review at commit
1771074. No ClickHouse dependency is enabled on the validation site; the temporary acceptance containers are stopped. We will resume this issue after the Live visual/data semantics are confirmed.ClickHouse-backed Runtime history is now implemented in PR #30 and deployed to the authorized validation host.
Validated end to end:
- periodic Runtime samples -> OTLP Collector -> ClickHouse -> Core history API -> Dashboard
- 30-second sampling, seven-day retention, and bounded 1h/6h/24h queries
- Session-scoped, tenant-authorized reads with Environment and Runtime-incarnation fences
- CPU, memory, and compute-uptime History survives a Core restart and browser reload
- Live and History remain explicit sources; token history remains Live-only
- public validation site serves the updated Web bundle and reports History available
The deployed acceptance records are explicitly marked validation points and do not trigger model calls. Full Web, client, Playwright, PostgreSQL, Go, build, and Linux Rust gates are recorded in the PR description. Independent re-review returned no findings after the CPU-capacity gap fix.
Goal
Add optional durable Runtime resource history for the Core Web Dashboard without changing current-snapshot execution ownership or making a telemetry backend mandatory for Sessions.
This follows the current-snapshot/API/Web work in #30 and Phase 4 of
contracts/agents-api/runtime-observability-design.md.Operator outcome
Proposed delivery slices
4A. Optional telemetry export
4B. History query adapter and public Core extension
runtimehistoryread interface; do not overloadruntimeobs.Serviceor current observation routes.unsupported,unavailable, andpartial/coverage metadata; never coerce missing samples to zero.packages/agents-client.4C. Web durable ranges
Liveas the existing browser-local bounded window and reset it on reload/navigation as today.Boundaries
Acceptance
Livebrowser memory from durable backend ranges.Open implementation decision
Qualify one operator backend and transport before implementation. Prefer OTLP-compatible export plus a tenant-authorized query adapter, but record the chosen backend's identity model, retention, downsampling, and query API before adding dependencies.