From 13b7ac61c707329f3109e7c1a801ca469d57b5a9 Mon Sep 17 00:00:00 2001 From: saski Date: Tue, 6 Oct 2026 17:51:05 +0200 Subject: [PATCH 1/2] docs: propose agent telemetry reliability learning lab --- .../agent-telemetry-reliability/design.md | 56 +++ .../agent-telemetry-reliability/proposal.md | 85 ++++ .../specs/agent-telemetry-reliability/spec.md | 136 +++++ .../agent-telemetry-reliability/tasks.md | 75 +++ ...6-10-06-agent-telemetry-reliability-lab.md | 467 ++++++++++++++++++ 5 files changed, 819 insertions(+) create mode 100644 docs/openspec/changes/agent-telemetry-reliability/design.md create mode 100644 docs/openspec/changes/agent-telemetry-reliability/proposal.md create mode 100644 docs/openspec/changes/agent-telemetry-reliability/specs/agent-telemetry-reliability/spec.md create mode 100644 docs/openspec/changes/agent-telemetry-reliability/tasks.md create mode 100644 docs/plans/2026-10-06-agent-telemetry-reliability-lab.md diff --git a/docs/openspec/changes/agent-telemetry-reliability/design.md b/docs/openspec/changes/agent-telemetry-reliability/design.md new file mode 100644 index 0000000..d6c71a3 --- /dev/null +++ b/docs/openspec/changes/agent-telemetry-reliability/design.md @@ -0,0 +1,56 @@ +# Design + +The detailed design, verified source references, alternatives, metric contracts, +failure matrix, learning slices and estimated effort are in the +[learning plan](../../../plans/2026-10-06-agent-telemetry-reliability-lab.md). +It is the design reference for this change; all runtime changes are proposed. + +## Decisions + +1. Use the existing fixture workflow and persistent gateway, independent of the + dirty Review Capacity implementation. Preserve existing source changes. +2. Retain the lab Collector as the stopped component in the outage exercise. + Forward to the receiver bundled with LGTM. Grafana and Prometheus must remain + alive when that ingress Collector stops. +3. Use OTel metrics with a direct Prometheus scrape from an authenticated gateway + endpoint. Neither application freshness nor Collector reachability depends + exclusively on the stopped Collector's OTLP pipeline. +4. Derive current task counts from canonical operational state, never received + spans or delivery attempts. Read-only aggregates have bounded labels and + distinguish legitimate zeros, disabled export, failed reads and stale data. +5. Compare expected journal identities against complete Tempo trace results for + one small run. At-least-once delivery, downstream gaps and query incompleteness + stay visible; no production loss detection or deduplication guarantee is made. +6. Keep host control, gateway permissions, worker isolation and human approval + unchanged. Only a scoped fixture demo path may inject bounded delay/failure. + +## Risks and mitigations + +- **Bundled backend failure:** the observer can fail with the backend. Retain the + original gateway dashboard and a host-side diagnostic; do not claim HA. +- **Post-acknowledgement loss:** the durable gateway outbox covers its own + acknowledgement boundary. Finite reconciliation can reveal missing downstream + spans; persisted Collector recovery remains deferred. +- **False interpretation:** event spans are not complete worker instrumentation; + policy denial and human rejection are not inherently technical crashes; + logical ticks are not seconds. Teach and test these distinctions. +- **Version drift:** pin tested image/SDK versions during implementation and + verify provisioning paths, metric names, TraceQL and alert semantics. +- **Data leakage or authority expansion:** preserve explicit resources and + attribute allowlists, viewer-only metrics, private networks and no operator + credentials in observation services. +- **Existing work or runtime collision:** review both checkouts again before + implementation; use additive files, explicit profile/ports and fresh evidence; + do not reset histories, acknowledgement flags or other running services. + +## Validation and rollback + +Use failing behavior tests, then relevant unit tests, canonical repository checks +and an explicitly started opt-in integration suite. Inspect the actual UI and +document observed timing and missing checks. Existing specifications are not +synced or archived until the implementation meets the acceptance contract. + +The optional profile can be stopped without removing data. Restore the prior +explicit exporter destination when leaving the exercise; do not replay previously +acknowledged history automatically. Preserve all run evidence and configuration +needed to reproduce the observation. No destructive cleanup is part of the demo. diff --git a/docs/openspec/changes/agent-telemetry-reliability/proposal.md b/docs/openspec/changes/agent-telemetry-reliability/proposal.md new file mode 100644 index 0000000..da9ca42 --- /dev/null +++ b/docs/openspec/changes/agent-telemetry-reliability/proposal.md @@ -0,0 +1,85 @@ +# Agent Telemetry Reliability Lab + +## Why + +A quiet activity screen can mean slow or failed execution, a legitimate human +review checkpoint, or interrupted telemetry. The lab already has canonical +history and recoverable OTLP trace delivery, but lacks a queryable trace backend +and independent metrics for learning how to distinguish those conditions. + +The primary outcome is learner understanding through a real fixture workflow, +queries and controlled failures. The [learning plan](../../../plans/2026-10-06-agent-telemetry-reliability-lab.md) +explains the baseline, options, exercises, sources, estimates and demonstration. + +## What Changes + +- Add an opt-in local LGTM backend, using the existing lab Collector as an + independently stoppable ingestion hop, with traces stored in Tempo. +- Add five read-only OTel metric families, scraped directly by Prometheus, plus + direct Collector self-metric collection outside its own delivery pipeline. +- Provision a compact Grafana dashboard and one local alert rule with explicit + missing/stale/error behavior. +- Add bounded fixture delay/failure exercises and a read-only comparison of a + finite run's expected journal spans against queryable Tempo evidence. +- Supply a startup guide, learner exercises, three-minute demo and limitations. + +## Acceptance Scope + +Fresh fixture runs, one gateway/exporter per database, normal execution, +human-review pending, short worker delay/failure, a stopped ingress Collector, +recovery and retransmission, and explicit missing-data evidence. Recovery must +preserve canonical history and event identities. Accepted-but-missing downstream +evidence must be detectable by finite comparison, without claiming permanent +retention or automatic replay after acknowledgement. + +Metrics and Grafana remain observers. They gain no task-execution authority. +No Review Capacity logical ticks are exported as live workflow time. + +## Non-Goals + +Implementation in this planning task; production monitoring; Kubernetes; fleet +management; a new control plane; Grafana Cloud; public deployment; external +notifications; live/paid providers; worker credential/network expansion; automatic +approval; correlated application logs; full worker instrumentation; persistent +downstream recovery or general exactly-once delivery. + +## Capabilities + +### New Capabilities + +- `agent-telemetry-reliability`: a local, reproducible observation and diagnosis + experiment with independent delivery evidence and explicit learning checks. + +### Modified Capabilities + +None. Existing `observable-activity`, `governed-execution` and +`systems-experiments` requirements remain the baseline and must keep passing. + +## Impact + +Expected changes after approval: optional Compose/configuration/provisioning +files, narrow gateway metric reads, fixture-only demo controls, integration +tests, Make targets, locked exporter dependency and user/development guides. +No new source or runtime configuration is implemented by this proposal. + +Status: proposed for user review on 6 October 2026. Implementation is pending. + +## Planning Validation + +On 6 October 2026, `make spec-check` passed all five current OpenSpec items, +including this proposal. Local Markdown links, balanced code fences and +whitespace in the five new documents were checked. No existing tracked files or +the secondary checkout were edited. No service was started and no dependency +was installed. `make check` and runtime/integration tests were not run for this +documentation-only task; none of the proposed runtime outcomes is verified. + +## Publication Validation + +Later on 6 October 2026, local `main` was synchronized with remote `main` at +`8bf1669a68a9072326d35abfb18ee979a3766531`, preserving the new proposal and backing +up the older local Review Capacity drafts. Before publication, `make check` +passed: lint, formatting, 83 tests and Compose configuration validation; five +opt-in container tests were skipped. `make spec-check` passed all five items. +Local links, code fences and whitespace were checked. The unrelated secondary +checkout remains untouched. No experiment service was started and no proposed +runtime outcome is verified by these checks. Implementation remains pending. diff --git a/docs/openspec/changes/agent-telemetry-reliability/specs/agent-telemetry-reliability/spec.md b/docs/openspec/changes/agent-telemetry-reliability/specs/agent-telemetry-reliability/spec.md new file mode 100644 index 0000000..1cc0958 --- /dev/null +++ b/docs/openspec/changes/agent-telemetry-reliability/specs/agent-telemetry-reliability/spec.md @@ -0,0 +1,136 @@ +## ADDED Requirements + +### Requirement: local reproducible trace exploration +The experiment SHALL provide an opt-in, version-pinned local Tempo/Grafana +configuration for current gateway-observed OTLP spans, preserving the default +file-export path and gateway permission boundaries. + +#### Scenario: follow a fresh fixture run +- GIVEN a newly created fixture run and the optional observation profile +- WHEN its pending telemetry is delivered +- THEN its journal trace and span identities are queryable in Tempo through Grafana +- AND original event times and fixture provenance are preserved +- AND the guide distinguishes event/completion spans from full worker instrumentation + +#### Scenario: preserve existing evidence and isolation +- GIVEN existing local histories and the normal worker restrictions +- WHEN the optional profile is started and later stopped +- THEN no database, journal or telemetry data is deleted or silently replayed +- AND no worker receives new credentials, host mounts, Docker socket or network access +- AND only intended loopback ports are published +- AND resolved Compose configuration excludes observation services from database and worker networks + +### Requirement: independent read-only metrics +The gateway SHALL expose authenticated, bounded-label OTel metrics for run state, +pending delivery count and age, exporter enablement and successful snapshot time. +Prometheus SHALL collect them without traversing the Collector delivery pipeline. + +#### Scenario: count tasks independently of retransmission +- GIVEN distinct canonical runs, including more than the activity overview limit +- WHEN an observation is delivered repeatedly or the exporter restarts +- THEN metric run counts still equal current distinct run rows in that database +- AND IDs, hashes, actor instances and arbitrary text do not become metric labels + +#### Scenario: preserve viewer authority +- GIVEN the read-only viewer credential used for metric collection +- WHEN it requests metrics or attempts to grant, decide or revoke a run +- THEN metrics can be read and operational writes remain unauthorized +- AND collection does not mutate canonical history +- AND no second unauthenticated metrics listener bypasses gateway viewer checks + +### Requirement: explicit zero and missing-data semantics +Panels and alert evaluation SHALL distinguish successful zero values, disabled +export, missing series, stale snapshots and failed data-source queries. + +#### Scenario: empty healthy lab +- GIVEN an enabled exporter, successful fresh database read and empty outbox +- WHEN the metrics are scraped +- THEN pending count and pending age are explicitly zero +- AND freshness evidence establishes why those zeros are meaningful + +#### Scenario: failed or partial observation +- GIVEN a failed scrape, missing expected series or stale snapshot +- WHEN Grafana displays or evaluates the data +- THEN it indicates missing, stale or attention state as appropriate +- AND it does not substitute a healthy zero or silently retain a green state +- AND a surviving series cannot conceal a missing required series + +### Requirement: meaningful local delivery alert +The experiment SHALL provision one local Grafana-managed alert rule for delivery +backlog, failed required scrape targets and stale snapshots, with explicit +No Data and Error behavior and documented evaluation timing. + +#### Scenario: ingest Collector is stopped +- GIVEN the gateway, LGTM backend and direct scrape paths remain available +- WHEN only the lab Collector is stopped and a fixture run produces events +- THEN the task and journal continue independently +- AND pending delivery and failed Collector reachability are visible +- AND the alert enters attention within the documented test budget +- AND no external message or automatic operational action is sent + +### Requirement: controlled fixture diagnosis +The experiment SHALL provide opt-in, bounded fault exercises that use the normal +gateway and worker lifecycle, without arbitrary commands or expanded authority. + +#### Scenario: distinguish short delay from failure +- GIVEN a new fixture run with one selected Builder intervention +- WHEN a fixed delay or controlled nonzero worker exit occurs +- THEN measured worker completion duration or worker failure evidence identifies it +- AND normal failure handling retains run revocation and permission enforcement +- AND unknown fault modes and non-fixture use are rejected + +#### Scenario: distinguish pending review from an incident +- GIVEN completed specialists and a run awaiting a digest-bound human decision +- WHEN observation paths are healthy +- THEN the UI identifies human review as the next step +- AND it does not classify waiting, human rejection or every policy denial as a technical failure +- AND Grafana cannot approve or resume the run + +### Requirement: identity-preserving recovery and finite reconciliation +Recovery SHALL retain original event identities and timestamps, leave canonical +history unchanged by retries, and compare a bounded expected span set with +complete backend query results without assuming exactly-once receipt. + +#### Scenario: recover a pending batch across exporter restart +- GIVEN an unavailable ingress Collector and durable pending journal events +- WHEN the exporter restarts with the same database and the Collector recovers +- THEN pending events become queryable with their original trace and span IDs +- AND the stored journal prefix and checkpoint still verify +- AND retransmission does not become a new task or canonical event + +#### Scenario: receiver accepts before acknowledgement fails +- GIVEN a receiver accepted a batch but its local acknowledgement was not saved +- WHEN the gateway retries that pending batch +- THEN the repeated delivery retains the original identities +- AND the evidence comparison counts unique identities and reports receipt behavior honestly + +#### Scenario: accepted evidence is missing downstream +- GIVEN a controlled receiver or filter omits one known fixture span after acceptance +- WHEN a bounded complete Tempo query is compared with expected journal identities +- THEN the missing identity is reported even if the gateway outbox is empty +- AND time range, query completeness and observation deadline are reported +- AND no claim of irreversible loss or automatic replay follows from that gap alone + +#### Scenario: entire observation backend is unavailable +- GIVEN Grafana and Prometheus stop with the LGTM backend +- WHEN the operator diagnoses the missing view +- THEN the guide directs them to the independent gateway dashboard and host check +- AND it does not claim that the unavailable Grafana alert can detect its own outage + +### Requirement: safe and honest learning evidence +The experiment SHALL preserve the telemetry allowlist and provide a hypothesis, +boundary, variables, observations, limits, learner exercises and reproducible +configuration, guide and demonstration. + +#### Scenario: sensitive sentinels stay local +- GIVEN credentials, prompts, tool arguments, content and raw exception text in test inputs +- WHEN traces, metric samples and demo evidence are generated +- THEN none of those values appears in exported payloads, labels, panels or saved shared evidence +- AND an authentication credential appears only where required for transport access + +#### Scenario: explain the outcome independently +- GIVEN completed learning slices +- WHEN the learner performs the prepared demonstration +- THEN they can query a run, predict and diagnose a fault, show recovery and explain a possible resend +- AND they distinguish observed wall time from Review Capacity logical ticks +- AND fixture evidence is not described as production performance or real-model quality diff --git a/docs/openspec/changes/agent-telemetry-reliability/tasks.md b/docs/openspec/changes/agent-telemetry-reliability/tasks.md new file mode 100644 index 0000000..d69a210 --- /dev/null +++ b/docs/openspec/changes/agent-telemetry-reliability/tasks.md @@ -0,0 +1,75 @@ +# Implementation tasks + +All tasks below are pending. User approval of implementation is required by the +task's explicit scope; this planning change does not provide it. + +## Slice 0: understand the existing run + +- [ ] Recheck both checkouts and current runtime/ports without replacing existing + work; select the implementation baseline and reload applicable rules. +- [ ] With service execution authorized, record a fresh fixture run, trace ID, + model mode, executor, artifact digest and human-review checkpoint. +- [ ] Have the learner predict and explain the checkpoint using existing evidence. + +## Slice 1: query existing traces + +- [ ] Select and record tested LGTM tag/digest and component versions; verify + architecture support, local resource use, configuration paths and port availability. +- [ ] Add the optional Compose override, narrow telemetry network and lab-Collector + forwarding configuration; preserve the existing file exporter/profile. +- [ ] Provision Tempo/Prometheus data sources and retain new runtime data under + ignored `.lab/observability/`; add canonical Make start/stop/config-check targets. +- [ ] Verify effective Compose boundaries and a fresh run's IDs/timestamps in Tempo; + have the learner inspect a worker completion and explain event-span limits. +- [ ] Verify override support and assert the resolved Collector has only the + telemetry network and no host ports; neither observation service may retain + database or worker network attachments through Compose merging. + +## Slice 2: metrics with honest meanings + +- [ ] Start with failing behavior tests for distinct run counts beyond 100 runs, + resend/restart without count inflation, empty data versus failed reads, + consistent snapshots, viewer authority and sensitive sentinel exclusion. +- [ ] Add the five aggregate OTel metrics and authenticated endpoint, using the + smallest locked exporter dependency; no trace-derived task counts. +- [ ] Serve metric exposition through the viewer-protected FastAPI route, without + a second unauthenticated listener; return a failed scrape on snapshot error + and validate exporter registration/cleanup across app lifecycle. +- [ ] Configure direct Prometheus scraping with viewer credentials from a file + and explicit Collector self-metric exposure only on the private network. +- [ ] Provision the compact dashboard with current states, backlog/age, enablement, + freshness, target reachability and a trace lookup; preserve null/stale states. +- [ ] Have the learner write review-state, backlog and absence queries and explain + why run IDs are excluded from metric labels. + +## Slice 3: introduce and diagnose bounded failures + +- [ ] Add failing behavior tests for fixture-only scoped delay/failure and reject + unknown modes; exercise real lifecycle recording and permission enforcement. +- [ ] Add the bounded demo fault path without arbitrary execution or new permissions. +- [ ] Provision the single alert with its tested truth table, missing-series checks, + No Data/Error handling and no external contact point. +- [ ] Test the alert for normal, review-pending, empty, disabled, delayed delivery, + failed targets, partial missing series, whole-query absence and query error. +- [ ] Stop only the lab Collector, run a fixture, observe independent evidence and + recover without clearing pending state; repeat with exporter restart. +- [ ] Add finite read-only span comparison with complete-query checks and a frozen + journal boundary; fail on missing/partial evidence, tolerate duplicate IDs. +- [ ] Test acceptance followed by failed local acknowledgement and a controlled + downstream omitted span even when the outbox is empty. +- [ ] Have the learner predict each intervention and explain two supporting facts + and one limitation; document the full-LGTM failure boundary. + +## Slice 4: demonstrate and hand over + +- [ ] Add experiment/quickstart/demo guides with inputs, hypothesis, provenance, + actual observations, timings and limits; label all unavailable checks. +- [ ] Reproduce dashboard and alert from files; inspect actual Grafana UI and + original activity dashboard without exposing credentials or sensitive content. +- [ ] Add an opt-in integration Make target, then run it and the applicable existing + container isolation tests with explicit service-start scope. +- [ ] Run `make check`, `make spec-check` and `git diff --check`; record results. +- [ ] Have the learner deliver the rehearsed three-minute demonstration and explain + missing data versus zero, duplicate deliveries and a post-acceptance gap. +- [ ] Update user-facing links only for implemented functionality. Review completed + evidence before syncing/archiving this change; no automatic publication. diff --git a/docs/plans/2026-10-06-agent-telemetry-reliability-lab.md b/docs/plans/2026-10-06-agent-telemetry-reliability-lab.md new file mode 100644 index 0000000..aa7da96 --- /dev/null +++ b/docs/plans/2026-10-06-agent-telemetry-reliability-lab.md @@ -0,0 +1,467 @@ +# Agent Telemetry Reliability Lab: learning plan + +Status: **proposal for review; implementation has not started**. +Prepared on 6 October 2026. Repository observations below come from source and +configuration inspection, not a live run. Technical artifacts are in English; +the accompanying explanation to the learner is in Spanish. + +## Start with one apparently stuck task + +You are running the lab and see no new activity. Before restarting anything, +you want to know whether the Builder is still working, a worker failed, a human +decision is pending, or information stopped reaching the screen. + +Imagine a fixture run reaches its human review checkpoint just as the Collector +stops. Grafana shows an incomplete trace. The existing activity dashboard still +shows `awaiting_review`, and the gateway has events waiting for delivery. There +are two simultaneous facts: the task needs a person, and telemetry delivery +needs repair. Restarting the task would not explain either fact. + +The first user is you, the lab operator and learner. Success means being able +to explain this situation with evidence, then reproduce the explanation without +depending on an AI-generated dashboard. A later engineering-management interview +may benefit from this example, but learning and a useful lab diagnostic come first. + +## Vocabulary for this experiment + +| Term | Everyday meaning here | +| --- | --- | +| Run | One execution of the three-specialist workflow, identified by `run_id`. | +| Gateway | The lab's doorkeeper: it checks permissions, executes registered operations and records what happened. | +| Journal | The database's ordered historical record. It is the reference for events, not Grafana. | +| Outbox | A persistent list of journal events still waiting for telemetry acknowledgement. It contains observations, not tasks waiting to execute. | +| OpenTelemetry / OTLP | Tools and conventions for describing observations; OTLP is the protocol used to send them. | +| Span / trace | A span describes one observed event or operation. A trace groups related spans under one ID; here one run has one trace ID. | +| Metric | A number sampled over time, such as how many events are waiting for delivery. | +| Collector | A receiver and forwarder for observations. It does not run or authorize agent tasks. | +| Tempo / Prometheus | The proposed stores for traces and numerical time series, respectively. | +| Grafana | The interface for querying those stores, displaying panels and evaluating an alert. | +| Scrape | Prometheus periodically asks an endpoint for its current metrics. | + +## Verified starting point: planning snapshot + +The following observations describe the checkout before synchronization. They +are retained as provenance, rather than a description of the current Git state. + +Primary checkout: `docs/dashboard-guide`, HEAD +`948564606fe7dd70aced53d1389793c35bd77362`. Its pre-existing untracked +`docs/openspec/changes/review-capacity-experiment/` remains untouched. The local +tracking ref reports the branch behind by one commit; no remote refresh was made. + +Related checkout: `/Users/saski/Code/agent-systems-lab_worktrees/review-capacity-v02`, +branch `feat/review-capacity-v02`, HEAD +`61a5c31c6babf882d28f4a1fddd5933f71c8b470`. It contains existing modified and +untracked Review Capacity implementation, tests and documentation. The telemetry, +journal, activity, Compose and observability files inspected are identical in +both checkouts. No branch was switched and no work was copied between them. + +Line references below refer to the primary checkout at the stated HEAD. + +| Existing evidence | What it establishes | Missing piece proposed here | +| --- | --- | --- | +| [Gateway](../../src/systems_lab/gateway.py), lines 115–197 | Delivery lifecycle, viewer-authenticated activity reads and a basic health endpoint | An authenticated metrics read, with real data freshness rather than just process health | +| [Journal](../../src/systems_lab/journal.py), lines 22–27, 153–242, 302–338 | Stable IDs; event and outbox committed together; pending selection and acknowledgements | Read-only aggregate metrics and a bounded comparison with Tempo | +| [Telemetry](../../src/systems_lab/telemetry.py), lines 85–124, 158–236 | Explicit OTLP/HTTP destination, journal-event spans, allowlisted attributes and measured completion durations | A searchable trace backend and learning queries | +| [Activity projection](../../src/systems_lab/activity.py), lines 11–51, 56–89 | Human-review stage; unknown worker after 120 seconds; delivery status; latest 100 runs | Grafana overview without using that 100-run window as a global count | +| [Collector configuration](../../observability/collector.yaml) and [Compose](../../compose.yaml), lines 46–74 | Optional `observability` service; trace input and file output only; loopback port 4318 | Forwarding to a local backend, direct metric scraping and reproducible Grafana resources | +| [Store](../../src/systems_lab/store.py), lines 318–424; [executor](../../src/systems_lab/orchestrator.py), lines 109–138 | Explicit review state, digest-bound decisions, measured operations, worker failure/timeout and revocation | Small, deliberate fixture fault exercises | +| [Delivery guide](../observability.md), lines 85–107, 126–145; [tests](../../tests/test_telemetry.py) | Documented at-least-once boundary and test cases for retry/restart, redaction and integrity failure | Actual Tempo/Grafana integration evidence; these tests were read, not rerun for this plan | + +Important interpretation limits: + +- `run.create` and `worker.started` are event spans, often with zero duration. + Worker completion spans derive elapsed time from lifecycle timestamps; tool + and model completion durations use the gateway's operation timer. This is not + complete instrumentation inside workers, nor a conventional enclosing root + span measuring all work. A trace's overall time extent may include human wait. +- A `run.decision` span does not export the private `detail` field containing + accepted/rejected status. Read that outcome from gateway state; do not infer it + from the trace's `allow` decision. A policy denial is not necessarily a crash. +- Review Capacity's implementation exists in the secondary working tree. Its + experiment guide and `review_capacity.py` use logical ticks and a synthetic + reviewer. Those results must not become wall-clock latency or live worker + telemetry. This proposal uses the existing real fixture workflow instead. + +## Proposed local design + +```mermaid +flowchart TB + subgraph task[Task path - existing] + U[Human operator] --> W[CLI and LangGraph
Researcher - Builder - Reviewer] + W --> H[Human review checkpoint] + W -->|authorized calls and lifecycle| G[Gateway] + end + subgraph evidence[Observation path] + G --> J[(Journal and run state)] + J --> Q[Q1 - durable telemetry outbox] + Q -->|existing exporter| C[Lab Collector
controlled outage target] + C --> L[LGTM receiver then Tempo
trace storage] + J -.->|gateway metrics endpoint| P[(Prometheus in LGTM)] + C -.->|direct self-metric scrape| P + L --> F[Grafana
queries and alert] + P --> F + J --> D[Existing activity dashboard] + end + classDef actor fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef queue fill:#ffedd5,stroke:#ea580c,color:#111827 + classDef store fill:#dcfce7,stroke:#16a34a,color:#111827 + class U,W,H,G,C,F,D actor + class Q queue + class J,P,L store +``` + +Solid arrows carry task actions or recorded events; dotted arrows are metric +reads. Q1 is the one application delivery queue: the journal produces entries +and the exporter consumes them. Journal and outbox are tables in the same +database. Collector/backend transport buffers are separate implementation +details, not additional task queues or durable guarantees in this first lesson. + +**Recommendation:** reuse the lab Collector as an independently stoppable +service and add `grafana/otel-lgtm` as the local backend. The bundle includes its +own Collector receiver; two receivers therefore exist intentionally. Stopping +the lab Collector must leave Grafana and Prometheus running. Sending directly +to the bundle would remove one hop, but stopping its whole container would +also remove the observer needed for the principal exercise. Separate Tempo, +Prometheus and Grafana containers remain a possible later decomposition. + +Grafana documents this bundle for development, testing and demos. That matches +this experiment. Its configurable backends and file-mounted dashboards support +the proposed local setup. Choose and record a tested image tag and digest during +implementation; do not commit a floating `latest` reference. No particular image +version, memory requirement or startup time has been verified on this machine. +See [LGTM documentation](https://grafana.com/docs/opentelemetry/docker-lgtm/) and +the [official repository](https://github.com/grafana/docker-otel-lgtm). + +The upstream [Collector configuration](https://github.com/grafana/docker-otel-lgtm/blob/main/docker/otelcol-config.yaml) +routes traces to Tempo and collects its own metrics through itself. That +self-monitoring route alone cannot cover a stopped Collector. The upstream +[Prometheus configuration](https://github.com/grafana/docker-otel-lgtm/blob/main/docker/prometheus.yaml) +also does not already define the lab's proposed direct scrape targets. + +### Boundaries to preserve + +- Add a separate opt-in Compose override, `observability/grafana/compose.yaml`. + Keep the existing file-export profile usable. Use the existing gateway and + its Postgres history for the taught container path; do not also launch the + SQLite dashboard on port 8765. One exporter process per database remains the + supported topology. New runs provide evidence at the new destination; already + acknowledged history is not automatically replayed. +- Record the existing run counts and pending backlog before the first exercise. + An enabled destination may receive previously pending events. Let that backlog + settle, or report it explicitly; do not clear it to obtain a clean chart. + Compare new-run deltas with the recorded baseline, not an assumed empty lab. +- Preserve `http://otel-collector:4318/v1/traces` as the gateway destination in + this profile. The Collector forwards to the LGTM receiver using internal + service DNS. Do not publish a second host port 4318. Publish Grafana only on + loopback, normally 3000; preflight occupied ports instead of stopping services. +- Attach observation services to a dedicated internal telemetry network, not + the workers or database network. LGTM may also use the operator-facing bridge + for its loopback UI. The gateway joins telemetry while retaining its required + existing networks. In the optional override, fully replace `otel-collector` + networks with `!override [telemetry]` and its host ports with `!override []`: + this lesson uses container-to-container ingestion, and an ordinary merge could + retain its old database-network attachment. Verify the resolved config lists + no `control` or `workers` network on either observation service. Preserve the + fixed worker network name. Add no worker egress, host credentials, Docker + socket, host PID namespace, privileged mode or automatic host instrumentation. + [Docker documents `!override` for Compose 2.24.4 and later](https://docs.docker.com/reference/compose-file/merge/#replace-value). + The installed Compose CLI reports 5.2.0; daemon and runtime behavior remain + unverified. Require the supported version for this optional profile only. +- Store new local LGTM runtime data under ignored `.lab/observability/`, with + validated ownership. Retain existing database and telemetry volumes. Stop + services without deleting data. No provider keys, external OTLP forwarding or + backend usage-reporting integration is part of the local profile. +- Reuse the viewer permission for `/metrics`; mount only the viewer token file + read-only into Prometheus, never the operator token or the entire `.lab/`. + Authentication headers are transport configuration, never telemetry labels, + dashboard fields or saved evidence. [Prometheus supports a credentials file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_config). +- Keep the export allowlist. Do not add automatic HTTP/body instrumentation or + arbitrary resource attributes. Prompts, credentials, tool arguments, content, + raw exception messages and user-supplied text remain outside telemetry. + +## Five application metrics, one alert + +Use OpenTelemetry metrics with its Prometheus exporter. Prometheus scrapes the +gateway directly, so these readings do not traverse either Collector. This +teaches both OTLP trace delivery and metric collection without adding another +push pipeline. [OpenTelemetry documents this exporter and collection model](https://opentelemetry.io/docs/languages/python/exporters/#prometheus). + +Proposed Prometheus-facing names below are contracts to verify against the +selected SDK/exporter versions. None exists yet. + +| Metric | Type and meaning | Allowed variable labels | +| --- | --- | --- | +| `lab_runs` | Gauge: distinct runs currently in each state, from run rows, across the chosen database | `state`: running, awaiting_review, accepted, rejected, revoked | +| `lab_telemetry_pending_events` | Gauge: undelivered outbox rows; not tasks | None | +| `lab_telemetry_oldest_pending_age_seconds` | Gauge: current time minus the oldest pending event timestamp; zero only when the queue was successfully read and is empty | None | +| `lab_telemetry_export_enabled` | Gauge: exporter enabled, 1 or 0; disabled is not healthy delivery | None | +| `lab_metrics_snapshot_timestamp_seconds` | Gauge: time of a successful database snapshot for this scrape | None | + +The dashboard must prominently label the context as fixture-backed, +gateway-observed activity with wall-clock seconds. A `running` run row alone is +not proof that its worker is healthy; use lifecycle evidence and freshness. + +Read aggregates from a consistent database snapshot, independently of the +dashboard's 100-run limit. Explicitly emit zero-valued state buckets when a +successful read finds no runs in them. A database read failure must fail the +scrape, not return zero or refresh a cached snapshot's timestamp. Preserve +read-only behavior and avoid a new writer or persisted metric ledger. + +Serve the SDK-backed exposition through the gateway's FastAPI `GET /metrics` +route with `Depends(viewer)`, not the exporter's standalone HTTP-server example. +Read the snapshot before rendering; metric callbacks must observe that same +successful snapshot. If it cannot be read, return HTTP 503 and no synthetic +measurements. Test unauthenticated access, snapshot failure and repeated app +lifecycle; do not open an unauthenticated second metrics listener. Limit resource +attributes and published metric families deliberately when wiring the exporter. + +Also show Prometheus's built-in `up` for fixed jobs `lab-gateway` and +`lab-collector`. Configure the latter's internal metrics endpoint on the private +network, scraped directly, not through its OTLP pipeline. This is endpoint +reachability, not proof of downstream acceptance or retention. Current Collector +docs use `service.telemetry.metrics.readers`; the older `address` setting is +ignored in recent versions. Verify actual names for the pinned image before +using diagnostic queue/error metrics. [Collector internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/) + +Keep only the bounded state label plus fixed deployment labels such as job and +service. `run_id`, actor instance IDs, event IDs, trace/span IDs and hashes stay +in traces or local comparison evidence. Every different label combination +creates another stored series: putting each task ID in a metric would make +storage and queries grow with every task. [Prometheus naming guidance](https://prometheus.io/docs/practices/naming/) + +No run counts or duration histograms will be generated from received spans in +v1. A retransmitted span is another delivery of an observation, not another +task. State gauges also avoid a new durable counter projection. Operation +duration is inspected in traces; review waiting is a separate state. + +The proposed Grafana-managed rule, `LabTelemetryNeedsAttention`, has this truth +table, to be tested before its query is finalized: + +- Attention when enabled delivery has pending events older than 10 seconds, + either required scrape target reports `up=0`, or its successful snapshot is + over 15 seconds old. +- Detect missing expected targets/series explicitly; a surviving series must + not hide another missing series. Whole-query No Data or Error remains visible + as an attention state, never Normal or a retained green value. +- Fresh successful reads, an enabled exporter, both targets up and an empty + outbox are normal even if every run is awaiting review. Disabled export gets + its own visible label and is not described as successful delivery. +- Start with 5-second scrapes and evaluations, a 10-second pending period, and + a 60-second observation budget for the exercise. These are teaching settings, + not a production service-level commitment. Record actual detection time. + +Use one provisioned rule and its health/No Data/Error states. Show it in the +local UI; no email, Slack, webhook or external notification destination. +[Grafana's documented No Data and Error behavior](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/) +must be checked for the bundled version. Panels must leave gaps and state +`No data`/`Stale`; never use blanket null-to-zero or `or vector(0)` substitutions. + +## Experiments and diagnoses + +Hypothesis: independent state and delivery observations let a learner distinguish +execution trouble, a human checkpoint and a telemetry interruption. After a +bounded receiver outage, pending observations become queryable with their original +identities while canonical task history remains unchanged by delivery retries. + +Boundary: one local operator, one gateway/database, fixture workers, the lab +Collector and the local LGTM bundle. Controlled variables: same fixture, versions, +permissions, machine, scrape intervals and small run count. Change one factor at +a time: worker delay, worker failure or Collector availability. Use wall-clock +seconds for actual execution and delivery; no Review Capacity ticks enter this +experiment. Record run/trace IDs, UTC observation timestamps, configuration +revision, fault onset/recovery, executor, model mode and unique evidence counts. + +| Case | Deliberate intervention | Evidence and interpretation | +| --- | --- | --- | +| Normal | Run one existing fixture workflow | Find its trace and successful worker completion; gateway ends at `awaiting_review`; pending delivery drains. This is completion of automated work, not human acceptance. | +| Human review | Leave that decision pending | `run.ready`, state `awaiting_review`, healthy collection and fresh data. Quiet activity is expected. A human rejection is also a business outcome, not automatically a technical incident. | +| Slow worker | Opt in to a fixed 5-second delay for Builder, after its recorded start, on one new fixture run | While active, the gateway shows running; after completion the worker span includes the delay. Gateway operation spans need not show that delay. No synthetic span timestamps or new permissions. | +| Failed worker | Opt in to a controlled nonzero Builder exit on one new fixture run | `worker.failed` and run revocation from the existing executor path. A red policy-denial span alone would not establish this diagnosis. | +| Collector unavailable | Stop only the lab Collector, then create one fixture run | Gateway and its original dashboard remain available; direct gateway metrics stay fresh, Collector `up` falls, pending count/age rise, and the alert requires attention. | +| Recovery | Restart that Collector without clearing state | Pending events drain; the same trace becomes more complete at original event times. Compare identities, not received-span counts. Alert returns to Normal after fresh successful readings. | +| Missing data | Make only the metrics read unavailable in an isolated test; also test a missing target/series | Visible missing/stale/error evidence, not a healthy zero. An empty but healthy lab instead supplies real zeros. | + +Future fault hooks belong to a bounded fixture-only demo path in the trusted +CLI/worker launcher: fixed cases, one selected role/run, maximum delay 5 seconds, +off by default. They must exercise the real lifecycle and permission path, reject +unknown/live modes, and never accept arbitrary commands. The host operator owns +Collector stop/start; no worker or Grafana action obtains that ability. A worker +timeout beyond the existing 90 seconds is optional after the short failure lesson. + +### Detecting observation loss honestly + +Use three different questions: + +1. **Is delivery interrupted before acknowledgement?** Inspect the direct scrape + and gateway outbox. These survive a stopped lab Collector. No new events means + no pending backlog may form; `up` still tests reachability. +2. **Did all expected observations become queryable?** For a small frozen run, + record a journal sequence boundary and expected `(trace_id, span_id)` set. + Compare it with a complete trace lookup from Tempo, deduplicating IDs. Include + all pages/limits on the journal side and reject partial/pruned/truncated trace + responses. Match timestamps/actions as well as identities. Query the original + event interval, not only the recovery time. [Tempo trace lookup API](https://grafana.com/docs/tempo/latest/api_docs/) +3. **Is the observation system itself unavailable?** If the whole LGTM container + is down, its Grafana alert cannot work. Use the existing gateway dashboard and + a bounded host-side reachability check. This is a shared failure domain, not + highly available monitoring. + +An acknowledgement proves acceptance by the next receiver, not durable retention +in Tempo. Collector memory queues, crashes, filtering, exhausted retries and disk +failure can produce a downstream gap after acknowledgement. The existing file +output is a troubleshooting copy, not automatic replay or immutable storage. +The first recovery claim covers a stopped ingress Collector and the durable +gateway outbox. Persisted Collector forwarding queues and their failure tests are +a later extension. [OpenTelemetry resiliency guidance](https://opentelemetry.io/docs/collector/resiliency/) + +In future validation, a controlled receiver/filter test must accept a fixture +batch while omitting one known span downstream. The comparison must report that +ID missing even with an empty outbox. Report `not found after 60 seconds`, with +query completeness and time range, rather than asserting irreversible loss from +an empty screen. Do not reset delivery flags, rewrite journal events or invent a +new event to repair the demonstration. No continuous reconciliation daemon is +needed for this finite experiment. + +## Learning slices + +Assume basic terminal/Python familiarity, a working Docker installation, assisted +implementation between exercises and a small fixture dataset. Allow roughly +**7–11 hours of active learning and implementation**, spread across sessions. +Image downloads, Docker repair, occupied ports and version incompatibilities are +additional uncertainty; these are estimates, not promised completion times. + +| Slice | What you learn | Proposed change | What you personally do | Visible result and verification | Estimate | +| --- | --- | --- | --- | --- | --- | +| 0. Follow the task | Permission, execution and human decision are separate | Establish a baseline using existing commands; record fresh run/trace IDs | Predict where the run will stop; inspect its artifacts and explain `awaiting_review` | Original dashboard and journal agree; no decision was fabricated | 0.5–1 h | +| 1. Find its trace | A trace groups observed events; collection is a separate route | Add the opt-in backend and forwarding config, pin versions, provision data sources | Paste the trace ID in Grafana Explore; inspect Builder completion and distinguish it from `run.create` | Same IDs and original timestamps in journal and Tempo; one fresh run becomes queryable within the tested budget | 1.5–2.5 h | +| 2. Read useful numbers | Metrics, labels, independent collection and absence | Add five read-only OTel metrics and a small dashboard | Write a query for review-pending runs and one for pending events; compare an empty queue with an unavailable endpoint | Auth/aggregation tests pass; replay does not increase run count; missing data is visibly different from zero | 2–3 h | +| 3. Diagnose an intervention | Slow work, failure, pending review and missing observations differ | Add bounded fault cases, direct Collector scrape, one alert and finite reconciliation | Predict the signal for each fault, introduce it, name two pieces of evidence, then recover | Controlled failure visible; alert changes state; recovery retains IDs; accepted-but-missing test is detected | 2–3 h | +| 4. Explain and reproduce | A claim needs evidence and a boundary | Finish provisioning, startup guide, demo script and evidence notes | Recreate provisioned views, run the three-minute demo, explain a duplicate and a missing-data case without assistance | Automated checks plus manual UI review pass; another checkout can follow the guide after authorized setup | 1–1.5 h | + +Do not move to the next slice merely because code exists. End each session with +one prediction, one observed result and one limitation in your own words. Phase 0 +is an early checkpoint to simplify the next session if the lab remains confusing. + +Short query exercises, proposed for the chosen versions: + +```text +TraceQL: { span.lab.run_id = "" } +TraceQL: { span.lab.action = "worker.failed" } +TraceQL: { span.lab.action = "worker.completed" && span:duration > 4s } +PromQL: lab_runs{state="awaiting_review"} +PromQL: lab_telemetry_pending_events +PromQL: up{job="lab-collector"} +PromQL: absent_over_time(lab_metrics_snapshot_timestamp_seconds{job="lab-gateway"}[15s]) +``` + +Explain why the slow-worker query is more appropriate here than overall trace +duration. Explain why a retry changes delivery evidence but not `lab_runs`. +Explain why an old sample is insufficient to conclude that the current value is +zero. The syntax follows [TraceQL](https://grafana.com/docs/tempo/latest/traceql/construct-traceql-queries/) +and [Prometheus functions](https://prometheus.io/docs/prometheus/latest/querying/functions/); +live exercises remain unverified until the selected versions are running. + +## Implementation map and future acceptance + +All paths in this table are proposals, not files implemented by this plan. + +| Area | Expected files | Acceptance evidence | +| --- | --- | --- | +| Optional stack | `observability/grafana/compose.yaml`, `collector.yaml`, `prometheus.yaml`; `Makefile` | Effective Compose config preserves isolation, host ports and data; pinned backend starts when explicitly requested | +| Metrics | `src/systems_lab/metrics.py`, narrow changes to `gateway.py` and `journal.py`; `pyproject.toml`, `uv.lock` | Viewer-only reads; accurate state counts beyond 100 runs; consistent snapshots; no sensitive labels or false zeros | +| Reproducible UI | `observability/grafana/provisioning/`, `observability/grafana/dashboards/agent-telemetry-reliability.json` | Same sources, dashboard and alert from files; screen has freshness, run states, backlog/age and a trace drill-down | +| Fault exercise | `src/systems_lab/cli.py`, `orchestrator.py`, `worker.py`, or a small dedicated demo module | Scoped fixture delay/failure reaches existing lifecycle handling; unknown cases rejected; normal execution unchanged | +| Evidence comparison | Bounded read-only command/helper and `tests/test_observability_integration.py` | Expected unique spans present after recovery; duplicate delivery tolerated; known missing span detected; incomplete queries never pass | +| Learning material | `experiments/agent-telemetry-reliability/README.md`, `demo.md`; small updates to `docs/observability.md`, `docs/development.md`, `README.md` | Hypothesis, variables, inputs, actual observations, limits, startup/stop/recovery instructions and demo are readable and repeatable | + +Keep dependencies under `uv` and the lockfile. Before changing Python or Make +targets, reload their contextual rules. Begin behavior changes with a failing +behavior test when practical: retry versus task count, empty versus unavailable +metrics, review versus failure, sensitive sentinel exclusion, and missing spans +after acknowledgement. Use isolated temporary data, not the user's history. + +Future checks: relevant pytest tests while iterating; `make check`, +`make spec-check`, and `git diff --check` for the finished implementation. Add a +canonical Make target for opt-in LGTM integration checks before documenting it as +available. Run the existing container isolation suite plus the new integration +target only after implementation and service-start authorization. Check actual +UI rendering and query/alert outcomes, not just HTTP status or container health. + +Proposed Make target names are `grafana-up`, `grafana-down`, `grafana-demo` and +`grafana-check`; **none is available yet**. They should encapsulate the explicit +base/override files and required profiles. Starting services stays explicit; +`grafana-check` tests a deliberately started stack. Save comparison evidence as +only IDs, actions, timestamps, hashes, versions and counts, not full API bodies. + +Acceptance requires following one run ID into its trace; diagnosing the known +fault from specific events and independent metrics; seeing pending delivery +recover without new canonical events; verifying the existing journal prefix and +hash checkpoint still agree; preserving timestamps/IDs on resend; demonstrating +missing data versus zero; and proving that a viewer cannot grant, approve or +revoke. A duplicate-receive test must simulate receiver acceptance followed by +local acknowledgement failure. Backend deduplication must be observed, not assumed. + +## Three-minute demonstration to prepare + +Prerequisite: images already available, stack ready, panels provisioned, versions +recorded, and one trace from each short worker fault prepared. Setup and image +downloads are not part of the three minutes. The exact timing must be rehearsed. + +| Time | Your action and explanation | +| --- | --- | +| 0:00–0:35 | Show a fresh normal run, its ID and trace. Point out the completed specialists and `awaiting_review`: the next action belongs to a person. | +| 0:35–1:00 | Open the prepared slow/failed Builder traces. Identify the measured worker duration or `worker.failed`; distinguish a technical failure from a review checkpoint. | +| 1:00–1:40 | Stop only the lab Collector and create one small run. Show that the gateway still advances, its direct metrics remain fresh, the Collector target fails and pending events accumulate. Show alert attention if the tested timing budget permits. | +| 1:40–2:25 | Restart the Collector. Watch the queue drain, reopen the same trace, and compare expected unique IDs with query results. The graph uses event time, so recovered events may appear earlier in the timeline. | +| 2:25–3:00 | Explain that a resend is not a new task, zero needs a successful read, and next-hop acceptance is not a permanent-storage guarantee. Show the saved missing-span test result and note the whole-stack outage limit. | + +Record what actually occurred; lengthen the demonstration or label a prepared +capture when recovery takes longer. Never shorten configured waits secretly or +present a prerecorded result as a live one. Actual results remain unmeasured. + +## Decisions and deferred work + +The first version answers where a fixture workflow is waiting and whether its +observations reached the local backend. It leaves permissions and historical +truth in the gateway, and produces reproducible configuration, one dashboard, +one rule, a startup guide, a demo and original local evidence under `.lab/`. +[Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) +and [alert provisioning](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/) +provide the intended file-based delivery mechanism. + +Defer correlated application logs, duration histograms, full worker tracing, +OTLP metric push, persistent downstream forwarding/replay, continuous loss +reconciliation, retention engineering and production availability. These each add +concepts or reliability claims beyond the first diagnostic. Loki or other +services included by the bundle do not imply application integration with them. + +No Kubernetes, fleet management, paid model route, Grafana Cloud subscription, +public deployment, new control plane or automatic human decisions are included. +The professional takeaway is an evidence-backed discussion of ingestion, +backpressure, missing signals and tradeoffs; this experiment does not demonstrate +production Ingest scale or model quality. + +## Planning record + +The related [OpenSpec proposal](../openspec/changes/agent-telemetry-reliability/proposal.md) +contains the reviewable acceptance contract and pending tasks. Existing +specifications and the unrelated Review Capacity proposal are not modified. +Current planning validation will be recorded there. This plan authorizes no +implementation or service execution by itself. + +### Publication checkpoint: 6 October 2026 + +After the planning inspection, the primary checkout's local `main` was advanced +to remote `main` at `8bf1669a68a9072326d35abfb18ee979a3766531`. The published +Review Capacity simulation and learning guide are now present in this checkout. +The four older untracked Review Capacity drafts were preserved in an ignored +local backup before replacing them with the tracked published versions. The +five new telemetry documents were preserved byte-for-byte during synchronization; +the separate implementation checkout was not changed. + +Publication validation passed `make check` (83 tests passed; five opt-in +container tests skipped, plus lint, formatting and Compose validation) and +`make spec-check` (five items). Local links, code fences and whitespace were +checked. These checks validate the existing repository and proposal structure; +the telemetry experiment is still unimplemented and its runtime outcomes remain +unverified. No experiment service was started for this publication. From 74cb26ac4ca03bae9c7eab39dafe8fd4508ded18 Mon Sep 17 00:00:00 2001 From: saski Date: Wed, 7 Oct 2026 14:57:00 +0200 Subject: [PATCH 2/2] docs: align telemetry learning with Grafana frontend role --- README.md | 1 + ...-07-grafana-frontend-learning-alignment.md | 289 ++++++++++++++++++ 2 files changed, 290 insertions(+) create mode 100644 docs/plans/2026-10-07-grafana-frontend-learning-alignment.md diff --git a/README.md b/README.md index 7d273ea..ca10185 100644 --- a/README.md +++ b/README.md @@ -117,6 +117,7 @@ integration into the Activity dashboard remains pending. - [Architecture and trust boundaries](docs/architecture.md) - [Development and validation](docs/development.md) - [Evolution roadmap](docs/roadmap.md) +- [Grafana Frontend learning alignment proposal](docs/plans/2026-10-07-grafana-frontend-learning-alignment.md) - [OpenSpec requirements](docs/openspec/) The first slice deliberately limits tools to describing, simulating, and diff --git a/docs/plans/2026-10-07-grafana-frontend-learning-alignment.md b/docs/plans/2026-10-07-grafana-frontend-learning-alignment.md new file mode 100644 index 0000000..2603491 --- /dev/null +++ b/docs/plans/2026-10-07-grafana-frontend-learning-alignment.md @@ -0,0 +1,289 @@ +# Grafana Frontend: bounded learning alignment + +Status: **research and proposal only; not implemented**. +Prepared October 7, 2026. Adds a product/frontend learning track to the +[existing telemetry reliability proposal](2026-10-06-agent-telemetry-reliability-lab.md). +It does not replace that proposal, approve its implementation or change the +gateway, permissions, agent runtime or funding boundaries. + +## Why Adjust The Learning Goal + +The learner now prioritizes a frontend engineering-management opportunity. +The most useful additional question is not simply whether telemetry arrives, +but whether a person can find the right resource, understand its state and keep +context while moving between views. + +Keep the existing distinction between execution, human review and observation +delivery. Add a narrow interface/navigation lesson around it. This remains a +useful lab experiment, not a reconstruction of Grafana's production platform or +proof of professional open-source leadership. + +## Observed Starting Point + +Re-inspection on October 7 of the primary checkout at `13b7ac6` found: + +- A read-only browser dashboard in `src/systems_lab/static/`, using JavaScript, + DOM rendering and authenticated same-origin API reads; no React application. +- Journal state, run selection and a trace ID are already visible. Stale-feed + handling and some keyboard/ARIA behavior exist; no new accessibility audit was + run, so do not claim either complete compliance or total absence of support. +- The Collector exports to a local file; no Grafana frontend extension or live + Grafana integration is demonstrated by this checkout. +- The October6 integration plan is still explicitly a proposal. Its future + target names must not be presented as current commands. + +Current branch: `docs/agent-telemetry-reliability-proposal`. GitHub metadata for +[PR #2](https://github.com/saski/agent-systems-lab/pull/2) reports open and not +merged, with head `13b7ac61c707329f3109e7c1a801ca469d57b5a9`. The pre-existing +README link and this untracked addendum were present before this refinement. +The README and other worktrees were left untouched. The chat +"Aprender Grafana con Agent Systems Lab" was read as historical context; the +current checkout and PR were checked separately. + +The [current UI source](../../src/systems_lab/static/app.js) keeps run selection +in memory and renders the trace ID as text. It has keyboard activation on run +rows, empty/detail states and a stale-feed indicator. No URL-based run selection +or Grafana trace link was found. A run-detail fetch failure currently reaches a +console warning; a complete visible loading/error contract is not established. +These are static observations, not a browser accessibility or usability result. + +The [lab Collector image](../../observability/Dockerfile) is +`otel/opentelemetry-collector-contrib:0.161.0`; its +[configuration](../../observability/collector.yaml) exports traces to a file. +There is no selected LGTM image in the current Compose configuration. Source +and existing tests support the journal/outbox/OTLP implementation, but were not +executed again for this documentation task. + +Only documentation was changed for this addendum. No Grafana/browser session +or containers were exercised for the proposed learning track. Existing automated +repository checks are recorded separately below. Other worktrees remain outside +its modification scope. + +## One Concrete Story + +An operator sees a run that has apparently stopped. They locate it, open its +detail and follow its trace. They discover that the work is waiting for a human, +while some observations are delayed. They return to the overview without losing +the selected run or relevant time context. + +```mermaid +flowchart LR + O["Operator"] --> V["Overview: find a run"] + V --> D["Run detail: understand state"] + D --> T["Trace: inspect evidence"] + T --> D + D --> V + J[("Journal: source of task truth")] -.-> D +``` + +This is a proposed user journey, not an implemented routing architecture. +Grafana is a diagnostic interface, not an authority to approve tasks or change +gateway permissions. + +## Hypothesis, Boundary And Observations + +Hypothesis: clear resource naming, preserved navigation context and explicit +data freshness help the learner distinguish the next human action from an +observation problem with fewer wrong turns. + +Boundary: one local learner, a bounded fixture dataset and the existing gateway +and diagnostic views. The first session may use the public Grafana demo and +documentation for product familiarization, without account creation. A real +lab-to-Grafana journey requires separately approved integration first. + +Compare the same task and dataset before/after one navigation improvement. +Record completion, elapsed time, wrong turns and misinterpreted states as actual +observations. One learner or a tiny convenience sample is not a general product +benchmark. An unperformed comparison has no result. Distinguish personal work, +agent-generated work, human review and observed behavior. + +## Small Learning Slices + +| Slice | Personal exercise | Proposed artifact | Completion evidence | Planning estimate | +| --- | --- | --- | --- | --- | +| 1. Product walkthrough | Find a diagnostic resource, inspect its state and describe what information is missing | A short journey map and two evidence-backed improvement hypotheses | The learner explains the path, naming and user problem without assistant prompts | 45-90 minutes | +| 2. Context-preserving navigation | Follow overview/detail/trace and return; use keyboard and a non-sensitive deep link | First try Grafana's built-in links and retain the existing journal UI; at most a small change to its selection/link behavior | Selection and a fixed time window survive back/reload; unavailable detail is visibly distinct from empty data; no credential enters URLs | 1-2 hours after an available integration; stop and re-estimate if a new adapter is needed | +| 3. Optional frontend extension | Own and explain one small React/TypeScript change using standard Grafana extension tooling | One read-only app-plugin path, not a rewrite of the dashboard or a new backend | Tests and actual browser states; behavior explained without relying on generated code | 3-6 hours, compatibility/setup may add time | +| 4. Platform and leadership judgment | Prioritize one foundation and one user improvement from inspected evidence | One-page brief: user, evidence, options, decision, measure and limitation | The learner challenges an AI-generated recommendation and explains the retained decision | 45-60 minutes; optional second-person walkthrough later, with no contact implied | + +Estimates are for bounded assisted learning, not delivery commitments. The +application-preparation lane is slices 1 and 4: **1.5-2.5 hours**, without +installing or implementing anything. A documented public-product walkthrough is +product familiarity, not proof that the local integration works. + +The smallest later live tranche reuses original telemetry slices 0-1 +(2-3.5 hours) and adds slice 2 here (1-2 hours): **3-5.5 hours**, excluding +environment/setup problems. It establishes a trace/navigation path, not the +complete reliability experiment. The original full proposal's **7-11 hours** +already includes its slices 0-1; do not add those hours twice. Metrics, the +alert and controlled failures remain pending until their original acceptance +checks pass. Slice 3 here is optional and never a career-application prerequisite. + +## Keep, Add And Defer + +| Decision | Scope | Reason and completion boundary | +| --- | --- | --- | +| Keep | Gateway permissions, canonical journal, persistent at-least-once outbox, stable IDs and existing read-only UI | Learning remains about execution, human review and observation delivery. Resending spans never creates extra runs | +| Keep in the original reliability plan | Independent gateway/Collector observations, five bounded metrics, one local alert, failure/recovery and finite span reconciliation | These distinguish an observation outage from zero activity. Navigation alone does not complete those requirements | +| Add first | User-journey critique and a one-page prioritization brief | Useful for the Frontend interview without waiting for a running backend; hypotheses must be labelled | +| Add after approval | One real run-to-trace round trip, state feedback and keyboard/context checks | A bounded frontend/product lesson built on the existing UI and trace integration | +| Defer by default | React/TypeScript/Scenes plugin | Reconsider only after built-in facilities and a small existing-UI change fail a concrete task | +| Defer | New backend, dashboard rewrite, OSS contribution campaign, multitenancy, stack migration, logs/profiling and production hardening | These do not resolve the immediate career evidence gaps within the bounded exercise | + +Earlier static suggestions about repeated simulation or eager imports are not +prerequisites. Revisit only if they cause a measured blocker for this exercise. + +## Reuse Before Extending + +First explore the existing Grafana dashboard, folder and linking facilities. +[Folders](https://grafana.com/docs/grafana/latest/visualizations/dashboards/manage-dashboards/) +organize dashboards; they do not supply the lab's run catalogue. +The official [dashboard-link guide](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/manage-dashboard-links/) +describes preserving time and variable context. This does not establish that a +particular installed version supports our intended run flow; validate it later. + +Use this data boundary for the first live path: + +| View | Source | Smallest credible implementation choice | +| --- | --- | --- | +| Run overview and detail | Existing viewer-authenticated journal activity API | Retain the current UI. Its overview contains recent runs, not a global run count | +| Trace | Tempo, once the original integration is approved and demonstrated | Open a Grafana-generated trace link for the same trace ID and original event window; verify the link format against the selected version | +| Return route | Selected run and fixed time window | Try existing links/history first; if inadequate, propose a small URL/selection change in the existing UI and test it | +| Aggregate health overview, later | Original plan's direct authenticated metrics | Use native dashboards; no run/trace IDs as metric labels and no reconstruction of canonical task state from received spans | + +There is no inspected Grafana data source for the journal API. Do not quietly +add a proxy, plugin or unauthenticated endpoint to make every view live inside +Grafana. A cross-application journey is sufficient for the first learning goal. +If a unified Grafana run catalogue becomes a real requirement, bring back a +separate data-access decision before increasing scope. + +If an actual user task needs a custom view, evaluate a minimal app plugin using +official scaffolding and [Scenes](https://grafana.com/developers/scenes/). +[Scenes app documentation](https://grafana.com/developers/scenes/scene-app/) +describes routing, breadcrumbs, tabs and drill-down. It is a learning option, +not evidence that the hiring squads use this exact library for every feature. +No package versions or implementation/API contract is selected by this plan. + +A custom view may begin with explicitly labeled, sanitized frozen fixtures. +Such a demo proves interaction behavior only, not current task state or real +delivery. A later live-data path must use supported authenticated data sources +and preserve isolation; do not move gateway/provider credentials into a plugin, +introduce an unrestricted proxy, or expose a private network to make it work. + +### Version And Store Check + +The [LGTM overview](https://grafana.com/docs/opentelemetry/docker-lgtm/) describes +a development/demo/test bundle and names Mimir. However, the +[release endpoint](https://github.com/grafana/docker-otel-lgtm/releases/tag/v0.35.0) +resolved to `v0.35.0` during this review, and that tag's +[Dockerfile](https://github.com/grafana/docker-otel-lgtm/blob/v0.35.0/docker/Dockerfile) +declares Grafana `v13.2.2`, Prometheus `v3.15.0`, Tempo `v3.0.3`, Loki `v3.7.8`, +Pyroscope `v2.3.1` and Collector `v0.161.0`. +Its [data-source configuration](https://github.com/grafana/docker-otel-lgtm/blob/v0.35.0/docker/grafana-datasources.yaml) +points at local Prometheus, Tempo, Loki and Pyroscope services. + +This is a documentation/source discrepancy, not evidence that the lab needs a +Mimir migration. Prefer a tested, pinned release's actual configuration. The tag +is a research candidate, not a selected or downloaded local image: registry +digest, actual bundled versions, architecture compatibility and resource use +remain runtime checks after approval. Included stores do not imply the lab uses +their pipelines. Keep the original Prometheus-oriented design pending that check. + +## Quality And Evidence Gates + +For any later frontend slice, start with one user-visible behavior test. Cover +context retention, keyboard navigation, loading, empty, stale and error states. +Missing or unavailable information must not become a healthy zero. Screenshots +cannot replace behavior checks; a successful build cannot establish usability. + +For the round trip, retain the same run ID and explicit from/to interval on +back/forward and reload, preserve a sensible keyboard focus destination, and +announce loading/errors where needed. Test successful empty data separately +from an unavailable read. Previously loaded data may remain visible only with +its original freshness and an explicit stale indication. A frozen fixture must +stay labelled as a fixture; generating a fresh timestamp does not make it live. + +Keep at least one explainable trade-off: built-in capability versus extension, +shared foundation versus user feature, or helpful AI synthesis versus an +unsupported recommendation. Use the original telemetry fault cases when +available; do not invent production incidents or synthesize fresh timestamps. + +Record one management-oriented AI use with its source material, proposed output, +human correction and decision. Research synthesis, feedback grouping or a +prioritization brief can be useful tasks. Do not equate generation volume with +quality or imply a measured workplace improvement from this local exercise. + +## What This Does Not Fulfill + +- Prior accountability for two real engineering squads or formal management of + managers. Those need historical evidence and role-scope clarification. +- Production ownership of an OSS ecosystem, multitenant platform or management + incident rota. A local single-user environment is not a substitute. +- Experience with paid Grafana Cloud/Assistant capabilities. Do not imply they + exist in the local OSS bundle or authorize an account, payment or live provider. +- Measured AI effects at Eventbrite or generalizable usability results. + +An application narrative should use existing professional evidence first and +name this work as ongoing learning, with its implementation status kept current. + +## Implementation Handoff, Not Authorization + +**Concrete decision for later approval:** implement only the original trace +integration slices 0-1 plus the navigation slice above, within a 3-5.5-hour +learning estimate. Keep the default activity dashboard and authorization +boundary, use fixture mode, and verify one real round trip. Stop for a scope +decision if this needs a new data adapter or plugin. Do not include React/Scenes, +new providers, metrics/alert/fault completion, publication or another worktree +in that first tranche. This decision is ready for review, not authorized now. + +Before implementing, confirm the chosen slice and reload applicable Python, +React and Makefile rules for any affected files. Pin the actual Grafana/plugin +versions, preserve user changes and capture the baseline. Prefer the original +telemetry proposal's smallest useful integration before adding a plugin. + +Use the repo's `make check` for implementation changes and `make spec-check` +when changing OpenSpec. Add focused frontend/browser validation to the canonical +workflow if a frontend extension is approved. No new target is claimed available +here. Full Grafana runtime and usability checks remain future work. + +This proposal changes documentation only. Source, dependencies, services and +OpenSpec requirements retain their existing behavior. Publishing or merging the +proposal does not authorize runtime implementation. It can be accepted, reduced +or deferred separately from the opportunity decision. + +## Earlier Documentation Verification + +Local links, Markdown fences and whitespace checks passed. `git diff --check` +passed. An initial sandboxed `make check` could not bind localhost sockets in +three existing tests; an authorized rerun passed Ruff,83 tests and Compose +configuration validation, with5 container tests skipped. Those legacy checks +do not validate the proposed Grafana/plugin integration. Existing tests used +ephemeral local listeners; no persistent service or new dependency was started. + +Those checks belong to the earlier October 7 task. The current refinement did +not rerun `make check` or `make spec-check`: source, dependencies and OpenSpec +were unchanged, and no runtime validation is claimed. Current document checks +passed for this addendum and the three associated career documents: 28 local +links, balanced fences, final newlines and whitespace. `git diff --check` passed; +the untracked addendum was checked separately. README and original-plan hashes +match the starting state. Git still distinguishes the open published proposal +from this untracked addendum. + +## Publication Scope + +This addendum and its README link are included in the documentation scope of +[PR #2](https://github.com/saski/agent-systems-lab/pull/2), alongside the original +telemetry reliability proposal. The inspection and verification records above +describe their respective pre-publication snapshots. + +The next learning step remains product familiarization and a prioritization +brief. The proposed live trace/navigation tranche requires a separate +implementation decision. Separating validation from simulation remains an +independent improvement, not a prerequisite for accepting these documents. + +Publication preparation on October 7 passed `make check` (Ruff, formatting, +83 tests and Compose configuration; 5 opt-in container tests skipped), +`make spec-check` (5 items), and checks of both changed documents and their +18 local links. The successful test run used permission for ephemeral localhost +listeners after the sandbox blocked three existing tests. These checks do not +demonstrate the proposed Grafana integration or frontend behavior.