Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ integration into the Activity dashboard remains pending.
- [Architecture and trust boundaries](docs/architecture.md)
- [Development and validation](docs/development.md)
- [Evolution roadmap](docs/roadmap.md)
- [Grafana Frontend learning alignment proposal](docs/plans/2026-10-07-grafana-frontend-learning-alignment.md)
- [OpenSpec requirements](docs/openspec/)

The first slice deliberately limits tools to describing, simulating, and
Expand Down
56 changes: 56 additions & 0 deletions docs/openspec/changes/agent-telemetry-reliability/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Design

The detailed design, verified source references, alternatives, metric contracts,
failure matrix, learning slices and estimated effort are in the
[learning plan](../../../plans/2026-10-06-agent-telemetry-reliability-lab.md).
It is the design reference for this change; all runtime changes are proposed.

## Decisions

1. Use the existing fixture workflow and persistent gateway, independent of the
dirty Review Capacity implementation. Preserve existing source changes.
2. Retain the lab Collector as the stopped component in the outage exercise.
Forward to the receiver bundled with LGTM. Grafana and Prometheus must remain
alive when that ingress Collector stops.
3. Use OTel metrics with a direct Prometheus scrape from an authenticated gateway
endpoint. Neither application freshness nor Collector reachability depends
exclusively on the stopped Collector's OTLP pipeline.
4. Derive current task counts from canonical operational state, never received
spans or delivery attempts. Read-only aggregates have bounded labels and
distinguish legitimate zeros, disabled export, failed reads and stale data.
5. Compare expected journal identities against complete Tempo trace results for
one small run. At-least-once delivery, downstream gaps and query incompleteness
stay visible; no production loss detection or deduplication guarantee is made.
6. Keep host control, gateway permissions, worker isolation and human approval
unchanged. Only a scoped fixture demo path may inject bounded delay/failure.

## Risks and mitigations

- **Bundled backend failure:** the observer can fail with the backend. Retain the
original gateway dashboard and a host-side diagnostic; do not claim HA.
- **Post-acknowledgement loss:** the durable gateway outbox covers its own
acknowledgement boundary. Finite reconciliation can reveal missing downstream
spans; persisted Collector recovery remains deferred.
- **False interpretation:** event spans are not complete worker instrumentation;
policy denial and human rejection are not inherently technical crashes;
logical ticks are not seconds. Teach and test these distinctions.
- **Version drift:** pin tested image/SDK versions during implementation and
verify provisioning paths, metric names, TraceQL and alert semantics.
- **Data leakage or authority expansion:** preserve explicit resources and
attribute allowlists, viewer-only metrics, private networks and no operator
credentials in observation services.
- **Existing work or runtime collision:** review both checkouts again before
implementation; use additive files, explicit profile/ports and fresh evidence;
do not reset histories, acknowledgement flags or other running services.

## Validation and rollback

Use failing behavior tests, then relevant unit tests, canonical repository checks
and an explicitly started opt-in integration suite. Inspect the actual UI and
document observed timing and missing checks. Existing specifications are not
synced or archived until the implementation meets the acceptance contract.

The optional profile can be stopped without removing data. Restore the prior
explicit exporter destination when leaving the exercise; do not replay previously
acknowledged history automatically. Preserve all run evidence and configuration
needed to reproduce the observation. No destructive cleanup is part of the demo.
85 changes: 85 additions & 0 deletions docs/openspec/changes/agent-telemetry-reliability/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Agent Telemetry Reliability Lab

## Why

A quiet activity screen can mean slow or failed execution, a legitimate human
review checkpoint, or interrupted telemetry. The lab already has canonical
history and recoverable OTLP trace delivery, but lacks a queryable trace backend
and independent metrics for learning how to distinguish those conditions.

The primary outcome is learner understanding through a real fixture workflow,
queries and controlled failures. The [learning plan](../../../plans/2026-10-06-agent-telemetry-reliability-lab.md)
explains the baseline, options, exercises, sources, estimates and demonstration.

## What Changes

- Add an opt-in local LGTM backend, using the existing lab Collector as an
independently stoppable ingestion hop, with traces stored in Tempo.
- Add five read-only OTel metric families, scraped directly by Prometheus, plus
direct Collector self-metric collection outside its own delivery pipeline.
- Provision a compact Grafana dashboard and one local alert rule with explicit
missing/stale/error behavior.
- Add bounded fixture delay/failure exercises and a read-only comparison of a
finite run's expected journal spans against queryable Tempo evidence.
- Supply a startup guide, learner exercises, three-minute demo and limitations.

## Acceptance Scope

Fresh fixture runs, one gateway/exporter per database, normal execution,
human-review pending, short worker delay/failure, a stopped ingress Collector,
recovery and retransmission, and explicit missing-data evidence. Recovery must
preserve canonical history and event identities. Accepted-but-missing downstream
evidence must be detectable by finite comparison, without claiming permanent
retention or automatic replay after acknowledgement.

Metrics and Grafana remain observers. They gain no task-execution authority.
No Review Capacity logical ticks are exported as live workflow time.

## Non-Goals

Implementation in this planning task; production monitoring; Kubernetes; fleet
management; a new control plane; Grafana Cloud; public deployment; external
notifications; live/paid providers; worker credential/network expansion; automatic
approval; correlated application logs; full worker instrumentation; persistent
downstream recovery or general exactly-once delivery.

## Capabilities

### New Capabilities

- `agent-telemetry-reliability`: a local, reproducible observation and diagnosis
experiment with independent delivery evidence and explicit learning checks.

### Modified Capabilities

None. Existing `observable-activity`, `governed-execution` and
`systems-experiments` requirements remain the baseline and must keep passing.

## Impact

Expected changes after approval: optional Compose/configuration/provisioning
files, narrow gateway metric reads, fixture-only demo controls, integration
tests, Make targets, locked exporter dependency and user/development guides.
No new source or runtime configuration is implemented by this proposal.

Status: proposed for user review on 6 October 2026. Implementation is pending.

## Planning Validation

On 6 October 2026, `make spec-check` passed all five current OpenSpec items,
including this proposal. Local Markdown links, balanced code fences and
whitespace in the five new documents were checked. No existing tracked files or
the secondary checkout were edited. No service was started and no dependency
was installed. `make check` and runtime/integration tests were not run for this
documentation-only task; none of the proposed runtime outcomes is verified.

## Publication Validation

Later on 6 October 2026, local `main` was synchronized with remote `main` at
`8bf1669a68a9072326d35abfb18ee979a3766531`, preserving the new proposal and backing
up the older local Review Capacity drafts. Before publication, `make check`
passed: lint, formatting, 83 tests and Compose configuration validation; five
opt-in container tests were skipped. `make spec-check` passed all five items.
Local links, code fences and whitespace were checked. The unrelated secondary
checkout remains untouched. No experiment service was started and no proposed
runtime outcome is verified by these checks. Implementation remains pending.
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
## ADDED Requirements

### Requirement: local reproducible trace exploration
The experiment SHALL provide an opt-in, version-pinned local Tempo/Grafana
configuration for current gateway-observed OTLP spans, preserving the default
file-export path and gateway permission boundaries.

#### Scenario: follow a fresh fixture run
- GIVEN a newly created fixture run and the optional observation profile
- WHEN its pending telemetry is delivered
- THEN its journal trace and span identities are queryable in Tempo through Grafana
- AND original event times and fixture provenance are preserved
- AND the guide distinguishes event/completion spans from full worker instrumentation

#### Scenario: preserve existing evidence and isolation
- GIVEN existing local histories and the normal worker restrictions
- WHEN the optional profile is started and later stopped
- THEN no database, journal or telemetry data is deleted or silently replayed
- AND no worker receives new credentials, host mounts, Docker socket or network access
- AND only intended loopback ports are published
- AND resolved Compose configuration excludes observation services from database and worker networks

### Requirement: independent read-only metrics
The gateway SHALL expose authenticated, bounded-label OTel metrics for run state,
pending delivery count and age, exporter enablement and successful snapshot time.
Prometheus SHALL collect them without traversing the Collector delivery pipeline.

#### Scenario: count tasks independently of retransmission
- GIVEN distinct canonical runs, including more than the activity overview limit
- WHEN an observation is delivered repeatedly or the exporter restarts
- THEN metric run counts still equal current distinct run rows in that database
- AND IDs, hashes, actor instances and arbitrary text do not become metric labels

#### Scenario: preserve viewer authority
- GIVEN the read-only viewer credential used for metric collection
- WHEN it requests metrics or attempts to grant, decide or revoke a run
- THEN metrics can be read and operational writes remain unauthorized
- AND collection does not mutate canonical history
- AND no second unauthenticated metrics listener bypasses gateway viewer checks

### Requirement: explicit zero and missing-data semantics
Panels and alert evaluation SHALL distinguish successful zero values, disabled
export, missing series, stale snapshots and failed data-source queries.

#### Scenario: empty healthy lab
- GIVEN an enabled exporter, successful fresh database read and empty outbox
- WHEN the metrics are scraped
- THEN pending count and pending age are explicitly zero
- AND freshness evidence establishes why those zeros are meaningful

#### Scenario: failed or partial observation
- GIVEN a failed scrape, missing expected series or stale snapshot
- WHEN Grafana displays or evaluates the data
- THEN it indicates missing, stale or attention state as appropriate
- AND it does not substitute a healthy zero or silently retain a green state
- AND a surviving series cannot conceal a missing required series

### Requirement: meaningful local delivery alert
The experiment SHALL provision one local Grafana-managed alert rule for delivery
backlog, failed required scrape targets and stale snapshots, with explicit
No Data and Error behavior and documented evaluation timing.

#### Scenario: ingest Collector is stopped
- GIVEN the gateway, LGTM backend and direct scrape paths remain available
- WHEN only the lab Collector is stopped and a fixture run produces events
- THEN the task and journal continue independently
- AND pending delivery and failed Collector reachability are visible
- AND the alert enters attention within the documented test budget
- AND no external message or automatic operational action is sent

### Requirement: controlled fixture diagnosis
The experiment SHALL provide opt-in, bounded fault exercises that use the normal
gateway and worker lifecycle, without arbitrary commands or expanded authority.

#### Scenario: distinguish short delay from failure
- GIVEN a new fixture run with one selected Builder intervention
- WHEN a fixed delay or controlled nonzero worker exit occurs
- THEN measured worker completion duration or worker failure evidence identifies it
- AND normal failure handling retains run revocation and permission enforcement
- AND unknown fault modes and non-fixture use are rejected

#### Scenario: distinguish pending review from an incident
- GIVEN completed specialists and a run awaiting a digest-bound human decision
- WHEN observation paths are healthy
- THEN the UI identifies human review as the next step
- AND it does not classify waiting, human rejection or every policy denial as a technical failure
- AND Grafana cannot approve or resume the run

### Requirement: identity-preserving recovery and finite reconciliation
Recovery SHALL retain original event identities and timestamps, leave canonical
history unchanged by retries, and compare a bounded expected span set with
complete backend query results without assuming exactly-once receipt.

#### Scenario: recover a pending batch across exporter restart
- GIVEN an unavailable ingress Collector and durable pending journal events
- WHEN the exporter restarts with the same database and the Collector recovers
- THEN pending events become queryable with their original trace and span IDs
- AND the stored journal prefix and checkpoint still verify
- AND retransmission does not become a new task or canonical event

#### Scenario: receiver accepts before acknowledgement fails
- GIVEN a receiver accepted a batch but its local acknowledgement was not saved
- WHEN the gateway retries that pending batch
- THEN the repeated delivery retains the original identities
- AND the evidence comparison counts unique identities and reports receipt behavior honestly

#### Scenario: accepted evidence is missing downstream
- GIVEN a controlled receiver or filter omits one known fixture span after acceptance
- WHEN a bounded complete Tempo query is compared with expected journal identities
- THEN the missing identity is reported even if the gateway outbox is empty
- AND time range, query completeness and observation deadline are reported
- AND no claim of irreversible loss or automatic replay follows from that gap alone

#### Scenario: entire observation backend is unavailable
- GIVEN Grafana and Prometheus stop with the LGTM backend
- WHEN the operator diagnoses the missing view
- THEN the guide directs them to the independent gateway dashboard and host check
- AND it does not claim that the unavailable Grafana alert can detect its own outage

### Requirement: safe and honest learning evidence
The experiment SHALL preserve the telemetry allowlist and provide a hypothesis,
boundary, variables, observations, limits, learner exercises and reproducible
configuration, guide and demonstration.

#### Scenario: sensitive sentinels stay local
- GIVEN credentials, prompts, tool arguments, content and raw exception text in test inputs
- WHEN traces, metric samples and demo evidence are generated
- THEN none of those values appears in exported payloads, labels, panels or saved shared evidence
- AND an authentication credential appears only where required for transport access

#### Scenario: explain the outcome independently
- GIVEN completed learning slices
- WHEN the learner performs the prepared demonstration
- THEN they can query a run, predict and diagnose a fault, show recovery and explain a possible resend
- AND they distinguish observed wall time from Review Capacity logical ticks
- AND fixture evidence is not described as production performance or real-model quality
Loading