diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index fbcdf04999..8149338aca 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -150,3 +150,5 @@ /plugins/dotnet11/skills/system-text-json-net11/ @dotnet/skills-csharp-language-reviewers /tests/dotnet11/system-text-json-net11/ @dotnet/skills-csharp-language-reviewers +/plugins/dotnet11/skills/lightweight-telemetry/ @dotnet/skills-csharp-language-reviewers +/tests/dotnet11/lightweight-telemetry/ @dotnet/skills-csharp-language-reviewers diff --git a/README.md b/README.md index 78c47fd0f3..e6600c3e5f 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,11 @@ $ skill-installer install https://github.com/dotnet/skills/tree/main/plugins/ +- Agent Skills standard: + ## License See [LICENSE](LICENSE) for details. diff --git a/plugins/dotnet11/skills/lightweight-telemetry/SKILL.md b/plugins/dotnet11/skills/lightweight-telemetry/SKILL.md new file mode 100644 index 0000000000..c55e139bd6 --- /dev/null +++ b/plugins/dotnet11/skills/lightweight-telemetry/SKILL.md @@ -0,0 +1,166 @@ +--- +name: lightweight-telemetry +description: Emit structured metrics from a .NET 11 console app, CLI, or build tool using the built-in System.Diagnostics.Metrics API with no OpenTelemetry, APM, or collector dependency. Use when asked to "report how long it took", "count how many times it ran", expose queue depth or a running total, pick between a counter, gauge, and histogram, split one metric by a tag/dimension, keep the measurement path cheap when nothing is listening, or print readings as JSON lines from a short-lived process. Do not use for distributed tracing across services (use configuring-opentelemetry-dotnet), cloud ingestion into Application Insights or Azure Monitor, or shipping log lines to Seq/Elasticsearch. +license: MIT +--- + +# Lightweight telemetry in .NET 11 + +A minimal, dependency-free way to expose operational metrics from a CLI or tool. +No OpenTelemetry SDK, no external collector required — metrics are written to the +console as structured lines and can be scraped or redirected. + +## When to use + +- A build tool, CLI, or local agent needs to report timing/counts. +- You want structured telemetry without an APM vendor SDK. +- The host may be resource-constrained (no background collector). + +## When not to use + +- You need distributed tracing across services → use the + `configuring-opentelemetry-dotnet` skill instead. +- You need cloud ingestion (Application Insights) → use the vendor SDK. + +## Pick the instrument first + +The instrument type is the decision that is most often wrong, and it is not +recoverable downstream — a consumer cannot turn a gauge back into a rate. + +| The value is | Use | Never use | Because | +|---|---|---|---| +| A total that only grows (bytes processed, runs) | `CreateCounter` | a gauge | the consumer derives the rate from the increasing total; a gauge that resets destroys it | +| The value right now (queue depth, open handles) | `CreateGauge` | a counter | a cumulative sum misrepresents a level that goes down again | +| A per-operation duration or size you want percentiles for | `CreateHistogram` | a counter | summing durations loses the distribution | +| A level you can only sample when asked | `CreateObservableGauge` | recording in a hot loop | the callback runs at collection time | + +Always pass the unit and description — put the unit in the **metadata**, not only +in a `.ms` name suffix, or a consumer cannot tell seconds from milliseconds: + +```csharp +meter.CreateHistogram("tool.step.duration", "ms", "Duration per build step"); +``` + +## Split a metric by a dimension, not by name + +One instrument plus a tag, never one instrument per value: + +```csharp +stepDuration.Record(elapsedMs, new TagList { { "step", "restore" } }); +``` + +Tag **values** must come from a bounded set (step names, status codes). Never tag +with a user id, path, or timestamp — each distinct value is a separate time +series downstream. + +## Keep the hot path cheap + +`Record`/`Add` are cheap, but building the tags and formatting values is not. +Guard the expensive part when nothing is collecting: + +```csharp +if (stepDuration.Enabled) // false when no listener is attached + stepDuration.Record(elapsedMs, new TagList { { "step", step } }); +``` + +Use `TagList` (a struct) rather than allocating a `KeyValuePair[]` per iteration. + +## Lifetime: set up the listener before the first measurement + +A `MeterListener` only sees measurements recorded **after** `Start()`. In a +short-lived CLI this is the difference between output and silence: + +```csharp +var listener = BuildListener(meter); // Start() called inside +// ... all recording happens after this point ... +listener.RecordObservableInstruments(); // pull observable gauges once before exit +listener.Dispose(); +meter.Dispose(); +``` + +Verified against the pinned preview SDK (`11.0.100-preview.3.26207.106`, +`net11.0`): a measurement recorded before `listener.Start()` produces **no** +output line, one recorded after it produces exactly one. Observable instruments +emit nothing at all unless `RecordObservableInstruments()` is called, so a +process that exits without it reports nothing for them. + +## The pattern + +Use `System.Diagnostics.Metrics.Meter` to define a counter and a histogram, drive +time measurement with `TimeProvider.System`, and flush a snapshot on exit. + +```csharp +using System.Diagnostics; +using System.Diagnostics.Metrics; + +var meter = new Meter("MyTool", "1.0.0"); // stable name = metric identity +var runs = meter.CreateCounter("tool.runs", "{run}", "Number of executions"); +var duration = meter.CreateHistogram("tool.step.duration", "ms", "Duration per step"); + +using var listener = new MetricListener(meter); // BEFORE the first measurement + +var clock = TimeProvider.System; // injectable, testable clock +var start = clock.GetTimestamp(); + +// ... work ... + +if (duration.Enabled) // skip tag building when idle + duration.Record(clock.GetElapsedTime(start).TotalMilliseconds, + new TagList { { "step", "compile" } }); +runs.Add(1); + +listener.Flush(); // pull observables before exit +``` + +Substitute a test `TimeProvider` (e.g. `Microsoft.Extensions.Time.Testing.FakeTimeProvider`) +to assert on recorded durations without sleeping. + +## Output contract + +Each reading is exactly one JSON object on its own line — no banner, no summary +line, nothing else on stdout. A consumer can `tail -f` and parse every line. + +```json +{"meter":"MyTool","instrument":"tool.step.duration","unit":"ms","description":"Duration per step","value":58.6,"tags":{"step":"compile"},"timestamp":"2026-08-29T18:32:07+00:00"} +``` + +Contract, in order: + +1. `meter` — the fixed `Meter` name, never per-run or per-environment +2. `instrument` — the stable instrument name +3. `unit` — from the instrument metadata, never only a name suffix +4. `description` — so a scraped reading is self-describing +5. `value` — the measurement +6. `tags` — object of the bounded dimensions +7. `timestamp` — ISO 8601, UTC + +Note that a non-standard unit is written in UCUM annotation form: a counter +measures `{run}` or `{item}`, not `runs` or `items`. That is the correct +convention, and it is what a consumer expects to see — do not "fix" it to a +plain noun. + +The listener must be constructed and `Start()`ed **before** the first `Record`/ +`Add`, and `RecordObservableInstruments()` must be called before exit. A +measurement taken before `Start()` produces no line at all — in a short-lived +CLI that is the whole difference between telemetry and silence. + +## Verify it works + +```bash +dotnet build -c Release # must succeed with 0 errors +dotnet run -c Release --no-build # one JSON line per measurement +``` + +If `run` prints nothing, the listener ordering above is the first thing to +check — not the instrument definitions. + +## Notes + +- `Meter`/`Counter`/`Histogram` are built into `System.Diagnostics.DiagnosticSource` + (no extra NuGet package for the API itself). +- For production scraping, attach an `IMetricsListener` or export to OTLP; this + skill intentionally stays at the smallest useful surface. +- Keep the meter name stable — it becomes the metric namespace downstream. The + meter *version* string is safe to bump; the name is not. +- One instrument + a tag beats one instrument per value, but keep tag values + bounded — unbounded values (ids, paths) create a time series each. diff --git a/tests/dotnet11/lightweight-telemetry/eval.yaml b/tests/dotnet11/lightweight-telemetry/eval.yaml new file mode 100644 index 0000000000..b1ca5d174b --- /dev/null +++ b/tests/dotnet11/lightweight-telemetry/eval.yaml @@ -0,0 +1,309 @@ +name: lightweight-telemetry +description: Evaluates the dotnet11/lightweight-telemetry skill +type: capability +defaults: + timeout: 6m + runs: 1 +stimuli: + - name: Structured metrics from a console tool without an APM SDK + prompt: | + I'm building a .NET 11 console tool and I want it to report how many times + it ran and how long each run took, as structured telemetry. I do NOT want to + pull in OpenTelemetry or any external APM SDK — the host has no collector and + is resource constrained. Show me a minimal `net11.0` program that reports a + count and a duration distribution and prints one structured line per + measurement. + tags: + capability: structured-metrics-emission + risk: external-sdk-dependency + journey: add-console-tool-metrics + graders: + - type: exit-success + - type: output-contains + config: + substring: System.Diagnostics.Metrics + - type: output-matches + config: + pattern: net11\.0 + - type: output-not-matches + config: + pattern: PackageReference.*OpenTelemetry + - type: prompt + rubric: + - Solves the problem with the metrics API that ships in the framework rather than an added OpenTelemetry or vendor APM package + - Provides both a cumulative count and a duration distribution, not just one of the two + - Targets net11.0 + - Emits machine-readable (structured/JSON) output per measurement rather than only a plaintext log line + + - name: Elapsed time measured through the framework clock abstraction + prompt: | + In a .NET 11 tool I need to measure how long an operation takes and record it + as a metric, and I want the timing to be testable — I don't want DateTime.Now + scattered through the code. Show a small `net11.0` snippet that takes the + start timestamp, computes the elapsed milliseconds, and records it. + tags: + capability: testable-elapsed-time + risk: untestable-wall-clock + journey: record-operation-duration + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: output-not-matches + config: + pattern: DateTime\.Now + - type: prompt + rubric: + - Obtains the elapsed time from an injectable clock abstraction (or an equivalent testable seam) instead of DateTime.Now + - Records the resulting duration as a metric measurement + - The timing code can be driven by a fake/controlled clock in a test without changing production code + - Targets net11.0 + + - name: Metric identity stays stable across releases + prompt: | + I'm adding metrics to several .NET 11 tools and I plan to scrape the emitted + values with an external system later. Give me a `net11.0` example of setting + up the metric source, and tell me what I must be careful about so my + dashboards and queries don't break when I ship the next version. + tags: + capability: stable-metric-identity + risk: dashboard-query-breakage + journey: define-stable-metrics + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: output-not-matches + config: + pattern: Guid\.NewGuid + - type: prompt + rubric: + - Uses an explicit, fixed metric source name rather than a generated or environment-derived one + - Explains that the source and instrument names form the identity downstream consumers query, so renaming them breaks existing dashboards + - Distinguishes the version string (safe to change) from the name (not safe to change) + - Targets net11.0 + + - name: Consume own measurements in-process with no extra packages + prompt: | + I have a .NET 11 tool that already records metrics with the built-in metrics + API, but nothing consumes them so I see no output. Without adding any NuGet + package, how do I subscribe to my own measurements in the same process and + write each reading to the console as a JSON line? Show a `net11.0` example. + tags: + capability: in-process-metric-consumption + risk: silent-unobserved-measurements + journey: consume-local-metrics + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: output-matches + config: + pattern: (Json|Serialize|serializ) + - type: output-not-matches + config: + pattern: PackageReference.*OpenTelemetry + - type: prompt + rubric: + - Subscribes to measurements in-process using a framework-provided listener, adding no NuGet package + - Explicitly opts the instruments in so callbacks actually fire (subscription alone produces nothing) + - Serializes each reading to JSON and writes it to the console + - Targets net11.0 + + - name: Instrument choice for an instantaneous value + prompt: | + I have a .NET 11 tool and I want to expose the number of items currently + queued as telemetry. A running total is wrong here — I need whatever the + value is right now. Show me a `net11.0` snippet that picks the right + instrument for that and publishes the value. + tags: + capability: current-value-instrument-selection + risk: cumulative-value-misrepresentation + journey: report-current-queue-depth + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: prompt + rubric: + - Chooses an instrument that reports the current value rather than a monotonically increasing sum + - Explains why a cumulative counter would misrepresent a queue depth + - Shows the value being published/observed, not merely the instrument being created + - Targets net11.0 + + - name: Instrument choice for a monotonic total + prompt: | + My .NET 11 tool processes files and I want to report the total number of + bytes it has processed since start, so an external system can compute a rate + from it. Show me a `net11.0` snippet with the right instrument for that and + explain why it is the right one. + tags: + capability: monotonic-total-instrument-selection + risk: incorrect-rate-semantics + journey: report-processed-byte-total + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: prompt + rubric: + - Chooses a monotonically increasing cumulative instrument for the byte total + - Explains that the consumer derives the rate from the increasing total, so the app must not reset or gauge it + - Does not model the running total as a distribution/percentile instrument + - Targets net11.0 + + - name: Split one metric by a dimension + prompt: | + In a .NET 11 tool I'm recording how long each build step takes. Right now all + the durations land in one bucket and I cannot tell "restore" from "compile". + I don't want a separate metric per step name. Show me a `net11.0` example + that fixes this. + tags: + capability: bounded-metric-dimensions + risk: metric-cardinality-explosion + journey: separate-build-step-durations + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: prompt + rubric: + - Attaches a key/value dimension to each measurement instead of creating one instrument per step + - Keeps a single instrument and passes the step name as the dimension value + - Warns (or by construction avoids) unbounded dimension values that would explode cardinality + - Targets net11.0 + + - name: Metadata that makes readings interpretable downstream + prompt: | + I'm defining metrics in a .NET 11 tool and a colleague scraping them cannot + tell whether a duration value is seconds or milliseconds, or what a metric + means. Show me a `net11.0` snippet that fixes that at the point where the + metrics are defined. + tags: + capability: metric-unit-description-metadata + risk: ambiguous-measurement-semantics + journey: define-interpretable-metrics + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: prompt + rubric: + - Declares the unit and a human-readable description on the instruments themselves rather than documenting them elsewhere + - Puts the unit in the metadata instead of relying only on a suffix in the metric name + - Shows the metadata reaching the consumer/reading output + - Targets net11.0 + + - name: Keep the measurement path cheap when nothing is listening + prompt: | + I want metrics in a hot loop in my .NET 11 tool, but most of the time nobody + is collecting them and I don't want to pay for building tag arrays and + formatting values on every iteration. How do I keep that path cheap? Show a + `net11.0` snippet. + tags: + capability: disabled-instrument-fast-path + risk: hot-path-allocation-overhead + journey: optimize-unobserved-measurements + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: prompt + rubric: + - Checks whether the instrument is actually being collected before doing the expensive work of assembling the measurement + - Keeps the always-executed path allocation-light (no per-iteration allocation of tag collections or strings) + - Still records correctly when a consumer is attached + - Targets net11.0 + + - name: Nothing is lost when the tool exits + prompt: | + My short-lived .NET 11 CLI records metrics, but when the process exits I + sometimes see no output at all for the last operations. Show me how to + structure a `net11.0` program so the recorded values are all accounted for + before it terminates. + tags: + capability: short-lived-listener-lifetime + risk: dropped-final-measurements + journey: flush-short-lived-tool-metrics + graders: + - type: exit-success + - type: output-matches + config: + pattern: net11\.0 + - type: prompt + rubric: + - Ensures the consumer/listener is set up before the first measurement is recorded, not after + - Disposes or flushes the metric source and its consumer deterministically before the process exits + - Explains that a short-lived process can terminate before pull-based collection ever happens + - Targets net11.0 + + - name: Cross-service tracing request stays out of scope + prompt: | + My .NET 11 service needs distributed tracing across several microservices, + with spans, context propagation, and a Jaeger backend so I can follow one + request end to end. How should I set that up? + tags: + capability: tracing-boundary + risk: incorrect-skill-activation + journey: route-distributed-tracing + expect_activation: false + graders: + - type: output-matches + config: + pattern: (OpenTelemetry|Activity|ActivitySource) + - type: prompt + rubric: + - Answers with a distributed tracing solution (spans, context propagation, an OTLP/Jaeger exporter) + - Does not answer a tracing question with a dependency-free in-process metrics recipe + - Does not claim that console-printed metrics give end-to-end request correlation + - Treats a dependency-free in-process metrics recipe as out of scope for this request + + - name: Cloud ingestion request stays out of scope + prompt: | + I want my .NET 11 app's telemetry to land in Application Insights so my team + gets hosted dashboards, retention, and alerting without running anything + ourselves. What should I use? + tags: + capability: cloud-ingestion-boundary + risk: incorrect-skill-activation + journey: route-hosted-telemetry + expect_activation: false + graders: + - type: output-matches + config: + pattern: (Application Insights|ApplicationInsights|Azure Monitor|OpenTelemetry) + - type: prompt + rubric: + - Recommends the hosted ingestion path (the vendor/Azure Monitor SDK or an OTLP exporter pointed at it) + - Does not propose printing measurements to the console as a substitute for hosted dashboards and alerting + - Addresses that the data must leave the process to reach the cloud service + - Treats a console-only, dependency-free metrics recipe as out of scope for this request + + - name: Log shipping request stays out of scope + prompt: | + I just want my .NET 11 app to ship its existing info/warn/error log lines to + a central place like Seq or Elasticsearch so I can search them. How do I do + that? + tags: + capability: log-shipping-boundary + risk: incorrect-skill-activation + journey: route-centralized-logging + expect_activation: false + graders: + - type: output-matches + config: + pattern: (ILogger|Serilog|Seq|Elasticsearch|logging) + - type: prompt + rubric: + - Answers with a logging pipeline (a logger plus a sink for the target system) + - Does not convert the request into numeric instruments, which would discard the log message text + - Keeps the searchable log lines intact rather than replacing them with aggregated values + - Treats a numeric metrics recipe as out of scope for a log-shipping request