Add fixture-based regression tests for CDM query layer (no live OpenSearch required)

[atheurer]

Problem

CDM has no automated tests today. Changes to query logic, schema, or metric
formatting can silently break results. Testing CDM is harder than testing tool
post-processors because it has an external dependency: a running OpenSearch
instance. A test suite that requires OpenSearch is expensive to run in CI
(~30s container startup, memory overhead) and impractical for developers
without a local cluster.

Proposed approach: fixture-based record/replay for queries

The key insight is that CDM's query path is a pure transformation:
OpenSearch HTTP responses → structured metric data. If we record real
OpenSearch responses once and commit them as fixtures, the query logic can
be tested anywhere with no OpenSearch running.

Two tiers of testing

Tier 1 — fixture-based unit tests (no OpenSearch, fast, in CI)

1. Run CDM queries against a real OpenSearch once in "record" mode,
   capturing the raw HTTP request/response pairs as JSON fixture files.
2. Commit fixtures to the repo alongside tests.
3. In CI, mock the HTTP layer (e.g. responses or vcrpy library) to
   replay the recorded responses.
4. Assert the CDM query output matches expected values.

This tests: query construction, response parsing, metric formatting,
pagination handling, error handling.

Does NOT test: whether OpenSearch actually accepts and stores documents
correctly, or whether the indexed schema is valid.

Tier 2 — integration tests (requires OpenSearch, run on demand)

For the indexing path and end-to-end round-trips, a real OpenSearch is
unavoidable. These tests should be runnable via a separate make test-integration
target, using either a Docker container (testcontainers) or an existing
instance pointed to via environment variable.

This keeps CI fast while still having full-fidelity coverage available.

Fixture directory structure

test/
  fixtures/
    opensearch-responses/
      metric-data-basic.json          # recorded GET /cdm-*/metric-data response
      metric-data-by-run.json         # filtered query response
      iterations.json                 # iteration listing response
      ...
  test_query_layer.py                 # unit tests using fixtures
  test_integration.py                 # integration tests (requires OpenSearch)
  record-fixtures.sh                  # run against live OpenSearch to refresh fixtures
  conftest.py                         # pytest fixtures, mock setup

record-fixtures.sh pattern

#!/bin/bash
# Run against a real OpenSearch to record HTTP responses
# Usage: OPENSEARCH_URL=http://localhost:9200 ./test/record-fixtures.sh
export VCR_MODE=record
pytest test/test_query_layer.py --opensearch-url "$OPENSEARCH_URL"
# Fixtures written to test/fixtures/opensearch-responses/

In normal test runs, VCR_MODE is unset and the fixture files are replayed.

What to test

Query layer (Tier 1, fixture-based)

• get_metric_data returns correct metric names, values, timestamps
• Breakout/aggregation parameters are correctly translated to OpenSearch queries
• Pagination is handled correctly for large result sets
• Period filtering works correctly
• Error responses from OpenSearch are handled gracefully

Indexing path (Tier 2, integration)

• Metric documents indexed by CDMMetrics can be queried back correctly
• Schema version compatibility
• Round-trip: log_sample() → index → query → verify values

Minimal infrastructure requirement

Tier 1 tests require only: Python, pytest, responses or vcrpy, and the
CDM + toolbox source. No OpenSearch, no crucible install, no Docker.

Tier 2 tests require only: an OpenSearch instance (URL via env var).
They do not require a full crucible stack.

Dependencies

• Requires toolbox#127 for shared output comparison utilities (optional but
  useful for comparing metric data output between old and new query logic).
• The fixture format should be compatible with the OpenSearch Python client
  (opensearch-py) response structure so mocking is straightforward.

Related context

This discussion grew out of adding golden-file tests for tool post-processors
(toolbox#127, tool-procstat#37, etc.). CDM poses a harder testing problem
because the data lives in OpenSearch rather than in files, but the record/replay
approach allows the same "committed fixtures = no live service needed in CI"
property.