Skip to content

Add test strategy guidance: stay grounded, don't shy from integration tests - #46

Merged
dwilding merged 1 commit into
mainfrom
agent-test-strategy
Aug 25, 2026
Merged

dwilding merged 1 commit into
mainfrom
agent-test-strategy

Conversation

@dwilding

Copy link
Copy Markdown
Owner

Problem

The agent's third run (PR #45) succeeded but wrote a generic pytest logging test instead of engaging with the issue's Jubilant context. It used a generic Python logger () instead of Jubilant's logger, and wrote a unit test instead of an integration test — because that's what it could fully validate with .

The issue (#42) is specifically about Jubilant integration tests, but the agent's test would pass or fail identically whether Jubilant is installed or not. It doesn't exercise the thing the issue is about.

Root cause

Two factors:

  1. The agent treated run_tox passing as the success criterion, rather than treating CI as the arbiter. It optimized for "what can I verify myself" rather than "what's the most faithful test."
  2. The prompt didn't explicitly tell the agent to stay grounded in the issue's specific context or to write integration tests when the claim is about integration test behaviour.

Changes

Add a "Test strategy" section to both probe_issue.py (composed prompt) and probe-issue.md (agent definition):

  1. Frame the agent's purpose as preparation for CI — "CI is the ultimate test. run_tox is a tool for increasing confidence that your preparation is sound. It is not the arbiter of whether your test is correct — CI is."

  2. "Stay grounded in the issue's context" — don't abstract away to generic tests. If the issue is about Jubilant's logging, use Jubilant's logger, not a generic one. Before writing the test, verify it exercises the thing the issue is actually about.

  3. "Do not be shy about integration tests" — write them when the claim is about integration test behaviour, even though run_tox can't run them. Use run_tox to validate imports/types; let CI validate behaviour.

  4. "Choose test type based on the issue, not on what you can run" — the test type should match what the issue is about.

Verification

  • uvx ruff format — passes
  • uvx ruff check — passes
  • uvx ty check — passes

… tests

The agent's third run succeeded but wrote a generic pytest logging test
instead of engaging with the issue's Jubilant context. It used a generic
Python logger instead of Jubilant's jubilant.wait logger, and wrote a unit
test instead of an integration test, because that's what it could validate
with run_tox.

Add a 'Test strategy' section to both the composed prompt (probe_issue.py)
and the agent definition (probe-issue.md):

1. Frame the agent's purpose as preparation for CI — CI is the ultimate
   test, run_tox is for increasing confidence in preparation, not for
   defining the test strategy.

2. 'Stay grounded in the issue's context' — don't abstract away to generic
   tests. If the issue is about Jubilant's logging, use Jubilant's logger.

3. 'Do not be shy about integration tests' — write them when the claim is
   about integration test behaviour, even though run_tox can't run them.
   Use run_tox to validate imports/types; let CI validate behaviour.

4. 'Choose test type based on the issue, not on what you can run' — the
   test type should match what the issue is about.
@dwilding
dwilding merged commit 07369e8 into main Aug 25, 2026
2 checks passed
@dwilding
dwilding deleted the agent-test-strategy branch August 25, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant