Add test strategy guidance: stay grounded, don't shy from integration tests - #46
Merged
Merged
Conversation
… tests The agent's third run succeeded but wrote a generic pytest logging test instead of engaging with the issue's Jubilant context. It used a generic Python logger instead of Jubilant's jubilant.wait logger, and wrote a unit test instead of an integration test, because that's what it could validate with run_tox. Add a 'Test strategy' section to both the composed prompt (probe_issue.py) and the agent definition (probe-issue.md): 1. Frame the agent's purpose as preparation for CI — CI is the ultimate test, run_tox is for increasing confidence in preparation, not for defining the test strategy. 2. 'Stay grounded in the issue's context' — don't abstract away to generic tests. If the issue is about Jubilant's logging, use Jubilant's logger. 3. 'Do not be shy about integration tests' — write them when the claim is about integration test behaviour, even though run_tox can't run them. Use run_tox to validate imports/types; let CI validate behaviour. 4. 'Choose test type based on the issue, not on what you can run' — the test type should match what the issue is about.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The agent's third run (PR #45) succeeded but wrote a generic pytest logging test instead of engaging with the issue's Jubilant context. It used a generic Python logger () instead of Jubilant's logger, and wrote a unit test instead of an integration test — because that's what it could fully validate with .
The issue (#42) is specifically about Jubilant integration tests, but the agent's test would pass or fail identically whether Jubilant is installed or not. It doesn't exercise the thing the issue is about.
Root cause
Two factors:
run_toxpassing as the success criterion, rather than treating CI as the arbiter. It optimized for "what can I verify myself" rather than "what's the most faithful test."Changes
Add a "Test strategy" section to both
probe_issue.py(composed prompt) andprobe-issue.md(agent definition):Frame the agent's purpose as preparation for CI — "CI is the ultimate test.
run_toxis a tool for increasing confidence that your preparation is sound. It is not the arbiter of whether your test is correct — CI is.""Stay grounded in the issue's context" — don't abstract away to generic tests. If the issue is about Jubilant's logging, use Jubilant's logger, not a generic one. Before writing the test, verify it exercises the thing the issue is actually about.
"Do not be shy about integration tests" — write them when the claim is about integration test behaviour, even though
run_toxcan't run them. Userun_toxto validate imports/types; let CI validate behaviour."Choose test type based on the issue, not on what you can run" — the test type should match what the issue is about.
Verification
uvx ruff format— passesuvx ruff check— passesuvx ty check— passes