[14H034160212]

Summary

Concrete implementation of the #226 review posted on #206 (comment), offered for discussion. Nothing here needs to merge before the protocol choices are agreed. Data changes are opt-in, and every default except invalid-run scoring keeps current behaviour. No paid calls were made.

Related: #206, #208.

Changes

Materials (opt-in; defaults reproduce the 2026-10-12 packet, τ = 3.79538%)

• --benchmark-dividends: benchmark cash dividends whose ex-date falls in (entry, exit] are added to the benchmark return.
• --tau-windows N: τ is estimated from the trailing N eligible windows instead of all of them.
• The protocol now records tau_calibration_windows, benchmark_return and the calibration-window climatology (UP/FLAT/DOWN base rates).

Scoring (report.py)

• Invalid runs are scored strictly: Brier 2.0, action 0. Under the v0.1 uniform fallback, an invalid run (0.667) scored better than every arm's mean on the development cases. Uniform and climatology fallbacks are kept as itt_sensitivity_brier. This is the only default behaviour change; development results are unaffected because no run was invalid.
• Uniform and climatology reference Brier, Brier skill vs climatology for every arm, and per-class reliability (mean predicted vs observed).
• B-mean: the mean of the completed B sample probabilities, with argmax action and neutral on ties. It reuses the same calls, so it is a scoring variant, not a separately budgeted arm. Reported with a paired B-mean − B interval, plus B's pairwise sample action disagreement.
• build_report(..., climatology=...); the CLI report command reads it from campaign.json.

Sampling (experiment.py, campaign.py)

• Optional sample_temperature applies only to B's independent samples. The default 0.2 is unchanged; 0.8 is the proposal for the formal arm definition. Every other call, including B's aggregation, stays at 0.2, and the temperature is recorded per call in the ledger sampling metadata.

Intervention controls (report.py, experiment.py)

• Deletion: the turn keeps its step slot for replay alignment but is omitted from every model-visible history, so there is no {"removed": true} marker.
• Conclusion flip: swaps p_up/p_down and the action, and keeps the original rationale verbatim. It no longer adds the "Intervention: …" text.
• Replacement control: digit-free, topic-unrelated prose from a versioned corpus (neutral-prose-v1), cut at a word boundary, under a neutral message key. It replaces the repeated filler and the unrelated_replacement key. Matching is still by characters, not provider tokens.

Validation

• tests/test_four_arm_pilot.py: 25 passed (5 new tests covering the dividend window and opt-in default, τ window and climatology, flip/replacement stimuli, B-only temperature, and references/skill/B-mean/strict ITT). The two existing tests were updated for strict ITT and the omitted-turn marker.
• Research suite: 266 passed, 3 skipped (dedicated Postgres not configured).
• ruff check is clean. mypy on pilot/ reports the same 3 errors as current main (artifacts.py, __main__.py), and no new ones.
• On real Yahoo data via the materials CLI:

Option	τ	Development labels
default	3.79538%	unchanged; reproduces the packet
--benchmark-dividends	3.79096%	no change
--tau-windows 252	3.01326%	2 of 10 move from FLAT (to UP and DOWN)

Open for agreement

• Whether to flip the dividend and τ-window defaults for formal materials.
• B sample temperature for the formal arm definition.
• Strict ITT as primary.
• Formal sample design (research: define and pilot compute-matched agent decision evaluation #206 §4).

Exact tokenizer-matched controls and blinded human process review remain open as before.

🤖 Generated with Claude Code

[mirror29]

Hi Qiming,

Thanks for the focused follow-up. I reviewed ef783287 against your review on #206, the requirements in #208, and the #226 discussion. No blocking execution defect found. Local verification: 266 Research tests passed, including 25 pilot tests; three dedicated-Postgres tests were skipped; Ruff passed. No paid calls were made.

The main changes listed in this PR are covered: reference forecasts and strict ITT, B-mean, B-only sampling temperature, opt-in material adjustments, and the contribution-control changes.

Before merging, please address two points:

1. Correct the ITT documentation. docs/usage/four-arm-pilot.md:132 still describes uniform probabilities as the primary treatment for invalid runs. Please align it with strict ITT, keeping uniform/climatology as sensitivity analyses.

2. Explicitly record the remaining scope from your review:

   • §2b: per-case token usage, IQR, and ratios to an agreed budget target. The current report only checks per-arm medians.
   • §3d: formal intervention selection and per-intervention eligible/attempted/completed/skipped counts, with clearly defined skip-rate denominators. The current runner uses the first N cases and retains individual skip reasons.

These can be linked follow-ups; they do not need to expand this PR into the full formal experiment.

Next, please propose a short protocol revision in #206 covering the primary metric and ITT rule, dividend convention, calibration window, B temperature, budget design, and formal sample/intervention selection. Once the reporting definitions are agreed, the remaining reporting work and regression tests can proceed offline.

Existing saved outputs can also be rescored into a separate report, preserving the originals. However, the previous paid pilot does not validate the new sampling temperature or intervention stimuli; those require a separately agreed configuration and budget.

I will continue coordinating #208’s supporting data and provenance work. Exact token matching, blinded human review, formal sample separation, and model revision/cutoff limitations remain open. Merging this development improvement does not freeze the formal protocol or close #206/#208.

[14H034160212]

Thanks for the review. Addressed in 60480bc:

1. four-arm-pilot.md now states strict ITT (Brier 2.0, action 0) as the primary rule, with uniform and climatology fallbacks as sensitivity analyses only.
2. §2b (budget dispersion) and §3d (formal intervention selection and per-intervention counts) are recorded in the doc as not yet covered, and are linked to the protocol revision on research: define and pilot compute-matched agent decision evaluation #206, where they are items 6 and 8.

No code changes since ef78328; tests unchanged (25 pilot / 266 suite).