research: define and pilot compute-matched agent decision evaluation #206

Activity

1. mirror29 commented on Oct 10, 2026

   

   mirror29

   on Oct 10, 2026

   OwnerAuthor

   More actions

   Supporting data and recording gaps are now tracked in #208, coordinated by the maintainer. This does not block starting protocol/schema design, the four-arm runner, usage instrumentation or intervention scaffolding with development fixtures.

   We will agree the minimal pilot input/logging format together. Benchmark-aligned labels and frozen point-in-time evidence are required before interpreting the real-data pilot, and the readiness checks in #208 must be resolved before formal evaluation. No additional ownership is assumed for Qiming beyond the contribution scope already discussed.
2. 14H034160212 commented on Oct 10, 2026

   

   14H034160212

   on Oct 10, 2026

   Contributor

   More actions

   Protocol v0.1 — clarifications (decision semantics, budget exhaustion, intervention 3)

   Proposed for the price-only TSLA/SPY development pilot; all values frozen before formal evaluation and open to revision until then.

   1. Decision, probability and scoring semantics

   Timing

   • Calendar: NYSE sessions. Decision time t = after the close of session t; only bars whose session has completed are model-visible.
   • Entry at the open of t+1, exit at the open of t+1+H. Primary H = 5 sessions (secondary H = 20 in formal evaluation only).

   Outcome

   • Excess return r = R_TSLA − R_SPY over the same open-to-open window (simple excess, no beta adjustment; limitation noted).

   Labels

   • UP if r > +τ, DOWN if r < −τ, FLAT otherwise.
   • τ = 0.5 × σ(5-day excess return), estimated on a window strictly before the first development case. Freeze it; never re-estimate on evaluation data.
   • Labels are generated by versioned deterministic code (feat(usage): unify LLM accounting and prepare research pilot evidence #208 item 5).

   Model output

   • Probability vector (p_up, p_flat, p_down) summing to 1, plus action s ∈ {+1, 0, −1} and cited evidence IDs.
   • If the action is inconsistent with argmax(p), log it as a constraint violation and score the action as stated.

   Metrics

   • Primary: multiclass Brier Σ_k (p_k − y_k)².
   • Secondary:
     • hit rate (neutral is correct iff the label is FLAT);
     • coverage = share of non-neutral actions, reported next to hit rate so always-neutral cannot look good;
     • net excess P&L.

   P&L and costs

   • Position P&L = s · r − c · |s|, with c = 20 bp round trip (10 bp per side).
   • Sensitivity at 0 and 50 bp round trip.
   • Neutral = flat = 0.
   • Short = −r; borrow cost is ignored in the pilot (documented limitation).

   Inference

   • Paired comparisons B/C/D vs A and D vs B/C on identical cases.
   • Moving-block bootstrap with block length ≥ H to account for overlapping outcome windows.

   2. Budget enforcement and incomplete runs

   Budget accounting

   • Per case per arm: total budget B = input + output tokens, including reasoning tokens where reported.
   • Provider-reported usage is authoritative; retries count against the budget.

   Final-answer reserve

   • B_final = 0.15·B is reserved for the final aggregation/synthesis call.
   • The working phase may spend at most B − B_final.

   Pre-call guard

   • Before each working call, compute tokenized_input + max_tokens. If this exceeds the remaining working budget, skip the call and go to the final stage.
   • Because max_tokens is set from the remaining budget, no call can overrun by construction.

   Early stop per arm

   • B aggregates the completed samples.
   • C uses the last complete revision.
   • D synthesizes from the completed turns.
   • The number of completed units is logged.

   Final-call failure

   • One retry is allowed within the reserve.
   • If the output is still truncated, unparseable or invalid, the case is marked INVALID with a reason code.

   Statistics

   • No case is ever dropped.
   • Primary (intention-to-treat): INVALID = action 0 and probabilities (1/3, 1/3, 1/3).
   • Also reported: invalid rate per arm, and a complete-case analysis on cases valid in all four arms.

   Parameter selection

   • k, r, N × R are chosen on development cases so that median usage ≈ B (tolerance ±10%), then frozen.
   • A's actual usage defines the 1× reference.

   3. Interventions — replay semantics

   Evidence IDs

   • Every item in a frozen evidence bundle has a stable ID and hash.
   • For the price-only pilot, an item is one derived feature block (e.g. 5/20-day returns, volatility, volume vs average, distance from 52-week range).

   Intervention 3 — remove claimed key evidence

   • Selection rule: the first evidence ID cited in the final output of the unmodified run. If nothing is cited, the case is excluded from this intervention with reason NO_CITED_EVIDENCE.
   • Intervention location: the input bundle itself.
   • Replay boundary: round 0. All agent turns and the synthesis are regenerated from scratch, so no retained message can reintroduce the removed item.
   • Placebo control: remove one uncited item of similar token length (closest length; ties broken by fixed seed).
   • Re-entry check: scan all regenerated messages for the removed item's ID and its distinctive numeric values. Report the re-entry rate rather than hiding it, because other features may still let the model infer it.
   • Optional: the same intervention can be applied to A/B/C (all arms cite evidence), which allows comparing key-evidence sensitivity across arms.

   Interventions 1 and 2 — explicit boundary

   • Modify agent a's message at round j.
   • Keep all earlier turns verbatim.
   • Regenerate all later turns and the synthesis (downstream replay).
   • Selection rule for a and j is fixed in advance: the analyst whose stance the final decision most resembles, last discussion round.

   Noise baseline (all interventions)

   • Rerun the unmodified configuration with a different seed using the same replay boundary.
   • Every intervention flip rate is reported next to this baseline, because deepseek-chat is not deterministic even at temperature 0.
   • An intervention effect is the paired difference against the noise baseline, not the raw flip rate.

   Reporting

   • Paired decision-flip rate, change in Brier, and change in the probability of the originally chosen class.
   • All results are interpreted as input-sensitivity, not causal attribution.

   Logging requirements for #208

   Per call:

   • experiment / case / arm / run / replay IDs;
   • stage, attempt;
   • provider, model and settings;
   • input/output/reasoning tokens;
   • latency, status, reason code.

   Per case:

   • bundle ID + hash;
   • cited evidence IDs;
   • the final structured output.
3. 14H034160212 commented on Oct 10, 2026

   

   14H034160212

   on Oct 10, 2026

   Contributor

   More actions

   Follow-up after reviewing the 2026-10-10 materials package

   Thanks for the inventory — clear about what exists versus what is only in the schema. Three items affect the v0.1 protocol; none changes its structure.

   1. Benchmark data is the first blocking gap

   • v0.1 scores TSLA excess return over SPY. Stored ^GSPC has only 3 rows (2026-07-02 to 07-07), and SPY is absent.
   • Request for feat(usage): unify LLM accounting and prepare research pilot evidence #208 item 3: SPY daily OHLCV covering the full stored TSLA range (2024-05-07 to 2026-07-07), from the same vendor and with the same adjustment convention as TSLA, with provenance and hashes.
   • The labels in v0.1 are open-to-open, so both series need opens as well as closes.

   2. Arm D must be adapted from the production research flow

   • Output: briefs[].confidence and the final confidence are self-reported scalars, not class probabilities. In the experiment, the synthesis step must emit (p_up, p_flat, p_down), an action and cited evidence IDs (v0.1 §1). Analyst briefs keep their native fields for logging.
   • Debate control: debate_trigger / debate_stop_reason can skip or shorten discussion in production. For the experiment, D runs a fixed N × R with these switches disabled, so the arm definition does not vary per case.
   • Usage: because complete_json drops usage, I will add a wrapper in the experimental runner that records per-call usage, latency, attempt and status (feat(usage): unify LLM accounting and prepare research pilot evidence #208 item 2). Production code does not need to change for the pilot.

   3. Contamination handling for the price-only pilot

   • The provider aliases differ: deepseek-chat in the research service, deepseek-flash in the E1 runs. Neither has a verified cutoff, and stored TSLA data ends 2026-07-07, so no case can be claimed to be post-cutoff.
   • Proposal: model-visible evidence contains only relative features — returns, realized volatility, volume ratio, position within the 52-week range. It contains no ticker, no calendar dates and no absolute price levels.
   • This reduces, but does not remove, the ability to recognize the episode. It will be documented as a limitation, not presented as a fix.
   • I will also record which model alias each run used, and pin a single alias for all four arms.

   Development sample

   Using the full stored TSLA series for lookback, the 41-row sample yields roughly 30 decision points at H = 5, enough for the 10–20-case machinery pilot. These dates will be excluded from the formal evaluation set.

   Question

   Hourly BTC/ETH data runs to 2026-10-10 and is the only stored series that could plausibly fall after a model cutoff. Would a crypto replication arm be useful later as a contamination check? It would use a bar-count horizon (e.g. 24 × 1h bars) in place of trading days. I would not add it to the first pilot.
4. mirror29

   closed this as completedin #226on Oct 12, 2026
5. mirror29

   reopened this on Oct 12, 2026
6. mirror29 commented on Oct 12, 2026

   

   mirror29

   on Oct 12, 2026

   Last edited by mirror29

   OwnerAuthor

   More actions

   Hi Qiming,

   Thank you for the v0.1 clarifications and the follow-up after reviewing the materials. I have now prepared the missing benchmark data and a downloadable development snapshot:

   TSLA/SPY development materials

   Read-only access is enabled for your existing contact account. The packet includes:

   • 542 aligned TSLA/SPY daily sessions, covering 2024-05-07–2026-07-07, with open prices and the same vendor/adjustment convention;
   • ten development cases, relative-feature inputs, separate outcome labels and provenance;
   • a README, checksums and a deterministic development rubric, which is not a substitute for blinded human annotations.

   This is the snapshot used for the reference runner's development acceptance. It replaces the earlier 41-case packet for that runner; the two packages have different provisional thresholds and are not interchangeable. The current tau is approximately 3.79538%. The estimation-window choice remains open for agreement, and the returns exclude SPY dividend cashflows.

   Following up on your implementation points

   PR #226 is now merged into main. It provides a reference four-arm adapter with three-class probabilities, evidence IDs, configured discussion rounds, budget-stop recording and per-call unified usage capture. I hope this gives us a useful starting point and saves some integration work.

   Your relative-feature-only approach is also reflected in the model-input projection. Dates, tickers and absolute prices remain outside that projection. Historical publication timing is still a stated development assumption, not verified archival availability, and anonymization does not establish absence of training contamination.

   The development acceptance used deepseek-flash to exercise the implementation. This is not a decision about the model for our formal protocol; immutable revision and training cutoff remain unknown.

   On BTC/ETH, I agree with leaving it out of the first pilot. It could be useful for later cross-market replication, although without verified cutoff evidence I would not describe it as a confirmed post-cutoff contamination control.

   Open points for discussion

   The current implementation still uses conservative token estimates rather than an exact provider tokenizer, and intervention length controls are not yet precisely token-matched. Human process review and independent formal cases also remain open. The ten-case run is development validation, not evidence of multi-agent superiority.

   Implementation guide

   I would be interested in your view on where this baseline matches v0.1 and where it should change, particularly around budget handling, intervention controls and scoring. Your protocol perspective would be valuable here; the implementation choices are not intended to pre-empt that discussion.

   I will continue coordinating the supporting data and materials under #208. No new paid run is assumed by this handoff; we can agree the run budget separately.
7. 14H034160212 commented on Oct 15, 2026

   

   14H034160212

   on Oct 15, 2026

   Contributor

   More actions

   Review of the #226 baseline against v0.1

   Thanks — the runner, ledger reconciliation and fail-closed accounting are more careful than v0.1 specified. I reviewed pilot/experiment.py, campaign.py, report.py, materials.py, four-arm-pilot.md and the 2026-10-12 readiness receipt; I have not re-run any model calls. Proposals below are ordered by how much they could change conclusions.

   0. Threshold and benchmark (your two open questions)

   I reproduced the packet's τ = 3.79538% exactly from Yahoo data with materials.py logic, so we are on the same series.

   τ window.
   The 491 pre-development windows span a volatility regime shift:

   • first half → τ = 4.44%;
   • second half → τ = 3.01%;
   • last 252 windows → τ = 3.01%.

   With 3.80%, FLAT is 49% of calibration windows.

   Proposal for formal:

   • τ is estimated per ticker from the 252 windows immediately preceding the formal period;
   • it is frozen once and never re-estimated per case.

   The development τ can stay as is for machinery testing.

   SPY dividends.

   • 40 of 491 calibration windows contain an SPY ex-date. The price-only benchmark overstates excess return by about 0.30% in those windows, and 2 calibration labels change. τ is essentially unchanged (3.791% vs 3.795%).
   • The bias is small but always in the asset's favour, and dividends are already retained via actions=True.

   Proposal: add the cash dividend to the benchmark return when the ex-date falls in (entry, exit]. TSLA has no dividends or splits in range.

   1. Scoring

   1a. Add reference forecasts.
   On the ten development cases:

   • Uniform (1/3, 1/3, 1/3) scores 0.667 and beats every arm (A .811, B .774, C .689, D .727).
   • Climatology (pre-development base rates: FLAT 49%, UP 26%, DOWN 24%) scores 0.719 on these cases, which are UP-heavy. C and D beat it; A and B do not.

   Without references, "D beats A" could mean "D is less overconfident".

   Proposal:

   • Report both references.
   • Use the Brier skill score vs climatology as the primary comparison.
   • Add a per-class reliability breakdown (mean predicted vs observed frequency).

   1b. The v0.1 ITT fallback rewards failure — my error, should be changed.
   The uniform fallback (0.667) beats every arm's actual mean, so an invalid run currently scores better than a valid one.

   Proposal:

   • Primary ITT = strict penalty: invalid → Brier 2.0, action 0.
   • Sensitivity = climatological fallback.
   • Keep the invalid rate per arm and the complete-case analysis, as reported now.

   Invalid count was 0 in development, so no current result changes.

   1c. B aggregation.
   v0.1 described B as vote/averaging; #226 uses an LLM aggregation call. Both are reasonable, but they are different arms.

   Proposal:

   • Keep the LLM aggregation as B.
   • Also report B-mean: the arithmetic mean of the completed sample probability vectors, action = argmax. This uses the same calls and costs nothing extra.
   • If B-mean ≈ B, the aggregation call adds nothing.

   2. Sampling and budget

   2a. Temperature.
   All calls use 0.2. Self-consistency relies on sample diversity, so 11 near-identical samples understate B.

   Proposal:

   • Preregister B sampling temperature = 0.8 as part of the arm definition.
   • Keep 0.2 elsewhere.
   • Report pairwise sample disagreement for B.

   2b. Per-case matching.
   Medians match within 10%, but the 24k cap never binds, so per-case usage can still differ systematically by arm.

   Proposal:

   • Report per-case usage IQR and the per-case ratio to the arm-mean target.
   • For the formal run, either keep fixed units and accept median matching (stated as such), or set arm_tokens = 1.1 × target so the cap binds and report the truncation rate per arm.

   I lean to fixed units plus reported dispersion, because a binding cap makes the number of discussion rounds vary across cases.

   3. Intervention controls

   3a. Flip conclusion currently also changes the evidence argument and announces itself.
   The rationale is replaced with "Intervention: the analyst's stated conclusion is reversed.". Downstream turns therefore see the word "Intervention" and lose the original argument.

   Proposal:

   • Keep the original rationale text verbatim.
   • Swap only p_up/p_down and action.
   • Add no meta text.

   This is the "evidence fixed, conclusion flipped" condition. The resulting rationale/conclusion inconsistency is part of the stimulus and should be noted as such.

   3b. Deletion marker.
   {"removed": true} tells later turns that something was taken out.

   Proposal: omit the entry entirely as the primary deletion. Optionally keep the marker as a secondary variant.

   3c. Replacement control.
   Repeated "A fictional room has plain walls." is trivially recognizable as filler.

   Proposal: use natural, topic-unrelated prose from a fixed, versioned corpus, matched by provider-reported prompt tokens. Measure this with one counting call per candidate, or with the DeepSeek tokenizer if it is available offline.

   3d. Coverage.
   --intervention-cases defaults to 1. Formally, preregister either all cases or a seeded random subset, and report the per-intervention skip rate. For example, the evidence placebo is skipped whenever every block is cited; NO_UNCITED_EVIDENCE has already occurred once.

   The zero flips across 6 replays are consistent with real robustness, with low temperature, or with the controls above. We can't distinguish these yet.

   4. Formal sample

   The ten development cases are consecutive with H = 5, so they span roughly two independent outcome windows. With TSLA alone, a stride of 5 sessions after the 252-session lookback gives about 55 non-overlapping cases, short of the 150–300 target.

   Proposal:

   • Use 5–8 liquid US large caps, all against SPY, with per-ticker τ as in §0.
   • Decision dates use a stride of 5 sessions.
   • Development dates are excluded.
   • Bootstrap resamples date blocks, so that same-date cases across tickers move together.

   Ticker choice can be made by a rule frozen in advance, for example the largest by average dollar volume at the start of the window.

   Proposed next step

   If you agree with 0–3, I can open a PR against pilot/ with:

   • dividend-adjusted benchmark returns;
   • reference forecasts and BSS;
   • the strict ITT fallback;
   • B-mean;
   • B temperature as a config field;
   • the three intervention-control fixes;
   • regression tests.

   No paid calls; the re-run would happen after you approve a budget. Section 4 can be settled in this thread before the formal freeze.
8. 14H034160212

   added a commit that references this issue on Oct 16, 2026

   docs(research): state strict ITT as primary and record uncovered revi…

   ...

   60480bc
9. 14H034160212 commented on Oct 17, 2026

   

   14H034160212

   on Oct 17, 2026

   Contributor

   More actions

   Protocol revision v0.2 (proposal for item-by-item confirmation)

   Follows the #248 review. Each item gives the proposed rule; please confirm, amend or reject per item. Nothing here is frozen until we agree.

   #	Item	Proposed rule
   1	Primary metric	Multiclass Brier skill score vs climatology. Climatology = UP/FLAT/DOWN base rates of the τ calibration windows. Primary contrasts: D−A, D−B, D−C on the paired per-case Brier difference. Secondary: B-mean (incl. B-mean − B), hit/coverage, net excess P&L at 20 bp (0/50 bp sensitivity), reliability.
   2	ITT rule	Strict: an invalid run is scored at Brier 2.0, action 0, and stays in all paired analyses. Sensitivity only: uniform fallback, climatology fallback, complete-case. Invalid rate reported per arm.
   3	Dividends	Benchmark total return: SPY cash dividends with ex-date in (entry, exit] are added (--benchmark-dividends). Price-only is kept as a sensitivity check.
   4	Calibration window	Per ticker, τ = 0.5 × SD over the 252 eligible windows whose exit precedes the first formal decision. Frozen once, never re-estimated per case.
   5	B temperature	0.8 for B's independent samples; 0.2 for every other call, including B's aggregation.
   6	Budget design	Fixed units (k/r/N×R) calibrated on development cases only, so that each arm's median input+output tokens is within ±10% of a preregistered target T. The cap stays a guard at 1.1·T, not a binding limit. Reported: per-case usage, IQR, and ratio to T per arm (§2b). Acceptance: median within ±10% of T and ≥80% of cases within ±25%.
   7	Formal sample	Tickers: the 6 largest S&P 500 constituents by average daily dollar volume over the 252 sessions before the formal start, all against SPY. Decision dates: stride of 5 sessions. Development dates and any window overlapping them are excluded. Bootstrap resamples date blocks, so same-date cases move together.
   8	Intervention selection	Seeded random subset of 40 formal cases (seed preregistered), drawn from cases whose D base run completed. Per intervention, report eligible / skipped / attempted / completed. Eligible = selected cases with a completed D base run; skipped = eligible cases where that intervention is inapplicable (e.g. NEUTRAL_CONCLUSION, NO_UNCITED_EVIDENCE); skip rate = skipped / eligible (§3d).

   Data requirement this creates (#208)

   Item 7 needs feature lookback (252 sessions), τ calibration (252 windows) and the formal period itself. The current packet starts 2024-05-07, which leaves essentially no room for formal decisions before the development window.

   Request: extend all formal tickers and SPY back to about 2023-05, under the same vendor and adjustment conventions. With formal decisions then spanning roughly 2025-05 to 2026-04, a stride of 5 gives about 46 dates × 6 tickers ≈ 275 cases, inside the 150–300 target.

   Rough cost (for the budget decision, not an authorization)

   • Main runs: ≈275 cases × (A + 3 × T ≈ 9.4k) ≈ 8M tokens.
   • Interventions: 40 × 6 replays × 9.4k ≈ 2.3M tokens.
   • At the current deepseek-flash estimate (input ≈ $0.3/M, output ≈ $1.2/M), this is on the order of $5.

   A short development re-run validating item 5 and the new intervention stimuli would come first, at a separately agreed budget.

   Still open (unchanged)

   • Exact tokenizer matching.
   • Blinded human process review.
   • Model revision and training cutoff.
   • These dates all precede any plausible cutoff, so the contamination limitation stays explicit.

   Once items 1–8 are agreed, I'll implement the §2b and §3d reporting, plus a rescoring command for existing outputs that writes a separate report and leaves originals untouched, as offline follow-ups with regression tests.
10. mirror29

    added a commit that references this issue on Oct 20, 2026

    feat(research): apply #206 review to the four-arm pilot (#248)

    ...

    c5dcb00
11. mirror29 commented on Oct 20, 2026

    

    mirror29

    on Oct 20, 2026

    OwnerAuthor

    More actions

    Hi Qiming,

    Thanks for the v0.2 proposal and the documentation follow-up. I have merged #248 after all 13 CI checks passed. This delivers the development improvements; it does not freeze the formal protocol or authorize new paid runs.

    I agree in principle with climatology-referenced scoring, strict ITT, frozen per-ticker calibration, and B sampling at 0.8 with other calls at 0.2. Before freezing v0.2, please refine these points:

    1. Budget design. A cap of 1.1 × T can truncate fixed-unit runs, especially with the current conservative input reservation. Please distinguish the matching target from the safety cap, and report budget stops, completed units and invalid rates. We should preregister how to handle missed matching thresholds without dropping cases or retuning after inspecting formal results.

    2. Data coverage and sample isolation. The feature lookback and calibration windows can overlap in the current implementation, so they do not automatically require two consecutive years of warm-up. Please derive the required dates and final case count explicitly. Parts of the proposed formal period were also used in development threshold/base-rate analysis; we should document that exposure and agree whether to use a retrospective held-out evaluation or select a stricter independent period.

    3. Multi-ticker conventions. Use S&P 500 membership known at the formal start, with explicit share-class and missing-data rules. Apply consistent dividend and corporate-action treatment to both stocks and SPY. The formal adapter will also need per-ticker climatology, a defined pooled BSS calculation, and date-block resampling that keeps same-date tickers together. Please specify the primary contrast direction, bootstrap block length and treatment of the three primary comparisons.

    4. Intervention accounting. Sampling 40 cases from valid D base runs is reasonable, provided conclusions are explicitly conditional on D validity. Please report the full pool, valid D pool, selected cases, inapplicable skips, failed/invalid replays, and valid intervention–noise pairs. Also define what happens if fewer than 40 cases qualify.

    Please continue the offline work in parallel; you do not need to wait for #208 to be completed. The protocol revision, budget/intervention reporting, separate rescoring command, and multi-ticker scaffolding can proceed with existing development materials or synthetic fixtures and regression tests. Rescoring should preserve the original outputs.

    I will continue coordinating supporting data and provenance under #208. If a specific step needs additional inputs, please identify the minimal fields and coverage required so I can supply them alongside your implementation.

    A small development re-run can use the existing materials once we agree its configuration and separate budget. Formal evaluation will require the relevant data-readiness gates and protocol freeze. The formal cost estimate should be updated once case counts, replay/control counts and retry allowances are fixed.

    Exact token matching, blinded human process review, and model revision/training-cutoff limitations remain open, but they need not block the offline steps above.

    Thanks again for moving this forward carefully.

    Best,
    Miro