Repository navigation
evolution: clarify E1 trial accounting and missing search-level DSR/PBO integration #225
Description
Problem
E1 candidate evaluation does not consume evolutionary trial counts or apply population-level DSR/PBO corrections. Persisted runs/candidates and the existing overfitting-risk classification are not equivalent to correction for repeated adaptive search.
This gap was identified by contributor JAYJAYTGG. An earlier maintainer explanation incorrectly implied the integration already existed.
Verified code path
Source inspection at revision 493175aa2554ea195fc60bedb2cfa1dd8590610b:
services/evolver/src/inalpha_evolver/evaluator/frozen.py:evaluate()callsevaluate_strategy_source()without evolutionary trial-count context.services/paper/src/inalpha_paper/strategy_evaluation.py: computes report, fitness and validation snapshot, without population-level DSR/PBO.services/paper/src/inalpha_paper/evaluation_metrics.py: time-splits an existing equity curve and computes segment metrics, decay and bootstrap diagnostics. These are not search-wide corrections; the segment named holdout is not automatically untouched by adaptive search.services/paper/src/inalpha_paper/engine/backtest.py: separate CV path callsdeflated_sharpe_ratio(n_strategies=n_paths); this counts CV paths, not evolutionary candidates.services/paper/src/inalpha_paper/engine/robustness.py: DSR/PBO utilities exist, but no E1 population-to-correction consumer was found.
There is no identified downstream logging function applying the missing E1 correction.
Contributor-reported reproduction
JAYJAYTGG reported a managed E1 test with three distinct candidates, all successfully reaching real evaluation. None of their snapshots contained CV, DSR, PBO or trial-count fields. Initial testing used deterministic mutation and synthetic/frozen data.
The contributor has now supplied the pinned runtime revision and preserved Experiment #1 records in this comment: Inalpha 3da8b103d61fc3c6af82ea479beade0cd7436e57, Testing Ground 4b6b86ec98e9f919bfa3c980022d3edd5e9b2f1a, the three-candidate trace, experiment notes, and successful execution/audit workflow links. The maintainer reviewed the linked records and workflow success metadata, but has not independently rerun the experiment. This runtime revision is distinct from the later source-inspection revision above. The synthetic 60-bar, deterministic-mutation evidence supports the tested E1 boundary; it does not establish strategy quality, the absence of correction elsewhere, or three independent statistical trials.
First implementation slice
Agree on a bounded accounting/evaluation contract before wiring a statistical utility:
- Define the search family: one run versus related runs reusing data/feedback. Distinguish generation, candidate, evaluation attempt and statistical trial.
- Define retry/resume, duplicate source/behavior, rejected-proposal and failed-evaluation accounting. Preserve attempts and reasons; operational recovery must not inflate statistical trials.
- Specify evidence needed for later correction: lineage, frozen data/config identity, selection history and aligned return series where required. Separate observed counts from model-based effective-trial estimates.
- Document uncorrected/unsupported status explicitly. Missing correction evidence must not become a passing significance result.
- Add deterministic regression coverage for the agreed evaluator/snapshot contract and applicable retry/deduplication rules.
A first PR can deliver focused documentation/contract and regression coverage without enabling a new statistical gate. Do not simply replace n_strategies with a slot count and claim adaptive search is corrected.
Acceptance criteria
- Document the E1 call chain and separate CV/DSR path accurately.
- Distinguish existing risk/validation diagnostics from search-level correction.
- Agree testable trial/search-family and retry/duplicate/failure accounting rules.
- Specify required evidence and missing/unsupported behavior for later statistical integration.
- Add deterministic regression coverage without requiring live LLM calls or claiming market performance.
- Explicitly avoid treating raw candidate count as a sufficient correction for correlated/adaptive search.
- Preserve owner scope, permissions and promotion/order boundaries.
Related work and boundaries
#177 covers E2 campaign-level evidence calibration; #178 covers E2 experiment/replication identity and repeated-campaign reporting. This E1 issue complements rather than duplicates that work. Neither is evidence that E1 DSR/PBO is already integrated.
No E2 redesign, live-trading enablement or stable-alpha claim. Review the statistical interpretation of the existing CV path-count input before reusing it; its presence does not establish evolutionary-search correction.
Activity
JJTGG commentedon Oct 15, 2026
Adding the existing Experiment #1 evidence here for reference. These are preserved records; I have not rerun the experiment for this comment.
Three-candidate E1 trace
The recorded run requested three candidate slots, generated three distinct source hashes, and all three candidates reached and completed the real E1 evaluation path. The snapshots contained no evolutionary trial-count or CV/DSR/PBO fields.
Separate trial-count audit
The audit recorded that the E1 evaluation API exposed no explicit evolutionary trial-count parameter, while the DSR utility itself accepted a strategy count and changed its expected-maximum-Sharpe calculation when that input changed.
Scope and limitations
These experiments used Inalpha revision "3da8b103d61fc3c6af82ea479beade0cd7436e57", deterministic mutation, and a controlled synthetic 60-bar dataset. They establish the behavior observed at the tested boundary, not strategy quality or the absence of multiple-testing correction elsewhere in Inalpha.
The later source-inspection revision recorded in this issue ("493175aa2554ea195fc60bedb2cfa1dd8590610b") is distinct from the pinned runtime revision above.
I agree that the next step is to establish the evaluation and trial-accounting contract before proposing statistical integration.
mirror29 commentedon Oct 16, 2026
Thanks for adding the preserved Experiment #1 evidence and clearly separating the runtime revision from the later source inspection.
I reviewed the linked trace, notes, and audit workflow, and confirmed that both linked Actions runs completed successfully. This fills the provenance gap noted in the issue. It is a review of your existing evidence, not an independent maintainer rerun.
The evidence supports the specific boundary described here: the tested E1 path evaluated three distinct candidate sources, but did not expose evolutionary trial accounting or search-level DSR/PBO in its evaluation snapshots. It does not imply that statistical utilities are absent elsewhere, or that three distinct source hashes represent three independent statistical trials.
I agree that the next step should be a small accounting/evaluation contract proposal. If you would like to take that forward, an issue comment or a focused docs PR would be a useful starting point, covering:
uncorrectedorunsupportedstate when evidence is insufficient.Concrete deterministic acceptance cases for those rules would help us agree on the contract before implementation. We should keep observed candidate counts separate from any estimate of effective trials; simply passing the candidate count as
n_strategieswould not establish a valid correction.No new live-LLM or paid runs are needed for this step, and it should leave owner isolation, approval, promotion, and execution behavior unchanged. Once the contract is agreed, we can scope the implementation and statistical integration separately.
Thanks again for documenting both the evidence and its limits so carefully.
JJTGG commentedon Oct 16, 2026
Proposed E1 accounting contract
Based on the existing experiment, here's how I think we could approach the accounting rules. These are starting points, so I'd appreciate your feedback before we settle on the implementation.
A new run ID shouldn't automatically mean a new independent search. If a run continues an earlier search or uses its results to guide further candidate generation or selection, that relationship should be recorded.
We should define what qualifies as a separate search before deciding how its trials are counted.
I'd keep candidate proposals, distinct candidate sources, evaluation attempts, and statistical trial counts separate.
Retries and recovery attempts should remain visible without automatically inflating the statistical trial count. Likewise, different source hashes don't necessarily mean statistically independent trials.
For DSR and PBO, we should agree on the relevant counting rules separately where their requirements differ.
We should preserve duplicate proposals, rejected candidates, failed evaluations, and their reasons rather than losing them from the search history.
Exact duplicates shouldn't automatically count as independent evidence. Failed evaluations should remain recorded, even when they cannot provide valid return data for a statistical method.
Runs that reuse data or feedback should also retain enough information to establish whether they belong to the same search family.
The contract should specify what we need to retain for later correction, including candidate lineage, data and configuration identity, selection history, and aligned return data where required.
Observed counts should remain separate from any estimated effective number of trials. If the required evidence is missing, the result should explicitly be marked "uncorrected" or "unsupported", rather than presented as statistically corrected.
I'd suggest using fixed data and a fixed seed to cover at least these cases:
The tests should verify the behavior at whichever boundary we agree on, whether that's candidate evaluation or the generation/search summary.
My preference is to settle these rules and the expected outputs first, then make the initial PR limited to the contract documentation and deterministic regression tests. We can scope statistical integration separately once we're confident the accounting is sound
mirror29 commentedon Oct 20, 2026
Thanks — this is aligned with the direction I had in mind. Let's use this as the basis for a focused contract PR, with statistical integration scoped separately.
A few details I'd like us to make explicit in that PR:
uncorrectedmean that no search-level correction has been applied, andunsupportedmean that a specified method cannot be applied because required evidence or assumptions are unmet. Include a reason, and leave unavailable effective-trial estimates unset.Your six acceptance cases are a good starting point. Please include a small example of the expected records or summary output for each, including a retry with changed evaluation inputs.
For the first PR, please distinguish tests that characterize today's E1 behavior from proposed contract cases that require future implementation. We should avoid tests that only validate fabricated fixtures and imply the runtime already implements the contract. If meaningful regression coverage requires production changes, call out that dependency so we can scope it explicitly.
No new paid or live-LLM runs are needed. Owner scope, permissions, promotion, and execution boundaries should remain unchanged. Thanks for turning the findings into a concrete proposal.