Repository navigation
Implement source-bound RRSI candidate and evaluation contracts - #26
Conversation
…x/fable-rrsi-source-20260930
|
P2: Nonterminal retry history can be reported as cost-comparable Reviewed head: In Expected gate: unresolved/nonterminal attempts must block cost comparability until their terminal outcomes and consumption are reconciled. Preserve all retry totals and the valid terminal failed-attempt followed by success control. Reproduction: save this as a temporary Python file and run import runpy
f = runpy.run_path("test/rrsi_evaluator_test.py")
e = f["e"]
config = f["config"]()
slot = f["slot"](config)
observed = {}
for status, terminal in (("failed", True), ("unknown", False), ("timeout", False)):
rows = [f["observation"](slot, status=status, terminal=terminal, result=None),
f["observation"](slot, attempt=1)]
result = e.aggregate_results(observations=rows, expected_slots=[slot],
config=config, synthetic=True)
observed[status] = result["comparable_synthetic"]
print(status, terminal, observed[status], result["total_cost_microusd"])
assert observed["failed"] is True
assert not any(observed[s] for s in ("unknown", "timeout")), observedThe existing evaluator suite passes 24/24 on this exact head. This is an offline synthetic-contract finding: |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 128a0eb373
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
jmborden1112-beep
left a comment
There was a problem hiding this comment.
LGTM
Needs Fixing
None. RRSI source identity, bounded synthetic evidence and explicit pairing contracts remain consistent. The apparent single-arm aggregation concern was refuted: aggregation is per arm and match_arms handles pairing separately. No live optimization/promotion is implemented.
Requires Human Review
None beyond the repository approval gate fulfilled under the owner's second-account authorization. This is agent source review, not independent human sign-off.
Recommended Optional
None.
Create Follow-up Issue
None.
Workflow Trace: exact final head6e65019db917c52ee4c7f9fe36c0a02488215d3f includes merged25,22,23. Source lens /root/fable_stack_review plus root verification, gpt-6-luna medium fallback. Final combined local npm suite211passed/3platformskips/0failed; both package validators and diff check pass. Earlier RRSI source30/30 and evaluator28/28 Python tests passed. All eight final-head hosted checks pass, including Windows Node24, packed artifact and Release gate. No paid model trials, provider qualification, source improvement claims, publication, or runtime activation.
Implements the source-only Fable RRSI candidate and evaluation path for issue #24. A reviewed Fable commit is exported into a disposable repository with exact Git-blob provenance; only two finite-vocabulary prompt fragments can change. Candidate admission binds actual trees, modes, worktree bytes and independently tagged line edits while preserving the six workflows, ECF/evidence contracts and model/worker configuration.
The evaluator freezes source excerpts, factual assertions, uniform weights, split/model/runtime identities and retry/cost policy. It grades typed facts with exact evidence citations, retains missing slots and all reported retry consumption, and rejects unsafe, unknown-cost or incomplete comparisons. Held-out/OOD packets remain sealed. Live imports, acceptance and promotion stay disabled; synthetic fixtures do not establish improved Fable performance.
The repaired head is
3829e76c0072e140a4a8e0883ba56d8a570ef9c6. Six GitHub review findings and the additional nonterminal retry-consumption defect were reproduced against the prior source and repaired:source_operation_deniedwith exit 2.This change remains stacked on #25 at
3ab154be0dd72a6a1ef7967afbdb4b6b39297b00, preserving its Windows subprocess-cleanup and pipe-read classification fixes. Merge that prerequisite normally before retargeting this PR to actual main. Companion host budget/revocation/resume and reviewed Memory contracts: integrations #448. OpenShell lifecycle reconciliation: integrations #447.Validation on the repaired candidate, Windows, Python 3.14.2 and Node 24.13.0:
npm test: 121 Node tests, 118 passed, 0 failed, 3 platform-specific skips. Its Laya Python suite reports 77 tests, 76 passed and one unavailable symlink test skipped.npm run validate,pwsh -NoProfile -File scripts/validate-package.ps1,npm run validate:artifactandgit diff --check: passed. Packed artifact reran the complete tests and package validation in a disposable consumer; the owner installation was only a dry run.Workflow trace: implementation and read-only review; parent owns edits/commit/push, worker
/root/fable_ecf_reviewreviewed correctness, source identity, privacy and bounded evaluation. Recommended Luna worker compatibility fallback was declared. No recursive delegation.Remaining qualification includes an authenticated host importer, durable all-in accounting and authority integration, private representative task suites, real isolated trials and owner-bound promotion/rollback. Required eligible GitHub approval and stack sequencing remain before merge. No paid model call, actual RRSI search, installed plugin change, Memory ingestion, OpenShell activation or deployment occurred.