Skip to content

Implement source-bound RRSI candidate and evaluation contracts - #26

Merged
rhein1 merged 8 commits into
mainfrom
codex/fable-rrsi-source-20260930
Oct 14, 2026
Merged

rhein1 merged 8 commits into
mainfrom
codex/fable-rrsi-source-20260930

Conversation

@rhein1

@rhein1 rhein1 commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Implements the source-only Fable RRSI candidate and evaluation path for issue #24. A reviewed Fable commit is exported into a disposable repository with exact Git-blob provenance; only two finite-vocabulary prompt fragments can change. Candidate admission binds actual trees, modes, worktree bytes and independently tagged line edits while preserving the six workflows, ECF/evidence contracts and model/worker configuration.

The evaluator freezes source excerpts, factual assertions, uniform weights, split/model/runtime identities and retry/cost policy. It grades typed facts with exact evidence citations, retains missing slots and all reported retry consumption, and rejects unsafe, unknown-cost or incomplete comparisons. Held-out/OOD packets remain sealed. Live imports, acceptance and promotion stay disabled; synthetic fixtures do not establish improved Fable performance.

The repaired head is 3829e76c0072e140a4a8e0883ba56d8a570ef9c6. Six GitHub review findings and the additional nonterminal retry-consumption defect were reproduced against the prior source and repaired:

  • All Git source reads disable replacement objects before resolving commits or blobs.
  • Diff identity uses canonical unified bytes from verified ASCII blobs, independent of repository diff configuration.
  • Aggregation keys bind task, split, trial and seed.
  • Frozen assertions reject repeated fact paths as well as repeated IDs.
  • Prompt screening caps each prompt at 1,024 characters and cumulative comparison work at four million character products before further comparison.
  • Root and subcommand parser errors emit only source_operation_denied with exit 2.
  • Every attempt must be terminal with known positive consumption for a successful retry to become comparable. Unresolved attempts retain reported consumption in totals and block comparison.

This change remains stacked on #25 at 3ab154be0dd72a6a1ef7967afbdb4b6b39297b00, preserving its Windows subprocess-cleanup and pipe-read classification fixes. Merge that prerequisite normally before retargeting this PR to actual main. Companion host budget/revocation/resume and reviewed Memory contracts: integrations #448. OpenShell lifecycle reconciliation: integrations #447.

Validation on the repaired candidate, Windows, Python 3.14.2 and Node 24.13.0:

  • Source regression suite: 30 passed, 0 failed, 0 skipped.
  • Evaluator regression suite: 28 passed, 0 failed, 0 skipped.
  • npm test: 121 Node tests, 118 passed, 0 failed, 3 platform-specific skips. Its Laya Python suite reports 77 tests, 76 passed and one unavailable symlink test skipped.
  • npm run validate, pwsh -NoProfile -File scripts/validate-package.ps1, npm run validate:artifact and git diff --check: passed. Packed artifact reran the complete tests and package validation in a disposable consumer; the owner installation was only a dry run.
  • Independent read-only review of all six changed files and their relevant callers found no remaining source defect. This is not a formal approving GitHub review.
  • Hosted validation on 3829e76: all eight checks passed (Ubuntu/macOS/Windows on Node 18/24, packed artifact, release gate). Literal tested merge ref bb5e71b has the same tracked tree as the reviewed branch head. Matrix runs have zero failures; platform-specific Node test skips are recorded in their logs. No required check was skipped.

Workflow trace: implementation and read-only review; parent owns edits/commit/push, worker /root/fable_ecf_review reviewed correctness, source identity, privacy and bounded evaluation. Recommended Luna worker compatibility fallback was declared. No recursive delegation.

Remaining qualification includes an authenticated host importer, durable all-in accounting and authority integration, private representative task suites, real isolated trials and owner-bound promotion/rollback. Required eligible GitHub approval and stack sequencing remain before merge. No paid model call, actual RRSI search, installed plugin change, Memory ingestion, OpenShell activation or deployment occurred.

@rhein1

rhein1 commented Oct 1, 2026

Copy link
Copy Markdown
Owner Author

P2: Nonterminal retry history can be reported as cost-comparable

Reviewed head: 128a0eb373b1d39e8577f6331b00692b060357c2.

In aggregate_results, completion comes from the final retry, while earlier attempts need only positive reported cost/token values. An earlier unknown or timeout attempt with terminal:false, followed by a successful retry, returns comparable_synthetic:true, score 1.0, and zero missing trials. Its unresolved execution cannot establish complete final consumption or cleanup.

Expected gate: unresolved/nonterminal attempts must block cost comparability until their terminal outcomes and consumption are reconciled. Preserve all retry totals and the valid terminal failed-attempt followed by success control.

Reproduction: save this as a temporary Python file and run python -I -B PATH_TO_FILE from this head's repository root. It uses the actual evaluator and existing fixtures; the final assertion fails for both unknown and timeout, while the terminal-failure control passes.

import runpy
f = runpy.run_path("test/rrsi_evaluator_test.py")
e = f["e"]
config = f["config"]()
slot = f["slot"](config)
observed = {}
for status, terminal in (("failed", True), ("unknown", False), ("timeout", False)):
    rows = [f["observation"](slot, status=status, terminal=terminal, result=None),
            f["observation"](slot, attempt=1)]
    result = e.aggregate_results(observations=rows, expected_slots=[slot],
                                 config=config, synthetic=True)
    observed[status] = result["comparable_synthetic"]
    print(status, terminal, observed[status], result["total_cost_microusd"])
assert observed["failed"] is True
assert not any(observed[s] for s in ("unknown", "timeout")), observed

The existing evaluator suite passes 24/24 on this exact head. This is an offline synthetic-contract finding: accepted:false, promotion_allowed:false, and the live-import denial remain intact. No provider call, live activation, owner-file edit or push was performed for this review.

@rhein1
rhein1 marked this pull request as ready for review October 1, 2026 23:10
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-01T23:24:46.942276Z 128a0eb Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 128a0eb373

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/fable5-codex/experiments/rrsi/source.py
Comment thread plugins/fable5-codex/experiments/rrsi/evaluator.py Outdated
Comment thread plugins/fable5-codex/experiments/rrsi/source.py Outdated
Comment thread plugins/fable5-codex/experiments/rrsi/evaluator.py Outdated
Comment thread plugins/fable5-codex/experiments/rrsi/cli.py
Comment thread plugins/fable5-codex/experiments/rrsi/evaluator.py Outdated
@rhein1
rhein1 changed the base branch from codex/fable-windows-cleanup-20260930 to main October 14, 2026 20:35

@jmborden1112-beep jmborden1112-beep left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Needs Fixing

None. RRSI source identity, bounded synthetic evidence and explicit pairing contracts remain consistent. The apparent single-arm aggregation concern was refuted: aggregation is per arm and match_arms handles pairing separately. No live optimization/promotion is implemented.

Requires Human Review

None beyond the repository approval gate fulfilled under the owner's second-account authorization. This is agent source review, not independent human sign-off.

Recommended Optional

None.

Create Follow-up Issue

None.

Workflow Trace: exact final head6e65019db917c52ee4c7f9fe36c0a02488215d3f includes merged25,22,23. Source lens /root/fable_stack_review plus root verification, gpt-6-luna medium fallback. Final combined local npm suite211passed/3platformskips/0failed; both package validators and diff check pass. Earlier RRSI source30/30 and evaluator28/28 Python tests passed. All eight final-head hosted checks pass, including Windows Node24, packed artifact and Release gate. No paid model trials, provider qualification, source improvement claims, publication, or runtime activation.

@rhein1
rhein1 merged commit 4e5a77e into main Oct 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants