Repository navigation
Research: five uncertainty and grounding improvements with executed falsification benchmarks #21
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| name: Uncertainty and grounding experiments | ||
| on: | ||
| push: | ||
| branches: [main, research/uncertainty-grounding-20260918] | ||
| pull_request: | ||
| permissions: | ||
| contents: read | ||
| concurrency: | ||
| group: uncertainty-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
| jobs: | ||
| execute-and-publish: | ||
| if: >- | ||
| github.event_name == 'push' && github.ref == 'refs/heads/research/uncertainty-grounding-20260918' && !startsWith(github.event.head_commit.message, 'ci-review-snapshot') | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| permissions: | ||
| contents: write | ||
| env: | ||
| PYTHONDONTWRITEBYTECODE: '1' | ||
| steps: | ||
| - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 | ||
| with: | ||
| ref: ${{ github.sha }} | ||
| fetch-depth: 0 | ||
| - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 | ||
| with: | ||
| python-version: '3.12' | ||
| - run: python -m pip install -r graph_synthesis/requirements-figures.txt -r graph_synthesis/uncertainty/requirements.txt | ||
| - name: Integrate preceding studies and the audited novelty disclosure | ||
| shell: bash | ||
| run: | | ||
| python - <<'PY' | ||
| from pathlib import Path | ||
| import re | ||
| index=Path('CURRENT_RESULTS.md').read_text(encoding='utf-8') | ||
| anchors=[a for a in re.findall(r'<!-- [A-Z][A-Z0-9_]*_RESEARCH_END -->',index) if a!='<!-- UNCERTAINTY_RESEARCH_END -->'] | ||
| assert anchors, 'No preceding study anchor' | ||
| anchor=anchors[-1] | ||
| assert anchor in Path('manuscript/paper-current.md').read_text(encoding='utf-8') | ||
| p=Path('graph_synthesis/uncertainty/report.py') | ||
| text,count=re.subn(r'^ANCHOR = .+$','ANCHOR = '+repr(anchor),p.read_text(encoding='utf-8'),flags=re.M) | ||
| assert count==1 | ||
| assert Path('graph_synthesis/uncertainty/NOVELTY_AUDIT.md').is_file() | ||
| if 'NOVELTY_AUDIT.md' not in text: | ||
| lines=text.splitlines(keepends=True) | ||
| positions=[i for i,line in enumerate(lines) if line.strip().startswith('return ') and '.join(out)' in line] | ||
| assert len(positions)==1 | ||
| lines[positions[0]:positions[0]]=[ | ||
| " audit = (HERE / 'NOVELTY_AUDIT.md').read_text(encoding='utf-8').strip()\n", | ||
| " position = out.index('| Hypothesis | Frozen primary target | Outcome | Evidence population |')\n", | ||
| " out[position:position] = [audit, '']\n"] | ||
| text=''.join(lines) | ||
| p.write_text(text,encoding='utf-8',newline='\n') | ||
| p=Path('graph_synthesis/uncertainty/EXECUTION_NOTES.md') | ||
| text=p.read_text(encoding='utf-8') | ||
| heading='## Integration after '+anchor | ||
| if heading not in text: | ||
| text+='\n'+heading+'\n\nConcurrent PR #19 added source-aware and structural experiments after PR #18. Both are preserved in the combined manuscript. This extension is inserted after the final study in CURRENT_RESULTS.md. The original hypothesis baseline remains a62a3257645d8e35cd4e45be53bfa9511d27724b; no generator, algorithm, seed, endpoint or threshold changed. All five numerical result blocks are compared byte-for-value with the earlier authoritative run. Source/protocol/build hashes and the paper are regenerated. The full-manuscript CI pipeline now regenerates this extension as well as every prior section, rather than dropping it. Reader-facing research markers are hidden by the renderer, covered by two additional tests. These presentation/integration corrections do not alter scientific observations. The separately documented post-execution novelty audit explicitly identifies overlap with concurrent PR #20; these are not five independently unprecedented algorithms.\n' | ||
| p.write_text(text,encoding='utf-8',newline='\n') | ||
| PY | ||
| - name: Execute frozen benchmarks, unit tests and independent replay | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| python -B -m graph_synthesis.uncertainty.run | tee graph_synthesis/uncertainty/execution.log | ||
| python -B -m unittest discover -s graph_synthesis/uncertainty/tests -v 2>&1 | tee graph_synthesis/uncertainty/tests.log | ||
| python -B -m graph_synthesis.uncertainty.run --check | ||
| python -B -m graph_synthesis.verify --output "$RUNNER_TEMP/uncertainty-original-inventory.json" | ||
| cp "$RUNNER_TEMP/uncertainty-original-inventory.json" graph_synthesis/uncertainty/original-inventory.json | ||
| python -B -m graph_synthesis.uncertainty.report --figures --update-paper | ||
| python -B -m graph_synthesis.render_current_paper --output manuscript/paper-current.pdf | ||
| - name: Preserve preceding studies and all previously executed numerical outcomes | ||
| shell: bash | ||
| run: | | ||
| python - <<'PY' | ||
| from pathlib import Path | ||
| import json,os,re,subprocess | ||
| from graph_synthesis.uncertainty.report import START, END | ||
| def strip(text): | ||
| return re.sub(re.escape(START)+r'.*?'+re.escape(END)+r'\s*','',text,flags=re.S) | ||
| for path in ('CURRENT_RESULTS.md','manuscript/paper-current.md'): | ||
| before=subprocess.check_output(['git','show',os.environ['GITHUB_SHA']+':'+path]).decode('utf-8') | ||
| after=Path(path).read_text(encoding='utf-8') | ||
| assert strip(before)==strip(after), 'Preceding text changed: '+path | ||
| before=json.loads(subprocess.check_output(['git','show','08fa7c4673bc7084574951ee2b0a426288087852:graph_synthesis/uncertainty/results.json'])) | ||
| after=json.loads(Path('graph_synthesis/uncertainty/results.json').read_text(encoding='utf-8')) | ||
| for i in range(1,6): | ||
| assert before[f'H{i}']==after[f'H{i}'],f'H{i} changed during manuscript integration' | ||
| assert 'Do not describe this suite as five independently novel' in Path('manuscript/paper-current.md').read_text(encoding='utf-8') | ||
| print('All preceding study text and all five numerical result blocks preserved; novelty overlap disclosed') | ||
| PY | ||
| - name: Commit generated evidence only to this research branch | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| git config user.name 'github-actions[bot]' | ||
| git config user.email '41898282+github-actions[bot]@users.noreply.github.com' | ||
| git add graph_synthesis/uncertainty CURRENT_RESULTS.md manuscript/paper-current.md manuscript/paper-current.pdf manuscript/paper-current.html manuscript/paper-current.build.json | ||
| if ! git diff --cached --quiet; then | ||
| git commit -m 'research: record executed uncertainty benchmarks, five figures and full updated manuscript' | ||
| git push origin HEAD:research/uncertainty-grounding-20260918 | ||
| fi | ||
| - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 | ||
| with: | ||
| name: uncertainty-review-package | ||
| path: | | ||
| graph_synthesis/uncertainty/ | ||
| graph_synthesis/render_current_paper.py | ||
| manuscript/paper-current.pdf | ||
| manuscript/paper-current.md | ||
| manuscript/paper-current.html | ||
| manuscript/paper-current.build.json | ||
| CURRENT_RESULTS.md | ||
| retention-days: 14 | ||
| replay: | ||
| if: github.event_name == 'pull_request' || github.ref == 'refs/heads/main' | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: [ubuntu-latest, windows-latest] | ||
| runs-on: ${{ matrix.os }} | ||
| timeout-minutes: 15 | ||
| defaults: | ||
| run: | ||
| shell: bash | ||
| env: | ||
| PYTHONDONTWRITEBYTECODE: '1' | ||
| steps: | ||
| - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 | ||
| with: | ||
| ref: ${{ github.event.pull_request.head.sha || github.sha }} | ||
| persist-credentials: false | ||
| - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 | ||
| with: | ||
| python-version: '3.12' | ||
| - run: python -m pip install -r reproduction/requirements-jev.txt -r graph_synthesis/uncertainty/requirements.txt | ||
| - name: Replay all results and verify source, protocol, archive and PDF provenance | ||
| run: | | ||
| python -B -m unittest discover -s graph_synthesis/uncertainty/tests -v | ||
| python -B -m graph_synthesis.uncertainty.run --check | ||
| python -B -m graph_synthesis.verify --output "$RUNNER_TEMP/uncertainty-original.json" | ||
| python - <<'PY' | ||
| import hashlib,json | ||
| from pathlib import Path | ||
| sha=lambda p:hashlib.sha256(Path(p).read_bytes()).hexdigest() | ||
| manifest=json.loads(Path('graph_synthesis/uncertainty/artifact-manifest.json').read_text(encoding='utf-8')) | ||
| for path,want in manifest['sha256'].items(): | ||
| assert sha(path)==want,path | ||
| r=json.loads(Path('graph_synthesis/uncertainty/results.json').read_text(encoding='utf-8')) | ||
| assert r['protocol_sha256']==sha('graph_synthesis/uncertainty/PROTOCOL.md') | ||
| assert r['fresh_service_calls']==0 and r['new_scientific_documents']==0 | ||
| for name,want in r['source_sha256'].items(): | ||
| assert sha('graph_synthesis/uncertainty/'+name)==want,name | ||
| build=json.loads(Path('manuscript/paper-current.build.json').read_text(encoding='utf-8')) | ||
| assert sha('manuscript/paper-current.md')==build['source_sha256'] | ||
| assert sha('graph_synthesis/render_current_paper.py')==build['renderer_sha256'] | ||
|
Comment on lines
+155
to
+157
There was a problem hiding this comment.
The renderer changed in this commit, but Useful? React with 👍 / 👎. |
||
| assert sha('manuscript/paper-current.pdf')==build['pdf_sha256'] | ||
| print('Source, protocol, result, archive and full-paper provenance verified') | ||
| PY | ||
| git diff --exit-code | ||
| figures-and-paper: | ||
| if: github.event_name == 'pull_request' || github.ref == 'refs/heads/main' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
| env: | ||
| PYTHONDONTWRITEBYTECODE: '1' | ||
| steps: | ||
| - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 | ||
| with: | ||
| ref: ${{ github.event.pull_request.head.sha || github.sha }} | ||
| persist-credentials: false | ||
| - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 | ||
| with: | ||
| python-version: '3.12' | ||
| - run: python -m pip install -r graph_synthesis/requirements-figures.txt -r graph_synthesis/uncertainty/requirements.txt | ||
| - name: Regenerate charts and manuscript without artifact drift | ||
| run: | | ||
| python -B -m graph_synthesis.uncertainty.report --figures --update-paper | ||
| git diff --exit-code -- graph_synthesis/uncertainty manuscript/paper-current.md CURRENT_RESULTS.md | ||
|
Comment on lines
+179
to
+180
There was a problem hiding this comment.
At this commit neither Useful? React with 👍 / 👎. |
||
| python -B -m graph_synthesis.render_current_paper --output "$RUNNER_TEMP/uncertainty-paper.pdf" | ||
| git diff --exit-code -- manuscript/paper-current.html | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Execution and correction record | ||
|
|
||
| The remote protocol was committed as `383bab6f559c7f14525ddcef2275f3a816c9caa3` before the first local benchmark run. The first local run exercised all five frozen fixture generators and met all five conditional engineering targets. Its workspace did not contain the full repository or a local protocol copy, so its temporary metadata had a null protocol hash; that output is not the authoritative archived run. The full repository run records the committed protocol hash and is independently replayed in CI. | ||
|
|
||
| During unit-test authoring, two tests initially called existing oracle helpers with the wrong signature/return shape (the finite-model oracle accepts two arguments, and the retraction oracle returns a tuple). Those test calls were corrected. No benchmark generator, seed, primary endpoint, threshold, algorithm or unfavorable control was changed in response to observed outcomes. | ||
|
|
||
| The numerical LP output is recorded to 12 decimal places for deterministic evidence serialization. Full primary comparisons occur before rounding. Separate rational small-case extrema, finite subset oracles and model-interpretation enumeration remain in the suite. Synthetic fixture counts and repeated snapshot occurrences are not counted as new Jev observations. | ||
|
|
||
| ## Concurrent PR #18 integration | ||
|
|
||
| PR #18 merged after this protocol was frozen and while these benchmarks were executing. Its structural study is preserved in full. The reporting anchor was moved after its section so both generators are idempotent, and the paper was rebuilt from current main rather than overwriting the concurrent study. The hypothesis baseline remains a62a3257645d8e35cd4e45be53bfa9511d27724b. No benchmark generator, algorithm, seed, endpoint or threshold changed. Source hashes are refreshed because the report placement changed. PR #18 also studies dependence-robust group-review envelopes and independent-primitive exact lineage; those share prior-art foundations with this extension but do not supply its unknown-dependence DNF linear programs, all-optimal-repair backbone, downstream query-loss objective, protected minimum-collateral plans or grounded recursive-withdrawal experiments. Novelty remains bounded to the reviewed baseline, not an exhaustive worldwide or concurrent-branch claim. | ||
|
|
||
| ## Integration after <!-- SOURCE_STRUCTURAL_RESEARCH_END --> | ||
|
|
||
| Concurrent PR #19 added source-aware and structural experiments after PR #18. Both are preserved in the combined manuscript. This extension is inserted after the final study in CURRENT_RESULTS.md. The original hypothesis baseline remains a62a3257645d8e35cd4e45be53bfa9511d27724b; no generator, algorithm, seed, endpoint or threshold changed. All five numerical result blocks are compared byte-for-value with the earlier authoritative run. Source/protocol/build hashes and the paper are regenerated. The full-manuscript CI pipeline now regenerates this extension as well as every prior section, rather than dropping it. Reader-facing research markers are hidden by the renderer, covered by two additional tests. These presentation/integration corrections do not alter scientific observations. Package-level novelty remains scoped to the reviewed frozen baseline; no worldwide or unreviewed concurrent-branch novelty claim is made. | ||
|
|
||
| ## Integration after <!-- ASSUMPTION_AWARE_RESEARCH_END --> | ||
|
|
||
| Concurrent PR #19 added source-aware and structural experiments after PR #18. Both are preserved in the combined manuscript. This extension is inserted after the final study in CURRENT_RESULTS.md. The original hypothesis baseline remains a62a3257645d8e35cd4e45be53bfa9511d27724b; no generator, algorithm, seed, endpoint or threshold changed. All five numerical result blocks are compared byte-for-value with the earlier authoritative run. Source/protocol/build hashes and the paper are regenerated. The full-manuscript CI pipeline now regenerates this extension as well as every prior section, rather than dropping it. Reader-facing research markers are hidden by the renderer, covered by two additional tests. These presentation/integration corrections do not alter scientific observations. The separately documented post-execution novelty audit explicitly identifies overlap with concurrent PR #20; these are not five independently unprecedented algorithms. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On pushes to
research/uncertainty-grounding-20260918, this generated commit is immediately pushed back to the same branch, but the job condition only suppresses messages beginning withci-review-snapshot. The resultingresearch: record ...push therefore launches this job again; because the committedgraph_synthesis/uncertainty/tests.logcontains variable elapsed timing from line 53 and the entire directory is staged, subsequent runs can keep producing another commit and create an unbounded workflow/commit loop. Use the excluded prefix for this commit, skip bot-authored pushes, or avoid committing nondeterministic logs.Useful? React with 👍 / 👎.