Skip to content

Document reproduction from the expanded repository #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CompleteDotTech merged 1 commit into from
Sep 28, 2026
Merged

Document reproduction from the expanded repository #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions .github/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

![Research: Empirical_study](https://img.shields.io/badge/Research-Empirical_study-0f766e?style=flat-square) ![Evidence: Frozen_archive](https://img.shields.io/badge/Evidence-Frozen_archive-475569?style=flat-square)

[Manuscript](../manuscript/paper.md) · [Claim map](../supplementary/CLAIM_EVIDENCE.md) · [Graph research](../graph_synthesis/README.md) · [Reproduction](#reproduce-and-verify)
[Manuscript](../manuscript/paper.md) · [Claim map](../supplementary/CLAIM_EVIDENCE.md) · [Graph research](../graph_synthesis/README.md) · [Reproduction](../CURRENT_REPRODUCTION.md)

</div>

Expand Down Expand Up @@ -37,15 +37,7 @@ The intervention combines question formulation and demonstrations. These finding

## Reproduce and verify

From the repository root, use Python 3.12 and an isolated environment:

```sh
python -m pip install -r graph_synthesis/requirements-verification.txt
python -B scripts/download_datasets.py
python -B -m graph_synthesis.verify --replay --tests
```

Dependency installation and the pinned dataset download need network access. The subsequent replay and original tests use a temporary runtime with network connections blocked and make no live model calls. The [extension guide](../graph_synthesis/README.md) explains cross-platform numerical tolerances and further experiments.
Follow the [current-repository reproduction guide](../CURRENT_REPRODUCTION.md) for Windows and Unix commands, pinned dependencies, dataset hash checks, offline replay, original tests, and an external verification report. The [extension guide](../graph_synthesis/README.md) explains further experiments.

Use the extension verifier for this expanded repository. The original `scripts/verify_package.py` expects the original closed inventory and rejects the additional extension files.

Expand Down
29 changes: 29 additions & 0 deletions CURRENT_REPRODUCTION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Reproduce the current repository

Run these commands from the root of a fresh checkout of this repository. Use Python 3.12. Keep the virtual environment and verification report in the parent directory so they are outside the archived package inventory.

## Windows PowerShell

```powershell
py -3.12 -m venv ..\paper-package-verify-venv
..\paper-package-verify-venv\Scripts\python.exe -m pip install -r graph_synthesis/requirements-verification.txt
..\paper-package-verify-venv\Scripts\python.exe -B -m graph_synthesis.datasets
..\paper-package-verify-venv\Scripts\python.exe -B -m graph_synthesis.datasets --check
..\paper-package-verify-venv\Scripts\python.exe -B -m graph_synthesis.verify --replay --tests --output ..\paper-package-verification.json
```

## Linux or macOS shell

```sh
python3.12 -m venv ../paper-package-verify-venv
../paper-package-verify-venv/bin/python -m pip install -r graph_synthesis/requirements-verification.txt
../paper-package-verify-venv/bin/python -B -m graph_synthesis.datasets
../paper-package-verify-venv/bin/python -B -m graph_synthesis.datasets --check
../paper-package-verify-venv/bin/python -B -m graph_synthesis.verify --replay --tests --output ../paper-package-verification.json
```

Dependency installation and dataset acquisition need network access. The downloader obtains four pinned source files, regenerates two prepared splits, and checks the sizes and SHA-256 hashes of all six files against `scripts/datasets.json`. The downloaded files live in the Git-ignored `reproduction/data/sources/` directory. The `--check` command verifies those six local files without downloading them again. See the [dataset acquisition guide](graph_synthesis/DATASETS.md) for the pinned SciFact source and recovery from an exact local archive if that source is unavailable.

The verifier checks all 161 files in the original archive against `MANIFEST.json`, then copies the reproduction runtime to a temporary directory. Replay and the original test suite run there with network connections blocked; they make no live model calls and require no API key. The verifier prints `PORTABLE_VERIFICATION` and writes the same JSON report to `../paper-package-verification.json`. A passing report has `original_files_changed: 0`, `network_attempts: 0`, zero original test failures and errors, and replay artifact comparisons. Newline normalization and narrowly bounded floating-point roundoff are reported explicitly; a passing replay is not a claim of bitwise equality across operating systems.

The root [REPRODUCE.md](REPRODUCE.md) records how to run the original closed 161-file archive. Its `scripts/verify_package.py` command rejects this expanded checkout by design. Use `graph_synthesis.verify` above for current main. The [graph synthesis guide](graph_synthesis/README.md) describes additional experiments and checks. Optional fresh service experiments are described in the archived guide; they require a key, network access, and paid model calls, and are separate from this offline replay.