From 4d68365933bd9381c9a2b70256f60489f4c11aec Mon Sep 17 00:00:00 2001 From: Timothy Wayne Gregg Date: Mon, 28 Sep 2026 13:45:23 -0400 Subject: [PATCH 1/3] Add verified version-pinned SciFact acquisition --- .github/workflows/graph-synthesis.yml | 2 +- graph_synthesis/DATASETS.md | 84 +++++++++++++++++ graph_synthesis/datasets.py | 123 +++++++++++++++++++++++++ graph_synthesis/tests/test_datasets.py | 113 +++++++++++++++++++++++ 4 files changed, 321 insertions(+), 1 deletion(-) create mode 100644 graph_synthesis/DATASETS.md create mode 100644 graph_synthesis/datasets.py create mode 100644 graph_synthesis/tests/test_datasets.py diff --git a/.github/workflows/graph-synthesis.yml b/.github/workflows/graph-synthesis.yml index b052227..6122506 100644 --- a/.github/workflows/graph-synthesis.yml +++ b/.github/workflows/graph-synthesis.yml @@ -34,7 +34,7 @@ jobs: - name: Install pinned original replay dependency run: python -m pip install -r graph_synthesis/requirements-verification.txt - name: Acquire standalone datasets and verify their pinned hashes - run: python -B scripts/download_datasets.py + run: python -B -m graph_synthesis.datasets - name: Verify immutable inputs and replay original observations and tests run: python -B -m graph_synthesis.verify --replay --tests --output "$RUNNER_TEMP/frozen-verification.json" - name: Test relationship semantics, provenance, lifecycle and adapters diff --git a/graph_synthesis/DATASETS.md b/graph_synthesis/DATASETS.md new file mode 100644 index 0000000..8fad94e --- /dev/null +++ b/graph_synthesis/DATASETS.md @@ -0,0 +1,84 @@ +# Acquiring the original dataset bytes + +From the repository root, use Python 3.12: + +```sh +python -B -m graph_synthesis.datasets +python -B -m graph_synthesis.datasets --check +``` + +The current acquisition command first verifies or downloads SciFact into the +ignored `reproduction/data/sources/` directory. It then invokes the unchanged +`scripts/download_datasets.py` to acquire the three commit-pinned DBLP-ACM source +files, regenerate both prepared datasets, and check all six original sizes and +SHA-256 hashes in `scripts/datasets.json`. `--check` uses that same six-file +verifier without downloading or preparing anything. The archived command +`python -B scripts/download_datasets.py --check` remains equivalent. + +## Official version-pinned SciFact source + +The preferred source is the publisher's original S3 object with an explicit +version ID: + + + +| Property | Verified value | +| --- | --- | +| Source owner | Allen Institute for AI, [SciFact repository](https://github.com/allenai/scifact) | +| S3 object version | `8LiW3OUBLBvhehDzrRdGD1ibIaXL.6PQ` | +| Bytes | 3,115,079 | +| Expected and observed SHA-256 | `11c621288d41ac144d29b13b0f8503b3820b7d6e8b1f6ff24dff335c196d76be` | +| Object Last-Modified | 2021-01-26 02:28:59 UTC | +| Verification date | 2026-09-28 UTC | + +The version-specific URL returned those exact archived bytes. Although the key +contains `latest`, the explicit version identifies the existing object: +[S3 versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html) +keeps previous versions when an object is replaced or an ordinary delete marker +is added. The object owner can still permanently remove a version or revoke +access; versioning does not promise perpetual availability. + +If that version cannot be downloaded or fails integrity validation, the command +reports the rejection and tries the original unversioned `latest` URL. Every +response must match the existing byte count and digest before being installed. +An unavailable or changed unversioned URL therefore does not prevent acquisition +while the preferred pinned object is accessible. Both URLs use the original +publisher's bucket; this change creates no new mirror or bundled data copy. + +The publisher's [2021 source change](https://github.com/allenai/scifact/commit/4816a3c7098f7121056276cdfdad228c60daa356) +records the move from `/release/2020-12-17/` to `/release/latest/`. The dated URL +returned HTTP 403 during this review and is not used as a fallback. + +## Recovery with existing verified bytes + +If the publisher's service is unavailable, obtain the original archive from a +previously verified local copy or another source you trust and run: + +```sh +python -B -m graph_synthesis.datasets --scifact-archive /path/to/data.tar.gz +python -B scripts/download_datasets.py --check +python -B -m graph_synthesis.verify --replay --tests +``` + +Use a Windows path and quotes where appropriate. The supplied file must match +the size and SHA-256 above. The command verifies it before replacing a cached +archive. It also reports the source and observed digest. A bad existing cache +is reported without automatically replacing it; a valid `--scifact-archive` +provides an explicit repair. If the three DBLP-ACM sources are also absent, +their normal commit-pinned downloads still require network access. + +Integrity failures include expected and observed sizes and hashes. Never edit +the pinned hashes to accept a different release. Downloads use temporary files +and publish only verified SciFact bytes. Downloaded and prepared datasets remain +ignored and must not be added to Git. + +## Attribution and rights + +Credit David Wadden and coauthors for [SciFact](https://aclanthology.org/2020.emnlp-main.609/) +and Kyle Lo and coauthors for [S2ORC](https://aclanthology.org/2020.acl-main.447/). +The publisher's [license notice at the reviewed revision](https://github.com/allenai/scifact/blob/68b98a56d93e0f9da0d2aab4e6c3294699a0f72e/LICENSE.md) +assigns CC BY 4.0 to claims/evidence annotations and ODC-By 1.0 to the corpus +abstracts; its Apache 2.0 code license is separate. See the preserved +[SciFact notice](../supplementary/licenses/SCIFACT_LICENSE.md) and +[full source, transformation, and rights record](../supplementary/REFERENCES_AND_DATA.md). +The acquisition change retains the exact original data and those notices. diff --git a/graph_synthesis/datasets.py b/graph_synthesis/datasets.py new file mode 100644 index 0000000..7a5cb18 --- /dev/null +++ b/graph_synthesis/datasets.py @@ -0,0 +1,123 @@ +"""Acquire pinned study data without modifying the original archived downloader. + +SciFact is fetched from an immutable version of the publisher's S3 object before +the frozen downloader prepares and verifies all six original dataset files. +""" +from __future__ import annotations + +import argparse +import hashlib +from http.client import HTTPException +import json +from pathlib import Path +import subprocess +import sys +import tempfile +from urllib.request import Request, urlopen + + +REPOSITORY = Path(__file__).resolve().parents[1] +DESTINATION = REPOSITORY / "reproduction/data/sources" +SCIFACT_NAME = "scifact-data.tar.gz" +SCIFACT_LATEST_URL = "https://scifact.s3-us-west-2.amazonaws.com/release/latest/data.tar.gz" +SCIFACT_VERSION_URL = SCIFACT_LATEST_URL + "?versionId=8LiW3OUBLBvhehDzrRdGD1ibIaXL.6PQ" +SCIFACT_SOURCES = (SCIFACT_VERSION_URL, SCIFACT_LATEST_URL) +SCIFACT_RECORD = next(row for row in json.loads( + (REPOSITORY / "scripts/datasets.json").read_text(encoding="utf-8") +) if row["name"] == SCIFACT_NAME) + + +def checked_scifact(content: bytes, source: str) -> str: + """Reject changed bytes before publishing a downloaded or supplied archive.""" + observed = hashlib.sha256(content).hexdigest() + if len(content) != SCIFACT_RECORD["bytes"] or observed != SCIFACT_RECORD["sha256"]: + raise ValueError( + f"SciFact integrity check failed for {source}: expected " + f"{SCIFACT_RECORD['bytes']} bytes, SHA-256 {SCIFACT_RECORD['sha256']}; " + f"observed {len(content)} bytes, SHA-256 {observed}." + ) + return observed + + +def recovery_message() -> str: + return ( + "Supply the original verified archive with " + "python -B -m graph_synthesis.datasets --scifact-archive PATH. " + f"Required SHA-256: {SCIFACT_RECORD['sha256']}. " + "See graph_synthesis/DATASETS.md; do not change the pinned digest." + ) + + +def install_scifact(content: bytes, destination: Path) -> None: + """Publish only checked bytes and remove temporary files if writing fails.""" + destination.parent.mkdir(parents=True, exist_ok=True) + temporary = None + try: + with tempfile.NamedTemporaryFile(dir=destination.parent, delete=False) as output: + temporary = Path(output.name) + output.write(content) + temporary.replace(destination) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + + +def ensure_scifact(destination: Path, local_archive: Path | None = None) -> None: + target = destination / SCIFACT_NAME + if local_archive is not None: + content = local_archive.read_bytes() + source = str(local_archive) + try: + observed = checked_scifact(content, source) + except ValueError as error: + raise ValueError(f"{error} {recovery_message()}") from error + install_scifact(content, target) + elif target.exists(): + source = str(target) + try: + observed = checked_scifact(target.read_bytes(), source) + except ValueError as error: + raise ValueError(f"{error} {recovery_message()}") from error + else: + failures = [] + for source in SCIFACT_SOURCES: + try: + request = Request(source, headers={"User-Agent": "pgc-research-reproduction/1"}) + with urlopen(request, timeout=120) as response: + # Bound a changed server response instead of reading it without limit. + content = response.read(SCIFACT_RECORD["bytes"] + 1) + observed = checked_scifact(content, source) + except (OSError, HTTPException, ValueError) as error: + failure = f"{source}: {error}" + failures.append(failure) + print(f"SciFact source rejected: {failure}", file=sys.stderr) + continue + install_scifact(content, target) + break + else: + raise ValueError("No verified SciFact source was available. " + + "\n".join(failures) + "\n" + recovery_message()) + print(f"Verified SciFact source {source}: {SCIFACT_RECORD['bytes']} bytes, " + f"expected/observed SHA-256 {observed}.", flush=True) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + options = parser.add_mutually_exclusive_group() + options.add_argument("--check", action="store_true", help="Verify all six local files without network access") + options.add_argument("--scifact-archive", type=Path, help="Use an existing original SciFact tarball after checking its bytes") + args = parser.parse_args(argv) + try: + if not args.check: + ensure_scifact(DESTINATION, args.scifact_archive) + command = [sys.executable, "-B", str(REPOSITORY / "scripts/download_datasets.py")] + if args.check: + command.append("--check") + # The archived preparation and six-file integrity rules remain unchanged. + return subprocess.run(command, cwd=REPOSITORY, check=False).returncode + except (OSError, ValueError) as error: + parser.exit(1, f"{error}\n") + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/graph_synthesis/tests/test_datasets.py b/graph_synthesis/tests/test_datasets.py new file mode 100644 index 0000000..9bb0b69 --- /dev/null +++ b/graph_synthesis/tests/test_datasets.py @@ -0,0 +1,113 @@ +"""Dataset acquisition rejects bad bytes before invoking frozen preparation.""" +import contextlib +import hashlib +import io +from pathlib import Path +import tempfile +import unittest +from unittest.mock import patch +from urllib.error import URLError + +from graph_synthesis import datasets + + +class DatasetAcquisitionTests(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.addCleanup(self.directory.cleanup) + self.destination = Path(self.directory.name) / "sources" + self.target = self.destination / datasets.SCIFACT_NAME + self.content = b"original source fixture" + record = {"name": datasets.SCIFACT_NAME, "bytes": len(self.content), + "sha256": hashlib.sha256(self.content).hexdigest()} + self.enterContext(patch.object(datasets, "SCIFACT_RECORD", record)) + self.stdout = self.enterContext(contextlib.redirect_stdout(io.StringIO())) + self.stderr = self.enterContext(contextlib.redirect_stderr(io.StringIO())) + + def test_empty_cache_prefers_versioned_object_without_requesting_latest(self): + with patch.object(datasets, "urlopen", return_value=io.BytesIO(self.content)) as fetch: + datasets.ensure_scifact(self.destination) + self.assertEqual(fetch.call_args.args[0].full_url, datasets.SCIFACT_VERSION_URL) + self.assertEqual(fetch.call_count, 1) + self.assertEqual(self.target.read_bytes(), self.content) + self.assertIn(datasets.SCIFACT_VERSION_URL, self.stdout.getvalue()) + + def test_unavailable_preferred_source_uses_verified_fallback(self): + with patch.object(datasets, "urlopen", side_effect=[URLError("unavailable"), io.BytesIO(self.content)]) as fetch: + datasets.ensure_scifact(self.destination) + self.assertEqual([call.args[0].full_url for call in fetch.call_args_list], + list(datasets.SCIFACT_SOURCES)) + self.assertEqual(self.target.read_bytes(), self.content) + self.assertIn("unavailable", self.stderr.getvalue()) + + def test_changed_source_is_reported_and_only_verified_fallback_is_installed(self): + with patch.object(datasets, "urlopen", side_effect=[io.BytesIO(b"wrong"), io.BytesIO(self.content)]): + datasets.ensure_scifact(self.destination) + self.assertEqual(self.target.read_bytes(), self.content) + self.assertIn(hashlib.sha256(b"wrong").hexdigest(), self.stderr.getvalue()) + self.assertIn(datasets.SCIFACT_VERSION_URL, self.stderr.getvalue()) + + def test_all_mismatches_fail_closed_with_expected_and_observed_digests(self): + bad = b"wrong" + with patch.object(datasets, "urlopen", side_effect=[io.BytesIO(bad), io.BytesIO(bad)]): + with self.assertRaises(ValueError) as raised: + datasets.ensure_scifact(self.destination) + message = str(raised.exception) + self.assertIn(datasets.SCIFACT_VERSION_URL, message) + self.assertIn(datasets.SCIFACT_LATEST_URL, message) + self.assertIn(hashlib.sha256(self.content).hexdigest(), message) + self.assertIn(hashlib.sha256(bad).hexdigest(), message) + self.assertIn("--scifact-archive PATH", message) + self.assertFalse(self.target.exists()) + + def test_all_unavailable_sources_provide_local_recovery(self): + with patch.object(datasets, "urlopen", side_effect=URLError("offline")): + with self.assertRaisesRegex(ValueError, "--scifact-archive PATH"): + datasets.ensure_scifact(self.destination) + self.assertFalse(self.target.exists()) + + def test_existing_verified_archive_requires_no_network(self): + self.destination.mkdir() + self.target.write_bytes(self.content) + with patch.object(datasets, "urlopen", side_effect=AssertionError("network attempted")): + datasets.ensure_scifact(self.destination) + self.assertEqual(self.target.read_bytes(), self.content) + + def test_corrupt_cached_archive_is_preserved_and_reported(self): + self.destination.mkdir() + self.target.write_bytes(b"corrupt") + with patch.object(datasets, "urlopen", side_effect=AssertionError("network attempted")): + with self.assertRaisesRegex(ValueError, "observed 7 bytes"): + datasets.ensure_scifact(self.destination) + self.assertEqual(self.target.read_bytes(), b"corrupt") + + def test_local_archive_recovery_checks_before_replacing_cache(self): + self.destination.mkdir() + self.target.write_bytes(b"corrupt") + local = Path(self.directory.name) / "supplied.tar.gz" + local.write_bytes(b"wrong") + with patch.object(datasets, "urlopen", side_effect=AssertionError("network attempted")): + with self.assertRaises(ValueError): + datasets.ensure_scifact(self.destination, local) + self.assertEqual(self.target.read_bytes(), b"corrupt") + local.write_bytes(self.content) + datasets.ensure_scifact(self.destination, local) + self.assertEqual(self.target.read_bytes(), self.content) + + def test_write_failure_removes_temporary_download(self): + with patch.object(Path, "replace", side_effect=OSError("write failed")): + with self.assertRaisesRegex(OSError, "write failed"): + datasets.install_scifact(self.content, self.target) + self.assertEqual(list(self.destination.iterdir()), []) + + def test_check_only_delegates_to_frozen_six_file_verifier(self): + with patch.object(datasets, "ensure_scifact", side_effect=AssertionError("acquisition attempted")): + with patch.object(datasets.subprocess, "run") as run: + run.return_value.returncode = 7 + self.assertEqual(datasets.main(["--check"]), 7) + self.assertEqual(run.call_args.args[0][-2:], + [str(datasets.REPOSITORY / "scripts/download_datasets.py"), "--check"]) + + +if __name__ == "__main__": + unittest.main() From 2e170a3f99a13cbd6494e4f885a834b4e542f028 Mon Sep 17 00:00:00 2001 From: Timothy Wayne Gregg Date: Mon, 28 Sep 2026 13:47:34 -0400 Subject: [PATCH 2/3] Point graph reproduction guide to current dataset acquisition --- graph_synthesis/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/graph_synthesis/README.md b/graph_synthesis/README.md index 2791380..035883a 100644 --- a/graph_synthesis/README.md +++ b/graph_synthesis/README.md @@ -1,6 +1,6 @@ # Evidence-preserving graph synthesis extension -This additive extension turns exact saved Jev observations into durable, source-bound graph assertions, analyzes relationships and topology, and exercises correction/retraction. It preserves all 161 files in the current baseline MANIFEST inventory. Standalone datasets remain ignored and are acquired through the existing pinned downloader. Read [the illustrated research paper](paper.md), [15-figure gallery](figures/README.md), [visual review and limitations](VISUAL_REVIEW.md), [results](RESULTS.md), [protocol](PROTOCOL.md), and [relationship design notes](RELATIONSHIPS.md). +This additive extension turns exact saved Jev observations into durable, source-bound graph assertions, analyzes relationships and topology, and exercises correction/retraction. It preserves all 161 files in the current baseline MANIFEST inventory. Standalone datasets remain ignored and use [verified source acquisition](DATASETS.md) with the original frozen preparation code. Read [the illustrated research paper](paper.md), [15-figure gallery](figures/README.md), [visual review and limitations](VISUAL_REVIEW.md), [results](RESULTS.md), [protocol](PROTOCOL.md), and [relationship design notes](RELATIONSHIPS.md). ## Complementary ten-figure analysis @@ -20,7 +20,7 @@ From the repository root, use Python 3.12 and the original NumPy pin for closest ```sh python -m pip install -r graph_synthesis/requirements-verification.txt -python -B scripts/download_datasets.py +python -B -m graph_synthesis.datasets python -B -m graph_synthesis.verify --replay --tests --output ../original-verification.json python -B -m unittest discover -s graph_synthesis/tests -v python -B -m graph_synthesis.study --output ../graph-study.json --export-graphs ../graphs From 234245671231941d2c38be3199b869817fd8cdff Mon Sep 17 00:00:00 2001 From: Timothy Wayne Gregg Date: Mon, 28 Sep 2026 14:32:57 -0400 Subject: [PATCH 3/3] Align manual validation with verified dataset acquisition --- .github/LOCAL_VALIDATION.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/LOCAL_VALIDATION.md b/.github/LOCAL_VALIDATION.md index b0b5435..b5efcce 100644 --- a/.github/LOCAL_VALIDATION.md +++ b/.github/LOCAL_VALIDATION.md @@ -10,13 +10,13 @@ instructions describe a closed historical package; expanded checkouts use ```sh python -B -m graph_synthesis.verify -python -B scripts/download_datasets.py --check +python -B -m graph_synthesis.datasets --check python -B -m graph_synthesis.verify --replay --tests --output ../original-verification.json python -B -m unittest discover -s graph_synthesis/tests -v ``` -Acquire missing datasets using the current reproduction guide before the second -and third commands. The portable replay blocks network access and reads recorded +Acquire missing datasets with `python -B -m graph_synthesis.datasets` before the +second and third commands. The portable replay blocks network access and reads recorded responses. Keep output reports outside the repository and retain the command, environment, commit SHA, and resulting verification report. @@ -71,4 +71,3 @@ manual dispatch. No write permission or model-call capability is newly enabled. For example, an authorized manual graph verification can be started with `gh workflow run graph-synthesis.yml --ref main`. This is optional; all local verification commands above remain available. -