From 9f4b9315f1760c6bbf101ac1fd1c302629663a39 Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Thu, 3 Sep 2026 15:01:59 +0100 Subject: [PATCH 01/11] docs: add code quality maintenance roadmap --- spec/roadmap.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 spec/roadmap.md diff --git a/spec/roadmap.md b/spec/roadmap.md new file mode 100644 index 0000000..aa44d7d --- /dev/null +++ b/spec/roadmap.md @@ -0,0 +1,58 @@ +# Maintenance roadmap + +Legend: [x] done, [~] in progress, [ ] not started + +This roadmap records the maintenance findings from the August 2026 repository audit. Items are ordered by current priority, but their stable identifiers must not be renumbered when priorities change. + +## Guiding constraints + +- Treat exported names as public API unless a compatibility review proves otherwise. +- Characterize calculation results, chart structures, and serialized response shapes before changing implementation or formatting clinical code. +- Do not regenerate or broadly rewrite clinical golden fixtures as part of mechanical cleanup. Any expected-value change needs an independently justified clinical or reference-data reason. +- Test distribution changes against the built wheel, not only against a source checkout. +- Run candidate wheels against the downstream server and its API contract tests after the current cross-repository provenance upgrade has landed. +- Do not delete clinical reference data, validation fixtures, source publications, or provenance material solely because static analysis reports no Python consumer. +- Keep runtime reference data in the package, but keep test data and provenance-only source material outside the wheel unless users have a documented need for it. +- Preserve the current and deprecated UK-WHO validation fixtures unless a separately reviewed reference change explicitly replaces them. +- Run the full test suite after each behavioural or reference-data change, with focused tests for the affected reference boundary or calculation. + +## Foundation and quality gates + +Complete these items before broad formatting, import cleanup, packaging reduction, or dead-code removal. They establish the baseline needed to distinguish intended maintenance from calculation or API regressions. + +- [ ] **R16 - Introduce Ruff in staged, reviewable increments.** Add project configuration and a lint entry point early, excluding generated/reference vectors such as `rcpchgrowth/tests/who_test_data/who_chart_precalculated_centiles.py` from formatting. Begin with audit-only checks and small rule groups; do not run global or unsafe auto-fixes. Configure the `python-dateutil` to `dateutil` module mapping and account for tooling dependencies so false positives do not encourage removal of required packages. Keep type checking as a separate future concern. +- [ ] **R17 - Record a reproducible pre-cleanup baseline.** Run the complete supported test suite, record collection/pass/skip totals, verify the primary UK-WHO fixture count and integrity, and retain focused commands for calculation, chart, and package tests. Understand any baseline failure before maintenance work starts. +- [ ] **R18 - Characterize the public Python import surface.** Add tests for the root names and module imports used by `digital-growth-charts-server`, including call signatures or representative invocations where practical. Use this contract to guide `__all__`, import cleanup, and deprecation decisions rather than assuming apparently unused exports are private. +- [ ] **R19 - Characterize serialized measurement contracts.** Add golden or schema-focused tests for the complete `Measurement.measurement` dictionary across representative successful, corrected-age, out-of-range, and validation-error cases. Assert key presence, nesting, nullability, value types, reference identifiers, and clinically significant numeric values. +- [ ] **R20 - Fill calculation-vector and boundary gaps.** Add independent representative vectors for Turner syndrome, Trisomy 21, and Trisomy 21 AAP; CDC infant head circumference and extended BMI; exact component reference-selection handovers; and documented error boundaries for every supported reference. Prefer authoritative published vectors or separately derived expected values over self-generated snapshots. +- [ ] **R21 - Characterize chart-builder outputs.** Test the structure, series selection, age domains, boundary ownership, and representative coordinates of dynamically generated charts for every supported reference, including the non-WHO chart families. Exercise generation directly rather than relying only on committed chart assets. +- [ ] **R22 - Add installed-wheel smoke and resource tests.** Build and install the candidate wheel into an isolated environment, verify public imports and version metadata, and execute representative calculations that load every required packaged data family. Assert that tests, notebooks, provenance sources, and other non-runtime material follow the packaging policy established by R4-R8. +- [ ] **R23 - Add a downstream candidate-wheel compatibility gate.** After the active provenance upgrade has landed, install the exact candidate wheel into a pinned `digital-growth-charts-server` revision and run its import, calculation, dynamic chart-generation, and HTTP golden-response suites. Include the server's pinned React consumer compatibility matrix where available, and keep this as a release gate for changes to exports, calculations, chart structures, packaging, or serialized output. + +## High priority + +- [ ] **R1 - Resolve the broken public `create_thrive_lines` API.** `rcpchgrowth/dynamic_growth.py:226-294` uses undefined `plt`, has its output-producing plotting code commented out, and returns nothing, while `rcpchgrowth/__init__.py:8` exports it publicly. Decide whether plotting remains supported; either restore it with an optional matplotlib dependency and tests, or remove the export through an explicitly managed breaking change. +- [ ] **R2 - Restore executable notebooks and expose package version metadata consistently.** `Quickstart.ipynb`, `AdditionalFunctions.ipynb`, `ExperimentalFunctions.ipynb`, and `ResearchTemplate.ipynb` all fail in their first setup cell because `rcpchgrowth.__version__` does not exist. Expose the installed version using `importlib.metadata.version()` or update the notebooks not to expect `__version__`, then execute every notebook in CI or another repeatable check. + +## Medium priority + +- [ ] **R3 - Decide the future of the unusable `BoneAge` class.** `rcpchgrowth/bone_age.py:12-48` returns a dictionary from `__init__`, so construction raises `TypeError: __init__() should return None`. The class has no internal consumers and is not exported from the package root, although related fields are supported by `Measurement`. Either implement and test a coherent object API or remove the module after checking external compatibility. +- [ ] **R4 - Define an explicit runtime package-data allowlist.** The published installation occupies roughly 15 MB and ships spreadsheets, a PDF, VBA source, validation CSVs, old WHO tables, experimental LMS tables, and other provenance material because `pyproject.toml:68-69` broadly includes file extensions beneath `data_tables`. Identify the JSON and correlation resources required at runtime, package only those files, and verify wheel and sdist contents in CI. +- [ ] **R5 - Exclude the test package from release wheels.** `pyproject.toml:60-62` discovers `rcpchgrowth*`, which includes 19 modules beneath `rcpchgrowth/tests` in the published wheel. Exclude `rcpchgrowth.tests` and its subpackages while retaining all tests in the source repository and sdist if deliberately required there. +- [ ] **R6 - Separate runtime data from source and validation provenance.** Review `rcpchgrowth/data_tables/uk-who_resources/` (about 6.2 MB), `test_lms_cole_tmp/`, `validation_who_vs_uk_who/`, `cdc_resources/`, WHO CSV sources, `who/pre_2025/`, AAP spreadsheets, `GrowthCharts.xls`, `huiqi_cole_methods.bas`, and `data_tables.txt`. Retain required provenance in the repository or documentation project, but move it outside runtime package data and document what generated each shipped table. +- [ ] **R7 - Retire or archive the legacy `cdc.json` table.** `rcpchgrowth/data_tables/cdc.json` is approximately 308 KB and is shipped, but runtime code uses `cdc_infants.json` and `cdc2-20.json`. Confirm that no external resource consumer relies on its path, then remove it from package data or move it to a clearly documented provenance archive. +- [ ] **R8 - Decide whether height-prediction data is a supported feature.** The Bayley-Pinneau CSV files and `constants/height_predictions_constants.py` are shipped and indirectly exposed through wildcard constant imports, but no implementation or internal consumer exists. Either implement and test the feature or stop packaging and exporting these placeholders after a compatibility review. + +## Low priority + +- [ ] **R9 - Stop eagerly loading unused UK90 term data.** `rcpchgrowth/uk_who.py:38-42` loads `uk90_term.json` into `UK90_TERM_DATA`, but no repository consumer reads that global. Confirm that direct module consumers do not depend on it, then remove the load to reduce import-time work and memory use. +- [ ] **R10 - Remove or relocate unreachable and commented-out implementations.** Move or delete the strings after unconditional returns at `rcpchgrowth/chart_functions.py:267`, `377`, `468`, `584`, `714`, and `852`; remove the obsolete commented WHO calculation at `rcpchgrowth/global_functions.py:104-125`, the interpolation alternatives around lines `15-16` and `498-499`, and the commented plotting output at `rcpchgrowth/dynamic_growth.py:288-294`. Preserve meaningful rationale in maintained documentation or tests rather than commented executable code. +- [ ] **R11 - Remove confirmed unused imports and duplicate imports.** Clean `json` from `global_functions.py`; `CDC`, `HEIGHT`, `WEIGHT`, `rounded_sds_for_centile`, and duplicate `HEAD_CIRCUMFERENCE`/`BMI` imports from `centile_bands.py`; `FENTON` and `UK_WHO_CHILD` from `chart_functions.py`; `FEMALE`, `MALE`, and `z_score` from `dynamic_growth.py`; `scipy.stats.norm` from `test_centile_band_for_centile.py`; and `json`, `pprint`, and the local `math` import from `test_who.py`. Do not remove the root-package imports that intentionally define the public API. +- [ ] **R12 - Remove unused WHO precalculated centile arrays after source review.** The large arrays `WHO_GIRL_WEIGHT_UNDER_FIVE_15`, `WHO_GIRL_BMI_OVER_FIVE_15`, `WHO_BOY_LENGTH_OVER_FIVE_15`, `WHO_BOY_WEIGHT_OVER_FIVE_15`, and `WHO_BOY_BMI_OVER_FIVE_15` in `rcpchgrowth/tests/who_test_data/who_chart_precalculated_centiles.py` have no test consumers. Confirm that they are not intentionally retained source vectors before deleting them. +- [ ] **R13 - Make the public namespace explicit.** Add deliberate `__all__` declarations to `rcpchgrowth/__init__.py` and the constants package, then replace internal wildcard imports incrementally. Preserve the existing supported names unless a breaking API change is explicitly approved; the current Ruff reports for root re-exports are expected false positives rather than automatic deletion candidates. +- [ ] **R14 - Make release builds isolated and reproducible.** A local wheel built with an existing ignored `rcpchgrowth.egg-info/SOURCES.txt` contained `data_tables.txt`, while the published wheel did not, demonstrating that stale build metadata can affect package contents. Build releases from a clean source tree, inspect wheel and sdist manifests automatically, and ensure ignored metadata cannot alter artifacts. +- [ ] **R15 - Clear disposable generated artifacts from development worktrees.** The audited worktree contained ignored `build/`, `dist/`, `rcpchgrowth.egg-info/`, `.pytest_cache/`, `__pycache__/`, and notebook checkpoint directories, including an obsolete `2.1.9` distribution artifact. Provide or document a safe cleanup command that removes only ignored generated output and never reference or provenance data. + +## Dependency audit result + +No declared dependency was confirmed safe to remove during the audit. `scipy` supports runtime interpolation and statistical calculations; `python-dateutil` is imported as `dateutil`; pandas, matplotlib, JupyterLab, and ipykernel support notebooks; and pytest, build, and bump-my-version support testing and release workflows. Reassess these dependencies only alongside the feature or workflow that currently uses them. From 3e98b8d433dccf6d38e70ef67135fc11cfa9e51b Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Thu, 3 Sep 2026 22:50:27 +0100 Subject: [PATCH 02/11] test: characterize calculation and public contracts --- rcpchgrowth/tests/test_fixture_integrity.py | 48 +++ .../tests/test_measurement_contract.py | 326 ++++++++++++++++++ rcpchgrowth/tests/test_public_api_contract.py | 202 +++++++++++ .../test_reference_vectors_and_boundaries.py | 201 +++++++++++ .../tests/test_supported_chart_outputs.py | 174 ++++++++++ 5 files changed, 951 insertions(+) create mode 100644 rcpchgrowth/tests/test_fixture_integrity.py create mode 100644 rcpchgrowth/tests/test_measurement_contract.py create mode 100644 rcpchgrowth/tests/test_public_api_contract.py create mode 100644 rcpchgrowth/tests/test_reference_vectors_and_boundaries.py create mode 100644 rcpchgrowth/tests/test_supported_chart_outputs.py diff --git a/rcpchgrowth/tests/test_fixture_integrity.py b/rcpchgrowth/tests/test_fixture_integrity.py new file mode 100644 index 0000000..804b810 --- /dev/null +++ b/rcpchgrowth/tests/test_fixture_integrity.py @@ -0,0 +1,48 @@ +"""Integrity contract for the primary UK-WHO clinical validation fixture.""" + +import hashlib +import json +from pathlib import Path + +FIXTURE_PATH = Path(__file__).with_name("sds_age_validation_2021_refactored_2026.json") +DEPRECATED_FIXTURE_PATH = Path(__file__).with_name( + "sds_age_validation_2021_deprecated.json" +) +EXPECTED_SHA256 = "e269b3ff4312bca76563eb8adc7d0e9941b655077d11888b34522c3ffe2f6c84" +EXPECTED_DEPRECATED_SHA256 = ( + "f50c8b8757bd21da2108a07f91bb6855c8f39c1728cbe9eb3b5d210d7f7d3bac" +) +EXPECTED_FIELDS = { + "birth_date", + "chronological_age", + "chronological_sds", + "corrected_age", + "corrected_sds", + "gestation_days", + "gestation_weeks", + "measurement_method", + "observation_date", + "observation_value", + "sex", +} + + +def test_primary_uk_who_fixture_has_expected_count_and_checksum(): + fixture_bytes = FIXTURE_PATH.read_bytes() + cases = json.loads(fixture_bytes) + + assert len(cases) == 3984 + assert hashlib.sha256(fixture_bytes).hexdigest() == EXPECTED_SHA256 + + +def test_every_primary_uk_who_fixture_case_has_the_complete_schema(): + cases = json.loads(FIXTURE_PATH.read_bytes()) + + assert all(set(case) == EXPECTED_FIELDS for case in cases) + + +def test_deprecated_uk_who_fixture_is_preserved_unchanged(): + fixture_bytes = DEPRECATED_FIXTURE_PATH.read_bytes() + + assert len(json.loads(fixture_bytes)) == 4002 + assert hashlib.sha256(fixture_bytes).hexdigest() == EXPECTED_DEPRECATED_SHA256 diff --git a/rcpchgrowth/tests/test_measurement_contract.py b/rcpchgrowth/tests/test_measurement_contract.py new file mode 100644 index 0000000..f4d331e --- /dev/null +++ b/rcpchgrowth/tests/test_measurement_contract.py @@ -0,0 +1,326 @@ +"""Semantic contract tests for the complete ``Measurement.measurement`` result.""" + +from datetime import date +from numbers import Real + +import pytest + +from rcpchgrowth import Measurement + +TOP_LEVEL_KEYS = { + "birth_data", + "bone_age", + "child_observation_value", + "events_data", + "measurement_calculated_values", + "measurement_dates", + "plottable_data", + "provenance", +} +BIRTH_DATA_KEYS = { + "birth_date", + "estimated_date_delivery", + "estimated_date_delivery_string", + "gestation_days", + "gestation_weeks", + "sex", +} +MEASUREMENT_DATE_KEYS = { + "observation_date", + "chronological_decimal_age", + "corrected_decimal_age", + "chronological_calendar_age", + "corrected_calendar_age", + "corrected_gestational_age", + "comments", + "corrected_decimal_age_error", + "chronological_decimal_age_error", +} +COMMENT_KEYS = { + "clinician_corrected_decimal_age_comment", + "lay_corrected_decimal_age_comment", + "clinician_chronological_decimal_age_comment", + "lay_chronological_decimal_age_comment", +} +CALCULATED_VALUE_KEYS = { + "corrected_sds", + "corrected_centile", + "corrected_centile_band", + "chronological_sds", + "chronological_centile", + "chronological_centile_band", + "corrected_measurement_error", + "chronological_measurement_error", + "corrected_percentage_median_bmi", + "chronological_percentage_median_bmi", +} +BONE_AGE_KEYS = { + "bone_age", + "bone_age_type", + "bone_age_sds", + "bone_age_centile", + "bone_age_text", +} +CENTILE_POINT_KEYS = { + "x", + "y", + "b", + "centile", + "sds", + "events_text", + "bone_age_label", + "bone_age_type", + "bone_age_sds", + "bone_age_centile", + "observation_error", + "age_type", + "calendar_age", + "lay_comment", + "clinician_comment", + "age_error", + "centile_band", + "observation_value_error", +} +SDS_POINT_KEYS = CENTILE_POINT_KEYS - {"sds", "observation_error"} + + +def make_measurement(**overrides): + values = { + "birth_date": date(2020, 1, 1), + "observation_date": date(2021, 1, 1), + "measurement_method": "height", + "observation_value": 75.0, + "reference": "uk-who", + "sex": "female", + } + values.update(overrides) + return Measurement(**values).measurement + + +def assert_complete_measurement_shape(measurement): + assert set(measurement) == TOP_LEVEL_KEYS + assert set(measurement["provenance"]) == { + "growth_reference", + "calculation_engine", + } + assert set(measurement["provenance"]["calculation_engine"]) == { + "name", + "version", + "commit", + } + assert set(measurement["birth_data"]) == BIRTH_DATA_KEYS + assert set(measurement["measurement_dates"]) == MEASUREMENT_DATE_KEYS + assert set(measurement["measurement_dates"]["corrected_gestational_age"]) == { + "corrected_gestation_weeks", + "corrected_gestation_days", + } + assert set(measurement["measurement_dates"]["comments"]) == COMMENT_KEYS + assert set(measurement["child_observation_value"]) == { + "measurement_method", + "observation_value", + "observation_value_error", + } + assert set(measurement["measurement_calculated_values"]) == CALCULATED_VALUE_KEYS + assert set(measurement["plottable_data"]) == {"centile_data", "sds_data"} + assert set(measurement["bone_age"]) == BONE_AGE_KEYS + assert set(measurement["events_data"]) == {"events_text"} + + for data_kind, point_keys in ( + ("centile_data", CENTILE_POINT_KEYS), + ("sds_data", SDS_POINT_KEYS), + ): + plot_data = measurement["plottable_data"][data_kind] + assert set(plot_data) == { + "chronological_decimal_age_data", + "corrected_decimal_age_data", + } + assert set(plot_data["chronological_decimal_age_data"]) == point_keys + assert set(plot_data["corrected_decimal_age_data"]) == point_keys | { + "corrected_gestational_age" + } + + +def test_successful_measurement_has_complete_typed_structure_and_values(): + measurement = make_measurement( + events_text=["Growth review"], + bone_age=1.2, + bone_age_type="greulich-pyle", + bone_age_sds=0.3, + bone_age_centile=61.8, + bone_age_text="Concordant bone age", + ) + + assert_complete_measurement_shape(measurement) + assert measurement["provenance"]["growth_reference"] == "uk-who" + assert ( + measurement["provenance"]["calculation_engine"]["name"] + == "rcpch/rcpchgrowth-python" + ) + assert all( + isinstance(measurement["provenance"]["calculation_engine"][key], str) + and measurement["provenance"]["calculation_engine"][key] + for key in ("version", "commit") + ) + assert measurement["birth_data"] == { + "birth_date": date(2020, 1, 1), + "gestation_weeks": 40, + "gestation_days": 0, + "estimated_date_delivery": date(2020, 1, 1), + "estimated_date_delivery_string": "Wed 01 January, 2020", + "sex": "female", + } + dates = measurement["measurement_dates"] + assert isinstance(dates["chronological_decimal_age"], float) + assert dates["chronological_decimal_age"] == pytest.approx(1.0020533881) + assert dates["corrected_decimal_age"] == dates["chronological_decimal_age"] + assert dates["corrected_decimal_age_error"] is None + assert dates["chronological_decimal_age_error"] is None + + calculated = measurement["measurement_calculated_values"] + assert calculated["corrected_sds"] == pytest.approx(0.3706868165) + assert calculated["chronological_sds"] == pytest.approx(0.3706868165) + assert calculated["corrected_centile"] == pytest.approx(64.4564594609) + assert calculated["chronological_centile"] == pytest.approx(64.4564594609) + assert all( + isinstance(calculated[key], Real) + for key in ( + "corrected_sds", + "chronological_sds", + "corrected_centile", + "chronological_centile", + ) + ) + assert calculated["corrected_measurement_error"] is None + assert calculated["chronological_measurement_error"] is None + assert calculated["corrected_percentage_median_bmi"] is None + assert calculated["chronological_percentage_median_bmi"] is None + assert measurement["bone_age"] == { + "bone_age": 1.2, + "bone_age_type": "greulich-pyle", + "bone_age_sds": 0.3, + "bone_age_centile": 61.8, + "bone_age_text": "Concordant bone age", + } + assert measurement["events_data"] == {"events_text": ["Growth review"]} + + centile_data = measurement["plottable_data"]["centile_data"] + sds_data = measurement["plottable_data"]["sds_data"] + for age_type, expected_age_type in ( + ("chronological_decimal_age_data", "chronological_age"), + ("corrected_decimal_age_data", "corrected_age"), + ): + assert centile_data[age_type]["age_type"] == expected_age_type + assert centile_data[age_type]["y"] == 75.0 + assert centile_data[age_type]["b"] == 1.2 + assert centile_data[age_type]["events_text"] == ["Growth review"] + assert sds_data[age_type]["y"] == pytest.approx(0.3706868165) + + +def test_preterm_measurement_distinguishes_corrected_and_chronological_results(): + measurement = make_measurement( + observation_date=date(2020, 7, 1), + measurement_method="weight", + observation_value=6.5, + sex="male", + gestation_weeks=32, + events_text=["review"], + ) + + assert_complete_measurement_shape(measurement) + dates = measurement["measurement_dates"] + assert dates["chronological_decimal_age"] == pytest.approx(0.4982888433) + assert dates["corrected_decimal_age"] == pytest.approx(0.3449691992) + assert dates["corrected_calendar_age"] == "4 months and 5 days" + assert "32+0 weeks" in dates["comments"]["lay_corrected_decimal_age_comment"] + + calculated = measurement["measurement_calculated_values"] + assert calculated["chronological_sds"] == pytest.approx(-1.7874654372) + assert calculated["corrected_sds"] == pytest.approx(-0.7524570442) + assert calculated["chronological_centile"] == pytest.approx(3.6931146757) + assert calculated["corrected_centile"] == pytest.approx(22.5888126476) + assert measurement["events_data"]["events_text"] == ["review"] + + +def test_out_of_range_measurement_retains_shape_and_reports_both_age_errors(): + measurement = make_measurement( + birth_date=date(2000, 1, 1), + observation_date=date(2025, 1, 1), + observation_value=170.0, + ) + + assert_complete_measurement_shape(measurement) + expected_error = "There is no UK90 reference data above the age of 20 years." + assert ( + measurement["child_observation_value"]["observation_value_error"] + == expected_error + ) + calculated = measurement["measurement_calculated_values"] + assert calculated["corrected_measurement_error"] == expected_error + assert calculated["chronological_measurement_error"] == expected_error + assert all( + calculated[key] is None + for key in ( + "corrected_sds", + "corrected_centile", + "corrected_centile_band", + "chronological_sds", + "chronological_centile", + "chronological_centile_band", + ) + ) + centile_data = measurement["plottable_data"]["centile_data"] + assert ( + centile_data["corrected_decimal_age_data"]["observation_error"] + == expected_error + ) + assert ( + centile_data["chronological_decimal_age_data"]["observation_error"] + == expected_error + ) + + +def test_invalid_observation_is_flagged_but_current_calculations_are_retained(): + measurement = make_measurement(observation_value=1.0) + + assert_complete_measurement_shape(measurement) + expected_error = "Height/length must be passed in cm, not metres" + assert ( + measurement["child_observation_value"]["observation_value_error"] + == expected_error + ) + calculated = measurement["measurement_calculated_values"] + assert calculated["corrected_sds"] == pytest.approx(-28.3556975398) + assert calculated["chronological_sds"] == pytest.approx(-28.3556975398) + assert calculated["corrected_measurement_error"] is None + assert calculated["chronological_measurement_error"] is None + centile_data = measurement["plottable_data"]["centile_data"] + assert ( + centile_data["corrected_decimal_age_data"]["observation_error"] + == expected_error + ) + assert ( + centile_data["chronological_decimal_age_data"]["observation_error"] + == expected_error + ) + + +def test_invalid_dates_retain_complete_shape_and_suppress_calculations(): + measurement = make_measurement( + observation_date=date(2019, 1, 1), observation_value=50.0 + ) + + assert_complete_measurement_shape(measurement) + dates = measurement["measurement_dates"] + assert dates["corrected_decimal_age"] is None + assert dates["corrected_decimal_age_error"] == ( + "Birth date cannot be after the date of observation." + ) + calculated = measurement["measurement_calculated_values"] + assert calculated["corrected_measurement_error"] == ( + "Dates error. Calculations impossible." + ) + assert calculated["chronological_measurement_error"] == ( + "Dates error. Calculations impossible." + ) + assert calculated["corrected_sds"] is None + assert calculated["chronological_sds"] is None diff --git a/rcpchgrowth/tests/test_public_api_contract.py b/rcpchgrowth/tests/test_public_api_contract.py new file mode 100644 index 0000000..b0efddb --- /dev/null +++ b/rcpchgrowth/tests/test_public_api_contract.py @@ -0,0 +1,202 @@ +"""Public import and call contracts used by digital-growth-charts-server.""" + +from datetime import date +from importlib import import_module +from inspect import Parameter, signature + +import pytest + +import rcpchgrowth +from rcpchgrowth import chart_functions, constants + +PUBLIC_SYMBOL_MODULES = { + "Measurement": "rcpchgrowth.measurement", + "centile": "rcpchgrowth.global_functions", + "chronological_calendar_age": "rcpchgrowth.date_calculations", + "corrected_decimal_age": "rcpchgrowth.date_calculations", + "create_chart": "rcpchgrowth.chart_functions", + "generate_fictional_child_data": "rcpchgrowth.fictional_child", + "lower_and_upper_limits_of_expected_height_z": "rcpchgrowth.mid_parental_height", + "measurement_from_sds": "rcpchgrowth.global_functions", + "mid_parental_height_z": "rcpchgrowth.mid_parental_height", + "sds_for_measurement": "rcpchgrowth.global_functions", +} + +PARAMETER_CONTRACTS = { + "Measurement": ( + "birth_date", + "measurement_method", + "observation_date", + "observation_value", + "reference", + "sex", + "gestation_days", + "gestation_weeks", + "events_text", + "bone_age", + "bone_age_type", + "bone_age_sds", + "bone_age_centile", + "bone_age_text", + ), + "centile": ("z_score",), + "chronological_calendar_age": ("birth_date", "observation_date"), + "corrected_decimal_age": ( + "birth_date", + "observation_date", + "gestation_weeks", + "gestation_days", + ), + "create_chart": ( + "reference", + "centile_format", + "measurement_method", + "sex", + "is_sds", + ), + "generate_fictional_child_data": ( + "measurement_method", + "sex", + "start_chronological_age", + "end_age", + "gestation_weeks", + "gestation_days", + "measurement_interval_type", + "measurement_interval_number", + "start_sds", + "drift", + "drift_range", + "noise", + "noise_range", + "reference", + "start_chronological_age_interval_type", + "end_age_interval_type", + ), + "lower_and_upper_limits_of_expected_height_z": ("mid_parental_height_z",), + "measurement_from_sds": ( + "reference", + "requested_sds", + "measurement_method", + "sex", + "age", + "default_youngest_reference", + ), + "mid_parental_height_z": ("maternal_height", "paternal_height", "reference"), + "sds_for_measurement": ( + "reference", + "age", + "measurement_method", + "observation_value", + "sex", + ), +} + +DEFAULT_CONTRACTS = { + "Measurement": { + "gestation_days": 0, + "gestation_weeks": 0, + "events_text": None, + "bone_age": None, + "bone_age_type": None, + "bone_age_sds": None, + "bone_age_centile": None, + "bone_age_text": None, + }, + "centile": {}, + "chronological_calendar_age": {}, + "corrected_decimal_age": {}, + "create_chart": { + "centile_format": "cole-nine-centiles", + "measurement_method": "height", + "sex": "female", + "is_sds": False, + }, + "generate_fictional_child_data": { + "start_chronological_age": 0.0, + "end_age": 20.0, + "gestation_weeks": 40, + "gestation_days": 0, + "measurement_interval_type": "days", + "measurement_interval_number": 20, + "start_sds": 0, + "drift": False, + "drift_range": -0.05, + "noise": False, + "noise_range": 0.01, + "reference": "uk-who", + "start_chronological_age_interval_type": "years", + "end_age_interval_type": "years", + }, + "lower_and_upper_limits_of_expected_height_z": {}, + "measurement_from_sds": {"default_youngest_reference": False}, + "mid_parental_height_z": {"reference": "uk-who"}, + "sds_for_measurement": {}, +} + +SERVER_MODULE_IMPORTS = ( + "rcpchgrowth.chart_functions", + "rcpchgrowth.constants", + "rcpchgrowth.constants.reference_constants", + "rcpchgrowth.constants.validation_constants", + "rcpchgrowth.date_calculations", + "rcpchgrowth.global_functions", +) + + +@pytest.mark.parametrize("name,module_name", PUBLIC_SYMBOL_MODULES.items()) +def test_server_root_imports_resolve_to_their_defining_modules(name, module_name): + assert getattr(rcpchgrowth, name) is getattr(import_module(module_name), name) + + +@pytest.mark.parametrize("name,parameter_names", PARAMETER_CONTRACTS.items()) +def test_server_callable_parameter_names_and_order_are_stable(name, parameter_names): + assert tuple(signature(getattr(rcpchgrowth, name)).parameters) == parameter_names + + +@pytest.mark.parametrize("name,defaults", DEFAULT_CONTRACTS.items()) +def test_server_callable_defaults_are_stable(name, defaults): + parameters = signature(getattr(rcpchgrowth, name)).parameters + + assert { + name: parameter.default + for name, parameter in parameters.items() + if parameter.default is not Parameter.empty + } == defaults + + +def test_server_module_imports_are_available_from_the_package_root(): + assert chart_functions is import_module("rcpchgrowth.chart_functions") + assert constants is import_module("rcpchgrowth.constants") + + +@pytest.mark.parametrize("module_name", SERVER_MODULE_IMPORTS) +def test_server_direct_module_imports_remain_available(module_name): + assert import_module(module_name).__name__ == module_name + + +def test_server_constant_imports_retain_their_public_values(): + expected = { + "CDC": "cdc", + "MAXIMUM_BMI_ERROR_SDS": 15, + "MAXIMUM_HEIGHT_WEIGHT_OFC_ERROR_SDS": 8, + "MINIMUM_BMI_ERROR_SDS": -15, + "MINIMUM_HEIGHT_WEIGHT_OFC_ERROR_SDS": -8, + "TRISOMY_21": "trisomy-21", + "TRISOMY_21_AAP": "trisomy-21-aap", + "TURNERS": "turners-syndrome", + "UK_WHO": "uk-who", + "WHO": "who", + } + + assert {name: getattr(rcpchgrowth, name) for name in expected} == expected + + +def test_representative_root_import_invocations_remain_usable(): + assert rcpchgrowth.centile(0.0) == pytest.approx(50.0) + assert ( + rcpchgrowth.chronological_calendar_age(date(2020, 1, 1), date(2021, 1, 1)) + == "1 year" + ) + assert rcpchgrowth.lower_and_upper_limits_of_expected_height_z( + 0.5 + ) == pytest.approx((-0.9, 1.9)) diff --git a/rcpchgrowth/tests/test_reference_vectors_and_boundaries.py b/rcpchgrowth/tests/test_reference_vectors_and_boundaries.py new file mode 100644 index 0000000..7ba4c1c --- /dev/null +++ b/rcpchgrowth/tests/test_reference_vectors_and_boundaries.py @@ -0,0 +1,201 @@ +"""Characterization vectors and boundaries for the supported references.""" + +import pytest + +from rcpchgrowth import cdc, trisomy_21_aap, uk_who, who +from rcpchgrowth.constants import ( + BMI, + CDC, + FEMALE, + FORTY_TWO_WEEKS_GESTATION, + HEAD_CIRCUMFERENCE, + HEIGHT, + MALE, + TRISOMY_21, + TRISOMY_21_AAP, + TURNERS, + UK_WHO, + WEIGHT, + WHO, +) +from rcpchgrowth.constants.reference_constants import WHO_2006_REFERENCE_UPPER_THRESHOLD +from rcpchgrowth.global_functions import measurement_from_sds, sds_for_measurement + + +@pytest.mark.parametrize( + ("reference", "age", "method", "sex", "sds", "expected"), + [ + pytest.param(TURNERS, 10, HEIGHT, FEMALE, 0, 119.45, id="turner-height"), + pytest.param(TRISOMY_21, 0, HEIGHT, MALE, 0, 49.50236, id="trisomy-21-height"), + pytest.param( + TRISOMY_21_AAP, + 0, + WEIGHT, + MALE, + 0, + 3.177, + id="trisomy-21-aap-weight", + ), + pytest.param( + CDC, + 0, + HEAD_CIRCUMFERENCE, + FEMALE, + 0, + 34.7115617, + id="cdc-infant-ofc", + ), + pytest.param( + CDC, + 2, + BMI, + MALE, + 2.326347874, + 21.10122398, + id="cdc-extended-bmi-p99", + ), + ], +) +def test_published_reference_vectors(reference, age, method, sex, sds, expected): + """Values are literal LMS-table vectors from Lyon et al. (1985), Styles et al. (ADC 2002), Zemel et al. (Pediatrics 2015), and the CDC/NCHS tables. CDC sources: https://www.cdc.gov/growthcharts/cdc-data-files.htm and https://www.cdc.gov/growthcharts/extended-bmi-data-files.htm. The p99 value is independently evaluated from the age-24-month male LMS row and sigma using the published extended-BMI equation.""" + assert measurement_from_sds(reference, sds, method, sex, age) == pytest.approx( + expected, abs=1e-4 + ) + assert sds_for_measurement(reference, age, method, expected, sex) == pytest.approx( + sds, abs=1e-4 + ) + + +@pytest.mark.parametrize( + ("reference", "valid_age", "invalid_age", "method", "sex", "message"), + [ + (UK_WHO, 20, 20.0001, HEIGHT, MALE, "above the age of 20 years"), + (WHO, 19, 19.0001, HEIGHT, MALE, "above 19 years"), + (CDC, 20, 20.0001, HEIGHT, MALE, "above the age of 20 years"), + (TURNERS, 20, 20.0001, HEIGHT, FEMALE, "above 20 years"), + (TRISOMY_21, 20, 20.0001, HEIGHT, MALE, "over the age of 20y"), + (TRISOMY_21_AAP, 20, 20.0001, HEIGHT, MALE, "over the age of 20y"), + ], +) +def test_every_reference_includes_its_upper_boundary_and_rejects_above_it( + reference, valid_age, invalid_age, method, sex, message +): + assert measurement_from_sds(reference, 0, method, sex, valid_age) is not None + with pytest.raises(LookupError, match=message): + measurement_from_sds(reference, 0, method, sex, invalid_age) + + +@pytest.mark.parametrize( + ("reference", "valid_age", "invalid_age", "method", "sex", "message"), + [ + (UK_WHO, 0, -1, WEIGHT, MALE, "below 23 weeks gestation"), + (WHO, 0, -0.0001, HEIGHT, MALE, "below term"), + (CDC, 0, -0.0001, HEIGHT, MALE, "below 40 weeks"), + (TURNERS, 1, 0.9999, HEIGHT, FEMALE, "below 1 year"), + (TRISOMY_21, 0, -0.0001, HEIGHT, MALE, "below 40 weeks"), + (TRISOMY_21_AAP, 0, -0.0001, WEIGHT, MALE, "below 40 weeks"), + ], +) +def test_every_reference_includes_its_lower_boundary_and_rejects_below_it( + reference, valid_age, invalid_age, method, sex, message +): + assert measurement_from_sds(reference, 0, method, sex, valid_age) is not None + with pytest.raises(LookupError, match=message): + measurement_from_sds(reference, 0, method, sex, invalid_age) + + +@pytest.mark.parametrize( + ("reference", "valid_age", "invalid_age", "method", "sex", "message"), + [ + ( + UK_WHO, + FORTY_TWO_WEEKS_GESTATION, + FORTY_TWO_WEEKS_GESTATION - 0.0001, + BMI, + MALE, + "below 2 weeks", + ), + (UK_WHO, 17, 17.0001, HEAD_CIRCUMFERENCE, FEMALE, "girls over 17"), + (WHO, 10, 10.0001, WEIGHT, MALE, "weight data"), + ( + WHO, + WHO_2006_REFERENCE_UPPER_THRESHOLD, + WHO_2006_REFERENCE_UPPER_THRESHOLD + 0.0001, + HEAD_CIRCUMFERENCE, + MALE, + "head circumference", + ), + (CDC, 2, 1.9999, BMI, MALE, "BMI below 2 years"), + (CDC, 3, 3.0001, HEAD_CIRCUMFERENCE, MALE, "head circumference above 3 years"), + (TRISOMY_21, 18.82, 18.8201, BMI, MALE, "BMI reference data"), + (TRISOMY_21, 18, 18.0001, HEAD_CIRCUMFERENCE, MALE, "head circumference"), + (TRISOMY_21_AAP, 0.083333333, 0.0829, HEIGHT, MALE, "below 1 month"), + (TRISOMY_21_AAP, 2, 1.9999, BMI, MALE, "below 2 years"), + ], +) +def test_measurement_specific_boundaries( + reference, valid_age, invalid_age, method, sex, message +): + assert measurement_from_sds(reference, 0, method, sex, valid_age) is not None + with pytest.raises(LookupError, match=message): + measurement_from_sds(reference, 0, method, sex, invalid_age) + + +@pytest.mark.parametrize( + ("method", "sex", "message"), + [ + (WEIGHT, FEMALE, "no reference data for weight"), + (HEIGHT, MALE, "only affects girls"), + ], +) +def test_turner_rejects_unsupported_measurements_and_sex(method, sex, message): + with pytest.raises(LookupError, match=message): + measurement_from_sds(TURNERS, 0, method, sex, 10) + + +def test_aap_declared_one_month_cutoff_precedes_first_height_lms_row(): + """The current 0.083 cutoff admits ages below the first 0.083333333 source row, where interpolation leaks ValueError rather than a domain LookupError.""" + invalid, _ = trisomy_21_aap.reference_data_absent(0.083, HEIGHT, MALE) + assert invalid is False + with pytest.raises(ValueError): + measurement_from_sds(TRISOMY_21_AAP, 0, HEIGHT, MALE, 0.083) + + +def test_uk_who_exact_handover_ownership(): + assert uk_who.uk_who_reference(FORTY_TWO_WEEKS_GESTATION) is uk_who.WHO_INFANTS_DATA + assert ( + uk_who.uk_who_reference(FORTY_TWO_WEEKS_GESTATION, True) + is uk_who.UK90_PRETERM_DATA + ) + assert uk_who.uk_who_reference(2) is uk_who.WHO_CHILD_DATA + assert uk_who.uk_who_reference(2, True) is uk_who.WHO_INFANTS_DATA + assert uk_who.uk_who_reference(4) is uk_who.UK90_CHILD_DATA + assert uk_who.uk_who_reference(4, True) is uk_who.WHO_CHILD_DATA + + +def test_who_exact_handover_ownership(): + assert who.who_reference(2) is who.WHO_CHILD_DATA + assert who.who_reference(2, True) is who.WHO_INFANTS_DATA + assert who.who_reference(WHO_2006_REFERENCE_UPPER_THRESHOLD) is who.WHO_CHILD_DATA + assert ( + who.who_reference(WHO_2006_REFERENCE_UPPER_THRESHOLD + 0.0001) + is who.WHO_2007_DATA + ) + + +def test_cdc_exact_handover_and_ofc_ownership(): + assert cdc.cdc_reference(2, HEIGHT) is cdc.CDC_CHILD_DATA + assert cdc.cdc_reference(2, HEIGHT, True) is cdc.WHO_INFANTS_DATA + assert cdc.cdc_reference(3, HEAD_CIRCUMFERENCE) is cdc.CDC_INFANT_DATA + assert cdc.cdc_reference(3.0001, HEIGHT) is cdc.CDC_CHILD_DATA + + +def test_aap_age_three_overlap_is_measurement_specific(): + """At age three AAP height/weight stay infant-owned, OFC can select either table, and BMI is child-only.""" + select = trisomy_21_aap.trisomy_21_aap_lms_array_for_measurement_and_sex + infant = trisomy_21_aap.TRISOMY_21_AAP_INFANT_DATA["measurement"] + child = trisomy_21_aap.TRISOMY_21_AAP_CHILD_DATA["measurement"] + assert select(HEIGHT, MALE, 3) is infant[HEIGHT][MALE] + assert select(HEAD_CIRCUMFERENCE, MALE, 3) is child[HEAD_CIRCUMFERENCE][MALE] + assert select(HEAD_CIRCUMFERENCE, MALE, 3, True) is infant[HEAD_CIRCUMFERENCE][MALE] + assert select(BMI, MALE, 3) is child[BMI][MALE] diff --git a/rcpchgrowth/tests/test_supported_chart_outputs.py b/rcpchgrowth/tests/test_supported_chart_outputs.py new file mode 100644 index 0000000..cafa1a2 --- /dev/null +++ b/rcpchgrowth/tests/test_supported_chart_outputs.py @@ -0,0 +1,174 @@ +"""Direct characterization of dynamic chart output for every public reference.""" + +import pytest + +from rcpchgrowth.chart_functions import create_chart +from rcpchgrowth.constants import ( + BMI, + CDC, + FEMALE, + HEIGHT, + MALE, + REFERENCES, + TRISOMY_21, + TRISOMY_21_AAP, + TURNERS, + UK_WHO, + WHO, +) + +CHART_CASES = [ + pytest.param( + UK_WHO, + HEIGHT, + MALE, + [ + # The nominal 25-week grid value is fractionally below the exact + # validation threshold, so the first emitted height point is 26 weeks. + ("uk90_preterm", -0.2683, 0.0383), + ("uk_who_infant", 0.0383, 2), + ("uk_who_child", 2, 4), + ("uk90_child", 4, 20), + ], + id="uk-who", + ), + pytest.param( + WHO, + HEIGHT, + MALE, + [ + ("who_2006_infant", 0.0383, 2), + ("who_2006_child", 2, 5), + ("who_2007_child", 5, 19), + ], + id="who", + ), + pytest.param( + CDC, + BMI, + MALE, + [ + ("fenton", None, None), + ("cdc_infant", None, None), + ("cdc_child", 2, 20), + ], + id="cdc-bmi", + ), + pytest.param( + TURNERS, + HEIGHT, + FEMALE, + [(TURNERS, 1, 20)], + id="turner", + ), + pytest.param( + TRISOMY_21, + HEIGHT, + MALE, + [(TRISOMY_21, 0, 20)], + id="trisomy-21", + ), + pytest.param( + TRISOMY_21_AAP, + HEIGHT, + MALE, + [ + ("trisomy_21_aap_infant", 0.0833, 3), + ("trisomy_21_aap_child", 3, 20), + ], + id="trisomy-21-aap", + ), +] + + +def test_every_public_reference_has_a_dynamic_chart_contract(): + assert {case.values[0] for case in CHART_CASES} == set(REFERENCES) + + +@pytest.mark.parametrize(("reference", "method", "sex", "components"), CHART_CASES) +def test_chart_structure_series_and_age_domains(reference, method, sex, components): + chart = create_chart(reference, [50], method, sex) + + assert [next(iter(component)) for component in chart] == [ + name for name, _, _ in components + ] + for component, (name, first_age, last_age) in zip( + chart, components, strict=True + ): + series = component[name][sex] + assert list(series) == [method] + assert len(series[method]) == 1 + centile = series[method][0] + assert centile["sds"] == 0 + assert centile["centile"] == 50 + + points = centile["data"] + if first_age is None: + # Fenton is unavailable and CDC has no infant BMI table. The builder + # intentionally preserves both component slots as empty series. + assert points == [] + continue + + assert points[0]["x"] == first_age + assert points[-1]["x"] == last_age + assert all(set(point) == {"l", "x", "y"} for point in points) + assert all(point["l"] == 50 for point in points) + assert all(point["y"] is not None for point in points) + + +@pytest.mark.parametrize( + ("reference", "method", "sex", "component", "age", "expected"), + [ + pytest.param(UK_WHO, HEIGHT, MALE, "uk90_child", 4, 102.49, id="uk-who"), + # Age two is clamped to the infant table's authoritative day-730 row. + pytest.param(WHO, HEIGHT, MALE, "who_2006_infant", 2, 87.8018, id="who"), + pytest.param(TURNERS, HEIGHT, FEMALE, TURNERS, 10, 119.45, id="turner"), + pytest.param( + TRISOMY_21, + HEIGHT, + MALE, + TRISOMY_21, + 0, + 49.50236, + id="trisomy-21", + ), + pytest.param( + TRISOMY_21_AAP, + HEIGHT, + MALE, + "trisomy_21_aap_infant", + 0.0833, + 52.632, + id="trisomy-21-aap-infant", + ), + pytest.param( + TRISOMY_21_AAP, + HEIGHT, + MALE, + "trisomy_21_aap_child", + 3, + 87.222, + id="trisomy-21-aap-child", + ), + pytest.param(CDC, BMI, MALE, "cdc_child", 2, 16.57502768, id="cdc"), + ], +) +def test_chart_representative_source_table_coordinates( + reference, method, sex, component, age, expected +): + """Median coordinates are literal rows from the UK90, WHO, Lyon 1985, Styles 2002, Zemel 2015, and CDC LMS source tables.""" + chart = create_chart(reference, [50], method, sex) + points = next(item[component] for item in chart if component in item)[sex][method][ + 0 + ]["data"] + point = next(point for point in points if point["x"] == age) + assert point["y"] == pytest.approx(expected, abs=1e-4) + + +def test_aap_chart_includes_both_age_three_height_coordinates(): + """Both AAP component slots plot age three through the infant-owned calculation path; the source tables happen to have the same median there.""" + chart = create_chart(TRISOMY_21_AAP, [50], HEIGHT, MALE) + infant = chart[0]["trisomy_21_aap_infant"][MALE][HEIGHT][0]["data"] + child = chart[1]["trisomy_21_aap_child"][MALE][HEIGHT][0]["data"] + assert infant[-1]["x"] == child[0]["x"] == 3 + assert infant[-1]["y"] == child[0]["y"] == pytest.approx(87.222) From 323f7b774cf439c3e4658cc2928399b0b202b744 Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Thu, 3 Sep 2026 22:50:52 +0100 Subject: [PATCH 03/11] ci: add lint and installed-wheel quality gates --- .../run-pytest-on-push-and-all-prs.yml | 24 +++++++ pyproject.toml | 17 +++++ s/lint | 31 +++++++++ s/test-wheel | 41 ++++++++++++ tests/distribution/check-wheel.py | 62 +++++++++++++++++ tests/distribution/installed-wheel-smoke.py | 67 +++++++++++++++++++ 6 files changed, 242 insertions(+) create mode 100755 s/lint create mode 100755 s/test-wheel create mode 100755 tests/distribution/check-wheel.py create mode 100755 tests/distribution/installed-wheel-smoke.py diff --git a/.github/workflows/run-pytest-on-push-and-all-prs.yml b/.github/workflows/run-pytest-on-push-and-all-prs.yml index f5dd01e..6e792d5 100644 --- a/.github/workflows/run-pytest-on-push-and-all-prs.yml +++ b/.github/workflows/run-pytest-on-push-and-all-prs.yml @@ -26,6 +26,9 @@ on: - staging - live +permissions: + contents: read + jobs: test: runs-on: ubuntu-latest @@ -36,6 +39,8 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 @@ -57,3 +62,22 @@ jobs: - name: Run pytest run: pytest -q + + - name: Run adopted Ruff checks + if: matrix.python-version == '3.13' + run: python -m ruff check . + + - name: Build and test the installed wheel + run: | + wheel_dir="$(mktemp -d)" + python -m build --wheel --outdir "$wheel_dir" + wheels=("$wheel_dir"/*.whl) + test "${#wheels[@]}" -eq 1 + wheel_path="${wheels[0]}" + python tests/distribution/check-wheel.py "$wheel_path" + python -m venv --system-site-packages "$wheel_dir/venv" + "$wheel_dir/venv/bin/python" -m pip install --force-reinstall --no-deps "$wheel_path" + ( + cd "$wheel_dir" + env -u PYTHONPATH "$wheel_dir/venv/bin/python" "$GITHUB_WORKSPACE/tests/distribution/installed-wheel-smoke.py" + ) diff --git a/pyproject.toml b/pyproject.toml index 4ce5ce7..5f20c2d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -52,8 +52,25 @@ dev = [ "pandas>=1.5", "bump-my-version", "build", + "ruff>=0.16.5", ] +[tool.ruff] +target-version = "py310" +extend-exclude = [ + "rcpchgrowth/tests/who_test_data/who_chart_precalculated_centiles.py", +] + +[tool.ruff.lint] +# Start with correctness rules that the existing tree passes. Broader rule +# families remain available as an audit and can be adopted in reviewed groups. +select = ["E9", "F63", "F7", "F82"] + +[tool.ruff.lint.per-file-ignores] +# R1 tracks the broken public plotting function; quarantine that known defect +# without allowing it to hide new undefined names elsewhere. +"rcpchgrowth/dynamic_growth.py" = ["F821"] + [tool.setuptools] include-package-data = true diff --git a/s/lint b/s/lint new file mode 100755 index 0000000..b751680 --- /dev/null +++ b/s/lint @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Run the adopted Ruff checks in the project container. Use --audit to report +# the broader candidate rule set without turning inherited findings into a +# blocking failure. Run `s/rebuild` after changing development dependencies. + +set -euo pipefail +cd "$(git rev-parse --show-toplevel)" + +use_running=false +audit=false +if [[ "${1:-}" == "--running" || "${1:-}" == "-r" ]]; then + use_running=true + shift +fi +if [[ "${1:-}" == "--audit" ]]; then + audit=true + shift +fi + +if [[ "$use_running" == false ]]; then + docker compose up -d +fi +if ! docker compose exec -T rcpchgrowth-python python -m ruff --version >/dev/null 2>&1; then + echo "Ruff is not installed in the development container. Run: s/rebuild" >&2 + exit 1 +fi + +if [[ "$audit" == true ]]; then + exec docker compose exec -T rcpchgrowth-python python -m ruff check --select E4,E7,E9,F,I,B,UP --statistics --exit-zero "$@" . +fi +exec docker compose exec -T rcpchgrowth-python python -m ruff check "$@" . diff --git a/s/test-wheel b/s/test-wheel new file mode 100755 index 0000000..3aa65e3 --- /dev/null +++ b/s/test-wheel @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Build and test an installed wheel in the development container. Pass an +# existing host wheel path to test those exact bytes instead. The smoke test +# runs outside /app with PYTHONPATH removed to prevent source-tree imports. + +set -euo pipefail +cd "$(git rev-parse --show-toplevel)" + +if ! docker info >/dev/null 2>&1; then + echo "Docker is not available. Start Docker and retry." >&2 + exit 1 +fi + +docker compose up -d +work_dir="/tmp/rcpchgrowth-wheel-$$" +docker compose exec -T rcpchgrowth-python rm -rf "$work_dir" +docker compose exec -T rcpchgrowth-python mkdir -p "$work_dir/dist" +trap 'docker compose exec -T rcpchgrowth-python rm -rf "$work_dir" >/dev/null 2>&1 || true' EXIT + +if (($# > 1)); then + echo "Usage: s/test-wheel [wheel path]" >&2 + exit 2 +fi +if (($#)); then + if [[ ! -f "$1" ]]; then + echo "Wheel not found: $1" >&2 + exit 1 + fi + host_wheel="$(realpath "$1")" + wheel_name="$(basename "$host_wheel")" + docker compose cp "$host_wheel" "rcpchgrowth-python:$work_dir/dist/$wheel_name" >/dev/null + wheel_path="$work_dir/dist/$wheel_name" +else + docker compose exec -T rcpchgrowth-python python -m build --wheel --outdir "$work_dir/dist" + wheel_path="$(docker compose exec -T rcpchgrowth-python sh -c 'set -- "$1"/*.whl; test "$#" -eq 1; printf "%s" "$1"' sh "$work_dir/dist")" +fi + +docker compose exec -T rcpchgrowth-python python /app/tests/distribution/check-wheel.py "$wheel_path" +docker compose exec -T rcpchgrowth-python python -m venv --system-site-packages "$work_dir/venv" +docker compose exec -T rcpchgrowth-python "$work_dir/venv/bin/python" -m pip install --force-reinstall --no-deps "$wheel_path" +docker compose exec -T -e PYTHONPATH= rcpchgrowth-python sh -c 'cd "$1" && exec "$2" /app/tests/distribution/installed-wheel-smoke.py' sh "$work_dir" "$work_dir/venv/bin/python" diff --git a/tests/distribution/check-wheel.py b/tests/distribution/check-wheel.py new file mode 100755 index 0000000..80d1b6c --- /dev/null +++ b/tests/distribution/check-wheel.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""Validate the minimum wheel structure needed by runtime consumers.""" + +import sys +from email.parser import Parser +from pathlib import Path +from zipfile import ZipFile + +REQUIRED_FILES = { + "rcpchgrowth/__init__.py", + "rcpchgrowth/_build_info.py", + "rcpchgrowth/data_tables/cdc2-20.json", + "rcpchgrowth/data_tables/cdc_infants.json", + "rcpchgrowth/data_tables/trisomy_21.json", + "rcpchgrowth/data_tables/trisomy_21_aap_children.json", + "rcpchgrowth/data_tables/trisomy_21_aap_infants.json", + "rcpchgrowth/data_tables/turner.json", + "rcpchgrowth/data_tables/uk90_child.json", + "rcpchgrowth/data_tables/uk90_preterm.json", + "rcpchgrowth/data_tables/uk90_term.json", + "rcpchgrowth/data_tables/uk_who_weight_correlation_matrices/weight_correlation_by_month.json", + "rcpchgrowth/data_tables/uk_who_weight_correlation_matrices/weight_correlation_by_week.json", + "rcpchgrowth/data_tables/who/pre_2025/who_infants.json", + "rcpchgrowth/data_tables/who/who_2007_children.json", + "rcpchgrowth/data_tables/who/who_children.json", + "rcpchgrowth/data_tables/who/who_infants.json", +} + + +def main() -> None: + wheel = Path(sys.argv[1]).resolve() + if not wheel.is_file() or wheel.suffix != ".whl": + raise SystemExit(f"Not a wheel: {wheel}") + + with ZipFile(wheel) as archive: + members = set(archive.namelist()) + metadata_names = [ + name for name in members if name.endswith(".dist-info/METADATA") + ] + if len(metadata_names) != 1: + raise AssertionError(f"Expected one METADATA file, found {metadata_names}") + metadata = Parser().parsestr(archive.read(metadata_names[0]).decode()) + + missing = sorted(REQUIRED_FILES - members) + forbidden = sorted( + name + for name in members + if "__pycache__" in name + or name.endswith((".pyc", ".pyo")) + or name.startswith("notebooks/") + ) + assert not missing, f"Required runtime files missing from wheel: {missing}" + assert not forbidden, f"Generated or notebook files found in wheel: {forbidden}" + assert metadata["Name"] == "rcpchgrowth" + assert metadata["Version"] + print( + f"Validated {wheel.name}: rcpchgrowth {metadata['Version']}, {len(members)} files" + ) + + +if __name__ == "__main__": + main() diff --git a/tests/distribution/installed-wheel-smoke.py b/tests/distribution/installed-wheel-smoke.py new file mode 100755 index 0000000..ccb214c --- /dev/null +++ b/tests/distribution/installed-wheel-smoke.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Exercise every runtime data family from an installed wheel.""" + +from datetime import date, timedelta +from importlib.metadata import version +from pathlib import Path + +import rcpchgrowth + +REFERENCE_CASES = { + "uk-who": ("female", 5.0), + "who": ("male", 1.0), + "cdc": ("male", 10.0), + "turners-syndrome": ("female", 10.0), + "trisomy-21": ("female", 5.0), + "trisomy-21-aap": ("female", 5.0), +} +SERVER_ROOT_IMPORTS = { + "Measurement", + "centile", + "chronological_calendar_age", + "corrected_decimal_age", + "create_chart", + "generate_fictional_child_data", + "lower_and_upper_limits_of_expected_height_z", + "measurement_from_sds", + "mid_parental_height_z", + "sds_for_measurement", +} + + +def main() -> None: + module_path = Path(rcpchgrowth.__file__).resolve() + assert "site-packages" in module_path.parts, module_path + assert version("rcpchgrowth") + assert all(hasattr(rcpchgrowth, name) for name in SERVER_ROOT_IMPORTS) + + birth_date = date(2015, 1, 1) + for reference, (sex, age) in REFERENCE_CASES.items(): + measurement = rcpchgrowth.Measurement( + birth_date=birth_date, + observation_date=birth_date + timedelta(days=age * 365.25), + measurement_method="height", + observation_value=110.0, + reference=reference, + sex=sex, + ).measurement + assert measurement["provenance"]["growth_reference"] == reference + assert ( + measurement["measurement_calculated_values"]["chronological_sds"] + is not None + ) + chart = rcpchgrowth.create_chart( + reference=reference, + centile_format=[50], + measurement_method="height", + sex=sex, + ) + assert chart and any(next(iter(component.values())) for component in chart) + + assert isinstance(rcpchgrowth.return_correlation(3, 4, "months"), float) + assert isinstance(rcpchgrowth.return_correlation(3, 4, "weeks"), float) + print(f"Installed-wheel smoke passed for {version('rcpchgrowth')} at {module_path}") + + +if __name__ == "__main__": + main() From 573631f8ea0174afeae8b323a5b8420ef24b33dd Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Thu, 3 Sep 2026 22:51:14 +0100 Subject: [PATCH 04/11] test: add downstream candidate-wheel gate --- s/test-downstream-wheel | 113 ++++++++++++++++++ .../normalize-downstream-provenance.py | 46 +++++++ .../test_normalize_downstream_provenance.py | 81 +++++++++++++ 3 files changed, 240 insertions(+) create mode 100755 s/test-downstream-wheel create mode 100755 tests/distribution/normalize-downstream-provenance.py create mode 100644 tests/distribution/test_normalize_downstream_provenance.py diff --git a/s/test-downstream-wheel b/s/test-downstream-wheel new file mode 100755 index 0000000..742d363 --- /dev/null +++ b/s/test-downstream-wheel @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# Test an exact candidate wheel against an immutable server revision. The +# server revision must contain the regression and pinned React compatibility +# suites introduced by digital-growth-charts-server PR #280. + +set -euo pipefail +cd "$(git rev-parse --show-toplevel)" +root="$PWD" + +if [[ $# -ne 2 || ! "$2" =~ ^[0-9a-f]{40}$ ]]; then + echo "Usage: s/test-downstream-wheel <40-character server commit>" >&2 + exit 2 +fi +for command in docker git; do + if ! command -v "$command" >/dev/null 2>&1; then + echo "Required command not found: $command" >&2 + exit 1 + fi +done + +server_revision="$2" +server_repository="${RCPCHGROWTH_SERVER_REPOSITORY:-https://github.com/rcpch/digital-growth-charts-server.git}" +if [[ ! -f "$1" ]]; then + echo "Wheel not found: $1" >&2 + exit 1 +fi +wheel="$(realpath "$1")" +if ! docker info >/dev/null 2>&1; then + echo "Docker is not available. Start Docker and retry." >&2 + exit 1 +fi + +work_dir="$(mktemp -d)" +host_user="$(id -u):$(id -g)" +cleanup() { + docker compose --project-directory "$work_dir/server" exec -T growth-api chown -R "$host_user" /app >/dev/null 2>&1 || true + docker compose --project-directory "$work_dir/server" down >/dev/null 2>&1 || true + rm -rf "$work_dir" +} +trap cleanup EXIT + +git clone --quiet --filter=blob:none --no-checkout "$server_repository" "$work_dir/server" +git -C "$work_dir/server" fetch --quiet --depth=1 origin "$server_revision" +git -C "$work_dir/server" checkout --quiet --detach FETCH_HEAD + +if [[ ! -x "$work_dir/server/s/compatibility-test" ]] || [[ ! -d "$work_dir/server/tests/regression/golden" ]]; then + echo "Server commit $server_revision does not contain the required regression and compatibility suites." >&2 + echo "Use an immutable live commit containing server PR #280." >&2 + exit 1 +fi + +candidate_version="$(python - "$wheel" <<'PY' +from email.parser import Parser +import sys +from zipfile import ZipFile + +with ZipFile(sys.argv[1]) as archive: + name = next(item for item in archive.namelist() if item.endswith('.dist-info/METADATA')) + print(Parser().parsestr(archive.read(name).decode())['Version']) +PY +)" +candidate_commit="$(python - "$wheel" <<'PY' +import ast +import sys +from zipfile import ZipFile + +with ZipFile(sys.argv[1]) as archive: + module = ast.parse(archive.read('rcpchgrowth/_build_info.py').decode()) + assignment = next( + node + for node in module.body + if isinstance(node, ast.Assign) + and any( + isinstance(target, ast.Name) and target.id == 'COMMIT' + for target in node.targets + ) + ) + print(ast.literal_eval(assignment.value)) +PY +)" +if [[ ! "$candidate_commit" =~ ^[0-9a-f]{40}$ ]]; then + echo "Candidate wheel has no immutable build commit: $candidate_commit" >&2 + exit 1 +fi + +golden_dir="$work_dir/server/tests/regression/golden" +golden_backup="$work_dir/golden-original" +cp -a "$golden_dir" "$golden_backup" +python "$root/tests/distribution/normalize-downstream-provenance.py" "$golden_dir" "$candidate_version" "$candidate_commit" + +GITHUB_SHA="$server_revision" docker compose --project-directory "$work_dir/server" up --build -d growth-api +wheel_name="$(basename "$wheel")" +docker compose --project-directory "$work_dir/server" cp "$wheel" "growth-api:/tmp/$wheel_name" >/dev/null +host_wheel_sha="$(sha256sum "$wheel" | cut -d' ' -f1)" +container_wheel_sha="$(docker compose --project-directory "$work_dir/server" exec -T growth-api sha256sum "/tmp/$wheel_name" | cut -d' ' -f1)" +if [[ "$host_wheel_sha" != "$container_wheel_sha" ]]; then + echo "Candidate wheel hash changed while copying into the server container." >&2 + exit 1 +fi +docker compose --project-directory "$work_dir/server" exec -T growth-api python -m pip install --force-reinstall --no-deps "/tmp/$wheel_name" +docker compose --project-directory "$work_dir/server" exec -T \ + -e EXPECTED_VERSION="$candidate_version" \ + -e EXPECTED_COMMIT="$candidate_commit" \ + growth-api python -c 'import os, pathlib; from importlib.metadata import version; import rcpchgrowth; from rcpchgrowth._build_info import COMMIT; assert version("rcpchgrowth") == os.environ["EXPECTED_VERSION"]; assert COMMIT == os.environ["EXPECTED_COMMIT"]; assert "/site-packages/" in pathlib.Path(rcpchgrowth.__file__).as_posix()' +docker compose --project-directory "$work_dir/server" restart growth-api +docker compose --project-directory "$work_dir/server" exec -T growth-api pytest -q +( + cd "$work_dir/server" + COMPONENT_REPOSITORY_PATH=/does-not-exist s/compatibility-test --reuse-api +) +rm -rf "$golden_dir" +mv "$golden_backup" "$golden_dir" +git -C "$work_dir/server" diff --exit-code diff --git a/tests/distribution/normalize-downstream-provenance.py b/tests/distribution/normalize-downstream-provenance.py new file mode 100755 index 0000000..d850910 --- /dev/null +++ b/tests/distribution/normalize-downstream-provenance.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""Substitute only candidate engine provenance in downstream golden files.""" + +import json +import sys +from pathlib import Path + +CANONICAL_ENGINE_NAME = "rcpch/rcpchgrowth-python" +SUPPORTED_ENGINE_NAMES = {"rcpchgrowth", CANONICAL_ENGINE_NAME} + + +def update(value, version: str, commit: str) -> int: + changes = 0 + if isinstance(value, dict): + engine = value.get("calculation_engine") + if isinstance(engine, dict) and engine.get("name") in SUPPORTED_ENGINE_NAMES: + engine["name"] = CANONICAL_ENGINE_NAME + engine["version"] = version + engine["commit"] = commit + changes += 1 + for child in value.values(): + changes += update(child, version, commit) + elif isinstance(value, list): + for child in value: + changes += update(child, version, commit) + return changes + + +def main() -> None: + golden_dir = Path(sys.argv[1]) + version = sys.argv[2] + commit = sys.argv[3] + changes = 0 + for path in golden_dir.rglob("*.json"): + value = json.loads(path.read_text()) + file_changes = update(value, version, commit) + if file_changes: + path.write_text(json.dumps(value, indent=2) + "\n") + changes += file_changes + if not changes: + raise SystemExit("No supported rcpchgrowth provenance found in downstream goldens") + print(f"Normalized {changes} downstream provenance records") + + +if __name__ == "__main__": + main() diff --git a/tests/distribution/test_normalize_downstream_provenance.py b/tests/distribution/test_normalize_downstream_provenance.py new file mode 100644 index 0000000..1e09467 --- /dev/null +++ b/tests/distribution/test_normalize_downstream_provenance.py @@ -0,0 +1,81 @@ +import json +import subprocess +import sys +from pathlib import Path + + +SCRIPT = Path(__file__).with_name("normalize-downstream-provenance.py") +CANONICAL_ENGINE_NAME = "rcpch/rcpchgrowth-python" +COMMIT = "a" * 40 + + +def run_normalizer(golden_dir: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, str(SCRIPT), str(golden_dir), "4.6.2", COMMIT], + capture_output=True, + check=False, + text=True, + ) + + +def test_normalizes_legacy_and_canonical_engine_provenance(tmp_path): + golden = tmp_path / "measurement.json" + golden.write_text( + json.dumps( + { + "measurements": [ + { + "provenance": { + "calculation_engine": { + "name": "rcpchgrowth", + "version": "4.6.0", + "commit": "old", + } + } + }, + { + "provenance": { + "calculation_engine": { + "name": CANONICAL_ENGINE_NAME, + "version": "4.6.1", + "commit": "previous", + } + } + }, + ] + } + ) + ) + + result = run_normalizer(tmp_path) + + assert result.returncode == 0, result.stderr + assert result.stdout == "Normalized 2 downstream provenance records\n" + engines = [ + measurement["provenance"]["calculation_engine"] + for measurement in json.loads(golden.read_text())["measurements"] + ] + assert engines == [ + {"name": CANONICAL_ENGINE_NAME, "version": "4.6.2", "commit": COMMIT}, + {"name": CANONICAL_ENGINE_NAME, "version": "4.6.2", "commit": COMMIT}, + ] + + +def test_preserves_unrelated_engine_provenance(tmp_path): + golden = tmp_path / "measurement.json" + original = { + "provenance": { + "calculation_engine": { + "name": "another-engine", + "version": "1.0.0", + "commit": "unchanged", + } + } + } + golden.write_text(json.dumps(original)) + + result = run_normalizer(tmp_path) + + assert result.returncode != 0 + assert "No supported rcpchgrowth provenance found" in result.stderr + assert json.loads(golden.read_text()) == original From 4acfe623f5977bad1ed0e748f68c7e8a3579842c Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Thu, 3 Sep 2026 22:52:13 +0100 Subject: [PATCH 05/11] docs: record quality gate progress and baseline --- s/README.md | 4 ++++ spec/pre-cleanup-baseline.md | 29 +++++++++++++++++++++++++++++ spec/roadmap.md | 17 +++++++++-------- 3 files changed, 42 insertions(+), 8 deletions(-) create mode 100644 spec/pre-cleanup-baseline.md diff --git a/s/README.md b/s/README.md index e4e193e..54f43fa 100644 --- a/s/README.md +++ b/s/README.md @@ -14,6 +14,10 @@ The `s/` directory contains convenience scripts that standardise repeated develo - `s/notebook` - start JupyterLab and open it in a browser. - `s/test [pytest arguments]` - start the container if needed and run pytest. - `s/test --running [pytest arguments]` - run pytest in an already-running container. +- `s/lint [Ruff arguments]` - run the adopted blocking Ruff rules in the development container. +- `s/lint --audit` - report the broader proposed Ruff rule set without failing. +- `s/test-wheel [wheel path]` - build or accept a wheel, inspect it, install it outside the source tree, and run package smoke tests. +- `s/test-downstream-wheel ` - test an exact wheel against an immutable compatible server revision. - `s/remove-containers-and-images` - stop the Compose stack and remove locally built images. ## Optional Release Preparation diff --git a/spec/pre-cleanup-baseline.md b/spec/pre-cleanup-baseline.md new file mode 100644 index 0000000..fe96348 --- /dev/null +++ b/spec/pre-cleanup-baseline.md @@ -0,0 +1,29 @@ +# Pre-cleanup baseline + +This baseline implements the evidence record required by R17 before mechanical maintenance changes. It was recorded on 2026-09-03 from the tree committed with this document, after rebasing the R16-R23 quality-gate implementation onto `v4.6.2`. + +## Environment + +- Container runtime: Python 3.13.13 +- Test runner: pytest 9.1.1 +- Full command: `s/test --running -q -rs` +- Result: 145239 collected, 145239 passed, 0 skipped, 0 failed in 167.84 seconds + +## Fixture integrity + +The primary UK-WHO fixture `rcpchgrowth/tests/sds_age_validation_2021_refactored_2026.json` contains 3984 cases. Its SHA-256 digest is `e269b3ff4312bca76563eb8adc7d0e9941b655077d11888b34522c3ffe2f6c84`. `test_fixture_integrity.py` enforces the count, digest, and complete field set without changing the fixture. + +## Focused commands + +- Calculation pipeline: `s/test --running rcpchgrowth/tests/test_global_functions.py rcpchgrowth/tests/test_uk_who.py -q` +- Measurement response contract: `s/test --running rcpchgrowth/tests/test_measurement_contract.py rcpchgrowth/tests/test_provenance.py -q` +- Chart generation: `s/test --running rcpchgrowth/tests/test_chart_functions.py rcpchgrowth/tests/test_chart_age_grids.py rcpchgrowth/tests/test_chart_age_two_overlap.py -q -rs` +- Source-package imports and fixture integrity: `s/test --running rcpchgrowth/tests/test_public_api_contract.py rcpchgrowth/tests/test_fixture_integrity.py -q` +- Adopted Ruff checks: `s/lint --running` +- Broader non-blocking Ruff audit: `s/lint --running --audit` +- Installed-wheel contract: `s/test-wheel` +- Complete supported suite: `s/test --running -q -rs` + +## Downstream compatibility + +The downstream runner was exercised with `digital-growth-charts-server` PR #280 commit `6918e3b8152963345f4c0996e7bd52bacf904583` and the published `rcpchgrowth` 4.6.0 wheel. The server suite passed 1066 tests with one documented skip, and both pinned React profiles passed all 47 assertions. This verifies the runner at those exact revisions, not against PR #280's moving head or the current candidate parcel. Repeat the gate with a commit-stamped candidate wheel against PR #280's immutable merge commit after it adopts the canonical engine identifier. diff --git a/spec/roadmap.md b/spec/roadmap.md index aa44d7d..fdbe42b 100644 --- a/spec/roadmap.md +++ b/spec/roadmap.md @@ -20,19 +20,20 @@ This roadmap records the maintenance findings from the August 2026 repository au Complete these items before broad formatting, import cleanup, packaging reduction, or dead-code removal. They establish the baseline needed to distinguish intended maintenance from calculation or API regressions. -- [ ] **R16 - Introduce Ruff in staged, reviewable increments.** Add project configuration and a lint entry point early, excluding generated/reference vectors such as `rcpchgrowth/tests/who_test_data/who_chart_precalculated_centiles.py` from formatting. Begin with audit-only checks and small rule groups; do not run global or unsafe auto-fixes. Configure the `python-dateutil` to `dateutil` module mapping and account for tooling dependencies so false positives do not encourage removal of required packages. Keep type checking as a separate future concern. -- [ ] **R17 - Record a reproducible pre-cleanup baseline.** Run the complete supported test suite, record collection/pass/skip totals, verify the primary UK-WHO fixture count and integrity, and retain focused commands for calculation, chart, and package tests. Understand any baseline failure before maintenance work starts. -- [ ] **R18 - Characterize the public Python import surface.** Add tests for the root names and module imports used by `digital-growth-charts-server`, including call signatures or representative invocations where practical. Use this contract to guide `__all__`, import cleanup, and deprecation decisions rather than assuming apparently unused exports are private. -- [ ] **R19 - Characterize serialized measurement contracts.** Add golden or schema-focused tests for the complete `Measurement.measurement` dictionary across representative successful, corrected-age, out-of-range, and validation-error cases. Assert key presence, nesting, nullability, value types, reference identifiers, and clinically significant numeric values. -- [ ] **R20 - Fill calculation-vector and boundary gaps.** Add independent representative vectors for Turner syndrome, Trisomy 21, and Trisomy 21 AAP; CDC infant head circumference and extended BMI; exact component reference-selection handovers; and documented error boundaries for every supported reference. Prefer authoritative published vectors or separately derived expected values over self-generated snapshots. -- [ ] **R21 - Characterize chart-builder outputs.** Test the structure, series selection, age domains, boundary ownership, and representative coordinates of dynamically generated charts for every supported reference, including the non-WHO chart families. Exercise generation directly rather than relying only on committed chart assets. -- [ ] **R22 - Add installed-wheel smoke and resource tests.** Build and install the candidate wheel into an isolated environment, verify public imports and version metadata, and execute representative calculations that load every required packaged data family. Assert that tests, notebooks, provenance sources, and other non-runtime material follow the packaging policy established by R4-R8. -- [ ] **R23 - Add a downstream candidate-wheel compatibility gate.** After the active provenance upgrade has landed, install the exact candidate wheel into a pinned `digital-growth-charts-server` revision and run its import, calculation, dynamic chart-generation, and HTTP golden-response suites. Include the server's pinned React consumer compatibility matrix where available, and keep this as a release gate for changes to exports, calculations, chart structures, packaging, or serialized output. +- [x] **R16 - Introduce Ruff in staged, reviewable increments.** Project configuration, a blocking correctness subset, and a non-blocking broader audit are available through `s/lint`; the generated WHO vector file is excluded from Ruff processing, and the known R1 undefined plotting name is quarantined explicitly. Do not run global or unsafe auto-fixes. Keep type checking and dependency analysis as separate future concerns; any dependency analyser must map `python-dateutil` to `dateutil` and account for tooling dependencies. +- [x] **R17 - Record a reproducible pre-cleanup baseline.** Run the complete supported test suite, record collection/pass/skip totals, verify the primary UK-WHO fixture count and integrity, and retain focused commands for calculation, chart, and package tests. Understand any baseline failure before maintenance work starts. +- [x] **R18 - Characterize the public Python import surface.** Add tests for the root names and module imports used by `digital-growth-charts-server`, including call signatures or representative invocations where practical. Use this contract to guide `__all__`, import cleanup, and deprecation decisions rather than assuming apparently unused exports are private. +- [x] **R19 - Characterize serialized measurement contracts.** Add golden or schema-focused tests for the complete `Measurement.measurement` dictionary across representative successful, corrected-age, out-of-range, and validation-error cases. Assert key presence, nesting, nullability, value types, reference identifiers, and clinically significant numeric values. +- [x] **R20 - Fill calculation-vector and boundary gaps.** Add independent representative vectors for Turner syndrome, Trisomy 21, and Trisomy 21 AAP; CDC infant head circumference and extended BMI; exact component reference-selection handovers; and documented error boundaries for every supported reference. Prefer authoritative published vectors or separately derived expected values over self-generated snapshots. +- [x] **R21 - Characterize chart-builder outputs.** Test the structure, series selection, age domains, boundary ownership, and representative coordinates of dynamically generated charts for every supported reference, including the non-WHO chart families. Exercise generation directly rather than relying only on committed chart assets. +- [~] **R22 - Add installed-wheel smoke and resource tests.** Build and install the candidate wheel into an isolated environment, verify public imports and version metadata, and execute representative calculations that load every required packaged data family. The isolated smoke and required-resource checks are active locally and in CI; finish the forbidden-content assertions after R4-R8 establish the approved package policy. +- [~] **R23 - Add a downstream candidate-wheel compatibility gate.** The exact-wheel runner validates artifact hashes and provenance, the server calculation/API regression suite, dynamic charts, and both pinned React profiles against an immutable server revision. The runner passed against PR #280 commit `6918e3b8152963345f4c0996e7bd52bacf904583` using the published 4.6.0 wheel; repeat it with a commit-stamped candidate wheel against PR #280's immutable merge commit after that PR adopts the canonical engine identifier, then use that merge commit as the permanent release/CI pin. ## High priority - [ ] **R1 - Resolve the broken public `create_thrive_lines` API.** `rcpchgrowth/dynamic_growth.py:226-294` uses undefined `plt`, has its output-producing plotting code commented out, and returns nothing, while `rcpchgrowth/__init__.py:8` exports it publicly. Decide whether plotting remains supported; either restore it with an optional matplotlib dependency and tests, or remove the export through an explicitly managed breaking change. - [ ] **R2 - Restore executable notebooks and expose package version metadata consistently.** `Quickstart.ipynb`, `AdditionalFunctions.ipynb`, `ExperimentalFunctions.ipynb`, and `ResearchTemplate.ipynb` all fail in their first setup cell because `rcpchgrowth.__version__` does not exist. Expose the installed version using `importlib.metadata.version()` or update the notebooks not to expect `__version__`, then execute every notebook in CI or another repeatable check. +- [ ] **R24 - Automate the post-merge release cascade.** Replace the documented manual tag and GitHub Release steps with CI that runs only after a validated `release/vX.Y.Z` PR is merged into `live`. The workflow must confirm that the merge commit contains the matching synchronized version in `pyproject.toml` and `CITATION.cff`, create an annotated `vX.Y.Z` tag at that exact merge commit without overwriting an existing tag, create the GitHub Release with generated notes, and invoke the existing tested PyPI publication path directly through `workflow_call` rather than relying on a `GITHUB_TOKEN` tag or release event to trigger another workflow. Refactor `python-publish.yml` into a least-privilege reusable workflow, retain the Python 3.10-3.13 test gate and trusted publishing, add concurrency and idempotency guards, update `s/version++` and `s/README.md` so merging the release PR is the final maintainer action, and validate the complete cascade in a non-publishing test or dry-run path before enabling it on `live`. ## Medium priority From 97c780df60d8cccf349f2a64d4aa1543b2b291ac Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Fri, 4 Sep 2026 00:20:11 +0100 Subject: [PATCH 06/11] fix: align AAP boundary and harden wheel checks --- .gitignore | 8 +++ .../test_reference_vectors_and_boundaries.py | 34 ++++----- rcpchgrowth/trisomy_21_aap.py | 4 +- s/test-downstream-wheel | 1 + s/test-wheel | 4 +- tests/distribution/check-wheel.py | 5 +- tests/distribution/installed-wheel-smoke.py | 10 ++- tests/distribution/test_check_wheel.py | 69 +++++++++++++++++++ 8 files changed, 113 insertions(+), 22 deletions(-) create mode 100644 tests/distribution/test_check_wheel.py diff --git a/.gitignore b/.gitignore index 6890fca..a863883 100644 --- a/.gitignore +++ b/.gitignore @@ -139,3 +139,11 @@ rcpchgrowth/data_tables/fenton/*.* # Private files .private + +# Disposable agent and browser-automation output +.claude/ +.playwright-mcp/ +.playwright/ +playwright-report/ +blob-report/ +test-results/ diff --git a/rcpchgrowth/tests/test_reference_vectors_and_boundaries.py b/rcpchgrowth/tests/test_reference_vectors_and_boundaries.py index 7ba4c1c..7fadafd 100644 --- a/rcpchgrowth/tests/test_reference_vectors_and_boundaries.py +++ b/rcpchgrowth/tests/test_reference_vectors_and_boundaries.py @@ -1,5 +1,7 @@ """Characterization vectors and boundaries for the supported references.""" +import math + import pytest from rcpchgrowth import cdc, trisomy_21_aap, uk_who, who @@ -11,6 +13,7 @@ HEAD_CIRCUMFERENCE, HEIGHT, MALE, + TWENTY_THREE_WEEKS_GESTATION, TRISOMY_21, TRISOMY_21_AAP, TURNERS, @@ -69,12 +72,12 @@ def test_published_reference_vectors(reference, age, method, sex, sds, expected) @pytest.mark.parametrize( ("reference", "valid_age", "invalid_age", "method", "sex", "message"), [ - (UK_WHO, 20, 20.0001, HEIGHT, MALE, "above the age of 20 years"), - (WHO, 19, 19.0001, HEIGHT, MALE, "above 19 years"), - (CDC, 20, 20.0001, HEIGHT, MALE, "above the age of 20 years"), - (TURNERS, 20, 20.0001, HEIGHT, FEMALE, "above 20 years"), - (TRISOMY_21, 20, 20.0001, HEIGHT, MALE, "over the age of 20y"), - (TRISOMY_21_AAP, 20, 20.0001, HEIGHT, MALE, "over the age of 20y"), + (UK_WHO, 20, math.nextafter(20, math.inf), HEIGHT, MALE, "above the age of 20 years"), + (WHO, 19, math.nextafter(19, math.inf), HEIGHT, MALE, "above 19 years"), + (CDC, 20, math.nextafter(20, math.inf), HEIGHT, MALE, "above the age of 20 years"), + (TURNERS, 20, math.nextafter(20, math.inf), HEIGHT, FEMALE, "above 20 years"), + (TRISOMY_21, 20, math.nextafter(20, math.inf), HEIGHT, MALE, "over the age of 20y"), + (TRISOMY_21_AAP, 20, math.nextafter(20, math.inf), HEIGHT, MALE, "over the age of 20y"), ], ) def test_every_reference_includes_its_upper_boundary_and_rejects_above_it( @@ -88,12 +91,12 @@ def test_every_reference_includes_its_upper_boundary_and_rejects_above_it( @pytest.mark.parametrize( ("reference", "valid_age", "invalid_age", "method", "sex", "message"), [ - (UK_WHO, 0, -1, WEIGHT, MALE, "below 23 weeks gestation"), - (WHO, 0, -0.0001, HEIGHT, MALE, "below term"), - (CDC, 0, -0.0001, HEIGHT, MALE, "below 40 weeks"), - (TURNERS, 1, 0.9999, HEIGHT, FEMALE, "below 1 year"), - (TRISOMY_21, 0, -0.0001, HEIGHT, MALE, "below 40 weeks"), - (TRISOMY_21_AAP, 0, -0.0001, WEIGHT, MALE, "below 40 weeks"), + (UK_WHO, TWENTY_THREE_WEEKS_GESTATION, math.nextafter(TWENTY_THREE_WEEKS_GESTATION, -math.inf), WEIGHT, MALE, "below 23 weeks gestation"), + (WHO, 0, math.nextafter(0, -math.inf), HEIGHT, MALE, "below term"), + (CDC, 0, math.nextafter(0, -math.inf), HEIGHT, MALE, "below 40 weeks"), + (TURNERS, 1, math.nextafter(1, -math.inf), HEIGHT, FEMALE, "below 1 year"), + (TRISOMY_21, 0, math.nextafter(0, -math.inf), HEIGHT, MALE, "below 40 weeks"), + (TRISOMY_21_AAP, 0, math.nextafter(0, -math.inf), WEIGHT, MALE, "below 40 weeks"), ], ) def test_every_reference_includes_its_lower_boundary_and_rejects_below_it( @@ -153,11 +156,10 @@ def test_turner_rejects_unsupported_measurements_and_sex(method, sex, message): measurement_from_sds(TURNERS, 0, method, sex, 10) -def test_aap_declared_one_month_cutoff_precedes_first_height_lms_row(): - """The current 0.083 cutoff admits ages below the first 0.083333333 source row, where interpolation leaks ValueError rather than a domain LookupError.""" +def test_aap_one_month_cutoff_matches_first_height_lms_row(): invalid, _ = trisomy_21_aap.reference_data_absent(0.083, HEIGHT, MALE) - assert invalid is False - with pytest.raises(ValueError): + assert invalid is True + with pytest.raises(LookupError, match="below 1 month"): measurement_from_sds(TRISOMY_21_AAP, 0, HEIGHT, MALE, 0.083) diff --git a/rcpchgrowth/trisomy_21_aap.py b/rcpchgrowth/trisomy_21_aap.py index 9b4d5c2..2bb56e7 100644 --- a/rcpchgrowth/trisomy_21_aap.py +++ b/rcpchgrowth/trisomy_21_aap.py @@ -53,7 +53,7 @@ def reference_data_absent( if age < 0: # lower threshold of trisomy_21 data return True, "No Trisomy 21 (AAP) reference data exists below 40 weeks gestation" - if age < 0.083 and measurement_method in ['height', 'ofc']: # lower threshold of trisomy_21 data + if age < TRISOMY_21_AAP_INFANT_AGES[0] and measurement_method in ['height', 'ofc']: # lower threshold of trisomy_21 data return True, "No Trisomy 21 (AAP) reference data exists below 1 month of age for height or head circumference." if age < 2 and measurement_method == 'bmi': # lower threshold of trisomy_21 data @@ -102,4 +102,4 @@ def select_reference_data_for_trisomy_21_aap(trisomy_21_aap_reference_name, meas return_value = trisomy_21_aap_lms_array_for_measurement_and_sex(measurement_method=measurement_method, sex=sex, age=4.0, default_youngest_reference=default_youngest_reference) # select arbitrary age of 4 y for child data except: raise LookupError(f"No data for {measurement_method} in the {sex} Trisomy 21 (US) dataset (>3y).") - return return_value \ No newline at end of file + return return_value diff --git a/s/test-downstream-wheel b/s/test-downstream-wheel index 742d363..587b41f 100755 --- a/s/test-downstream-wheel +++ b/s/test-downstream-wheel @@ -78,6 +78,7 @@ with ZipFile(sys.argv[1]) as archive: print(ast.literal_eval(assignment.value)) PY )" +python "$root/tests/distribution/check-wheel.py" "$wheel" if [[ ! "$candidate_commit" =~ ^[0-9a-f]{40}$ ]]; then echo "Candidate wheel has no immutable build commit: $candidate_commit" >&2 exit 1 diff --git a/s/test-wheel b/s/test-wheel index 3aa65e3..7940094 100755 --- a/s/test-wheel +++ b/s/test-wheel @@ -36,6 +36,6 @@ else fi docker compose exec -T rcpchgrowth-python python /app/tests/distribution/check-wheel.py "$wheel_path" -docker compose exec -T rcpchgrowth-python python -m venv --system-site-packages "$work_dir/venv" -docker compose exec -T rcpchgrowth-python "$work_dir/venv/bin/python" -m pip install --force-reinstall --no-deps "$wheel_path" +docker compose exec -T rcpchgrowth-python python -m venv "$work_dir/venv" +docker compose exec -T rcpchgrowth-python sh -c 'cd "$1" && exec env -u PYTHONPATH "$2" -m pip install "$3"' sh "$work_dir" "$work_dir/venv/bin/python" "$wheel_path" docker compose exec -T -e PYTHONPATH= rcpchgrowth-python sh -c 'cd "$1" && exec "$2" /app/tests/distribution/installed-wheel-smoke.py' sh "$work_dir" "$work_dir/venv/bin/python" diff --git a/tests/distribution/check-wheel.py b/tests/distribution/check-wheel.py index 80d1b6c..4903486 100755 --- a/tests/distribution/check-wheel.py +++ b/tests/distribution/check-wheel.py @@ -47,12 +47,15 @@ def main() -> None: for name in members if "__pycache__" in name or name.endswith((".pyc", ".pyo")) - or name.startswith("notebooks/") + or "notebooks" in Path(name).parts ) assert not missing, f"Required runtime files missing from wheel: {missing}" assert not forbidden, f"Generated or notebook files found in wheel: {forbidden}" assert metadata["Name"] == "rcpchgrowth" assert metadata["Version"] + assert wheel.name.startswith(f"rcpchgrowth-{metadata['Version']}-"), ( + f"Wheel filename and metadata version differ: {wheel.name}, {metadata['Version']}" + ) print( f"Validated {wheel.name}: rcpchgrowth {metadata['Version']}, {len(members)} files" ) diff --git a/tests/distribution/installed-wheel-smoke.py b/tests/distribution/installed-wheel-smoke.py index ccb214c..447c22a 100755 --- a/tests/distribution/installed-wheel-smoke.py +++ b/tests/distribution/installed-wheel-smoke.py @@ -1,11 +1,13 @@ #!/usr/bin/env python3 """Exercise every runtime data family from an installed wheel.""" +import os from datetime import date, timedelta from importlib.metadata import version from pathlib import Path import rcpchgrowth +from rcpchgrowth._build_info import COMMIT REFERENCE_CASES = { "uk-who": ("female", 5.0), @@ -32,7 +34,9 @@ def main() -> None: module_path = Path(rcpchgrowth.__file__).resolve() assert "site-packages" in module_path.parts, module_path - assert version("rcpchgrowth") + assert rcpchgrowth.__version__ == version("rcpchgrowth") + if expected_commit := os.environ.get("EXPECTED_COMMIT"): + assert COMMIT == expected_commit assert all(hasattr(rcpchgrowth, name) for name in SERVER_ROOT_IMPORTS) birth_date = date(2015, 1, 1) @@ -46,6 +50,10 @@ def main() -> None: sex=sex, ).measurement assert measurement["provenance"]["growth_reference"] == reference + assert ( + measurement["provenance"]["calculation_engine"]["version"] + == rcpchgrowth.__version__ + ) assert ( measurement["measurement_calculated_values"]["chronological_sds"] is not None diff --git a/tests/distribution/test_check_wheel.py b/tests/distribution/test_check_wheel.py new file mode 100644 index 0000000..571a840 --- /dev/null +++ b/tests/distribution/test_check_wheel.py @@ -0,0 +1,69 @@ +import subprocess +import sys +from pathlib import Path +from runpy import run_path +from zipfile import ZipFile + + +SCRIPT = Path(__file__).with_name("check-wheel.py") +REQUIRED_FILES = run_path(str(SCRIPT))["REQUIRED_FILES"] + + +def make_wheel(tmp_path, *, version="4.6.2", omitted=None, extra=()): + wheel = tmp_path / f"rcpchgrowth-{version}-py3-none-any.whl" + members = REQUIRED_FILES - ({omitted} if omitted else set()) + with ZipFile(wheel, "w") as archive: + for name in members: + archive.writestr(name, "") + archive.writestr( + f"rcpchgrowth-{version}.dist-info/METADATA", + f"Name: rcpchgrowth\nVersion: {version}\n", + ) + for name in extra: + archive.writestr(name, "") + return wheel + + +def check(wheel): + return subprocess.run( + [sys.executable, str(SCRIPT), str(wheel)], + capture_output=True, + check=False, + text=True, + ) + + +def test_accepts_minimal_valid_wheel(tmp_path): + result = check(make_wheel(tmp_path)) + assert result.returncode == 0, result.stderr + + +def test_rejects_missing_runtime_resource(tmp_path): + missing = "rcpchgrowth/data_tables/who/who_infants.json" + result = check(make_wheel(tmp_path, omitted=missing)) + assert result.returncode != 0 + assert missing in result.stderr + + +def test_rejects_nested_notebook_content(tmp_path): + result = check(make_wheel(tmp_path, extra=("rcpchgrowth/notebooks/demo.ipynb",))) + assert result.returncode != 0 + assert "demo.ipynb" in result.stderr + + +def test_rejects_filename_metadata_version_mismatch(tmp_path): + wheel = make_wheel(tmp_path) + mismatched = wheel.with_name("rcpchgrowth-4.6.1-py3-none-any.whl") + wheel.rename(mismatched) + result = check(mismatched) + assert result.returncode != 0 + assert "filename and metadata version differ" in result.stderr + + +def test_rejects_multiple_metadata_files(tmp_path): + wheel = make_wheel(tmp_path) + with ZipFile(wheel, "a") as archive: + archive.writestr("other-1.0.dist-info/METADATA", "Name: other\nVersion: 1.0\n") + result = check(wheel) + assert result.returncode != 0 + assert "Expected one METADATA file" in result.stderr From 298a469139f79ef0ea1ad576d00b868beed9d13e Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Fri, 4 Sep 2026 00:20:17 +0100 Subject: [PATCH 07/11] feat: expose version and execute notebooks --- .../run-pytest-on-push-and-all-prs.yml | 21 +++++++++++-- notebooks/ExperimentalFunctions.ipynb | 2 +- notebooks/Quickstart.ipynb | 30 +++++++++---------- notebooks/ResearchTemplate.ipynb | 6 ++-- pyproject.toml | 5 ++-- rcpchgrowth/__init__.py | 1 + rcpchgrowth/_version.py | 7 +++++ rcpchgrowth/measurement.py | 9 ++---- rcpchgrowth/tests/test_provenance.py | 2 ++ 9 files changed, 53 insertions(+), 30 deletions(-) create mode 100644 rcpchgrowth/_version.py diff --git a/.github/workflows/run-pytest-on-push-and-all-prs.yml b/.github/workflows/run-pytest-on-push-and-all-prs.yml index 6e792d5..ff0a215 100644 --- a/.github/workflows/run-pytest-on-push-and-all-prs.yml +++ b/.github/workflows/run-pytest-on-push-and-all-prs.yml @@ -17,6 +17,11 @@ on: # publish workflow) so that a release is gated on the SAME tests that run on # every PR and push - no duplicated, drift-prone test steps. workflow_call: + inputs: + ref: + description: Optional exact commit to test + required: false + type: string pull_request: branches: - "*" @@ -40,6 +45,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + ref: ${{ inputs.ref || github.sha }} persist-credentials: false - name: Set up Python ${{ matrix.python-version }} @@ -67,6 +73,14 @@ jobs: if: matrix.python-version == '3.13' run: python -m ruff check . + - name: Execute notebooks + if: matrix.python-version == '3.13' + env: + MPLBACKEND: Agg + run: | + pip install .[notebook] + jupyter nbconvert --to notebook --execute --ExecutePreprocessor.timeout=600 --output-dir /tmp/executed-notebooks notebooks/*.ipynb + - name: Build and test the installed wheel run: | wheel_dir="$(mktemp -d)" @@ -75,8 +89,11 @@ jobs: test "${#wheels[@]}" -eq 1 wheel_path="${wheels[0]}" python tests/distribution/check-wheel.py "$wheel_path" - python -m venv --system-site-packages "$wheel_dir/venv" - "$wheel_dir/venv/bin/python" -m pip install --force-reinstall --no-deps "$wheel_path" + python -m venv "$wheel_dir/venv" + ( + cd "$wheel_dir" + env -u PYTHONPATH "$wheel_dir/venv/bin/python" -m pip install "$wheel_path" + ) ( cd "$wheel_dir" env -u PYTHONPATH "$wheel_dir/venv/bin/python" "$GITHUB_WORKSPACE/tests/distribution/installed-wheel-smoke.py" diff --git a/notebooks/ExperimentalFunctions.ipynb b/notebooks/ExperimentalFunctions.ipynb index 6943e14..68ccb28 100644 --- a/notebooks/ExperimentalFunctions.ipynb +++ b/notebooks/ExperimentalFunctions.ipynb @@ -69,7 +69,7 @@ "from datetime import date\n", "\n", "# RCPCHGrowth imports\n", - "from rcpchgrowth.dynamic_growth import *\n", + "from rcpchgrowth import Measurement, acceleration, velocity\n", "\n", "# first generate two measurements\n", "\n", diff --git a/notebooks/Quickstart.ipynb b/notebooks/Quickstart.ipynb index c53a1f9..2c9882a 100644 --- a/notebooks/Quickstart.ipynb +++ b/notebooks/Quickstart.ipynb @@ -497,11 +497,11 @@ "# Create a measurement object for the child\n", "measurement = Measurement(sex=sex, birth_date=dob, measurement_method='height', observation_date=md, observation_value=height_cm, reference='uk-who', gestation_weeks=40, gestation_days=0).measurement\n", "\n", - "uk_who_chart = create_chart(reference=\"uk-who\", measurement_method=\"height\", sex=\"male\")\n", - "uk_90_preterm = uk_who_chart[0]['uk90_preterm']['male']['height']\n", - "who_infant = uk_who_chart[1]['uk_who_infant']['male']['height']\n", - "who_child = uk_who_chart[2]['uk_who_child']['male']['height']\n", - "uk90_child = uk_who_chart[3]['uk90_child']['male']['height']\n", + "uk_who_chart = create_chart(reference=\"uk-who\", measurement_method=\"height\", sex=sex)\n", + "uk_90_preterm = uk_who_chart[0]['uk90_preterm'][sex]['height']\n", + "who_infant = uk_who_chart[1]['uk_who_infant'][sex]['height']\n", + "who_child = uk_who_chart[2]['uk_who_child'][sex]['height']\n", + "uk90_child = uk_who_chart[3]['uk90_child'][sex]['height']\n", "\n", "refs = {\n", " \"uk_90_preterm\": uk_90_preterm,\n", @@ -542,16 +542,16 @@ " if ref_name==\"uk90_child\":\n", " cent_label = c.get(\"centile\")\n", " end_x, end_y = xs[-1], ys[-1]\n", - " # Slight offset to avoid covering line\n", - " plt.text(\n", - " end_x, end_y,\n", - " str(cent_label),\n", - " fontsize=7,\n", - " color=base_color,\n", - " ha=\"left\",\n", - " va=\"center\",\n", - " clip_on=True,\n", - " )\n", + " # Slight offset to avoid covering line\n", + " plt.text(\n", + " end_x, end_y,\n", + " str(cent_label),\n", + " fontsize=7,\n", + " color=base_color,\n", + " ha=\"left\",\n", + " va=\"center\",\n", + " clip_on=True,\n", + " )\n", "# add the measurement point - there is one for the chronological age, one for the corrected age\n", "# standard practice is for the corrected age to be used and rendered as cross, joined to the chronological\n", "# age which is typically represented as a filled circle\n", diff --git a/notebooks/ResearchTemplate.ipynb b/notebooks/ResearchTemplate.ipynb index d400e06..3faf949 100644 --- a/notebooks/ResearchTemplate.ipynb +++ b/notebooks/ResearchTemplate.ipynb @@ -183,14 +183,14 @@ "metadata": {}, "outputs": [], "source": [ - "from rcpchgrowth import sds_for_measurement, centile, select_reference_data_for_uk_who_chart\n", + "from rcpchgrowth import centile, sds_for_measurement\n", "\n", "def compute_sds(row, measurement_name, value_col):\n", " val = row.get(value_col)\n", " if pd.isna(val):\n", " return pd.Series({f'{measurement_name}_sds': pd.NA, f'{measurement_name}_centile': pd.NA})\n", - " ref = select_reference_data_for_uk_who_chart(measurement_name, row.sex, row.age_decimal_years)\n", - " sds_val = sds_for_measurement(measurement_name, val, ref)\n", + " sex = {'F': 'female', 'M': 'male'}.get(row.sex, row.sex)\n", + " sds_val = sds_for_measurement(reference='uk-who', age=row.age_decimal_years, measurement_method=measurement_name, observation_value=val, sex=sex)\n", " return pd.Series({f'{measurement_name}_sds': sds_val, f'{measurement_name}_centile': centile(sds_val)})\n", "\n", "for m, col in [('weight','weight_kg'), ('height','height_cm')]:\n", diff --git a/pyproject.toml b/pyproject.toml index 5f20c2d..f435d0e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -42,6 +42,7 @@ notebook = [ "matplotlib>=3.7", "jupyterlab", "ipykernel", + "nbconvert==7.17.1", ] # Everything needed to run the test suite and release tooling. This is the # SINGLE source of truth for test dependencies: every environment (local, @@ -95,8 +96,8 @@ rcpchgrowth = ["data_tables/**/*.csv", "data_tables/**/*.json", "data_tables/**/ # bump-my-version bump minor # 4.6.0 -> 4.7.0 # bump-my-version bump major # 4.6.0 -> 5.0.0 # Add --dry-run --verbose to preview. commit/tag are left off deliberately so -# the version bump stays a reviewable change, matching the existing release -# flow (manual bump commit, then a GitHub Release triggers PyPI publish). +# the version bump stays reviewable on release/vX.Y.Z. Merging that PR with a +# merge commit triggers the validated tag, GitHub Release, and publish cascade. [tool.bumpversion] current_version = "4.6.2" allow_dirty = true diff --git a/rcpchgrowth/__init__.py b/rcpchgrowth/__init__.py index 82e092a..bbc2b4e 100644 --- a/rcpchgrowth/__init__.py +++ b/rcpchgrowth/__init__.py @@ -1,3 +1,4 @@ +from ._version import __version__ from .age_advice_strings import comment_prematurity_correction from .bmi_functions import bmi_from_height_weight, weight_for_bmi_height from .cdc import select_reference_data_for_cdc_chart diff --git a/rcpchgrowth/_version.py b/rcpchgrowth/_version.py new file mode 100644 index 0000000..e208358 --- /dev/null +++ b/rcpchgrowth/_version.py @@ -0,0 +1,7 @@ +from importlib.metadata import PackageNotFoundError, version + + +try: + __version__ = version("rcpchgrowth") +except PackageNotFoundError: + __version__ = "unknown" diff --git a/rcpchgrowth/measurement.py b/rcpchgrowth/measurement.py index dd0a7c5..594b976 100644 --- a/rcpchgrowth/measurement.py +++ b/rcpchgrowth/measurement.py @@ -1,6 +1,5 @@ # standard imports from datetime import date -from importlib.metadata import PackageNotFoundError, version from typing import Literal # rcpch imports @@ -11,11 +10,7 @@ from .global_functions import sds_for_measurement, centile, percentage_median_bmi from .age_advice_strings import comment_prematurity_correction from ._build_info import COMMIT - -try: - _ENGINE_VERSION = version("rcpchgrowth") -except PackageNotFoundError: - _ENGINE_VERSION = "unknown" +from ._version import __version__ class Measurement: @@ -204,7 +199,7 @@ def __init__( 'growth_reference': self.reference, 'calculation_engine': { 'name': 'rcpch/rcpchgrowth-python', - 'version': _ENGINE_VERSION, + 'version': __version__, 'commit': COMMIT, }, }, diff --git a/rcpchgrowth/tests/test_provenance.py b/rcpchgrowth/tests/test_provenance.py index de91a59..e0de5b8 100644 --- a/rcpchgrowth/tests/test_provenance.py +++ b/rcpchgrowth/tests/test_provenance.py @@ -13,6 +13,7 @@ # third-party imports import pytest +import rcpchgrowth # rcpch imports from rcpchgrowth import Measurement, generate_fictional_child_data @@ -75,6 +76,7 @@ def test_provenance_calculation_engine_is_always_present(reference): engine = measurement["provenance"]["calculation_engine"] assert engine["name"] == "rcpch/rcpchgrowth-python" + assert engine["version"] == rcpchgrowth.__version__ assert isinstance(engine["version"], str) and engine["version"] != "" assert isinstance(engine["commit"], str) and engine["commit"] != "" From ae5ac3fc848b0d4ef88a29f2121e6c68cd9359b6 Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Fri, 4 Sep 2026 00:20:28 +0100 Subject: [PATCH 08/11] ci: automate merged release publication --- .github/scripts/validate-release.py | 180 ++++++++++++++++++++ .github/workflows/python-publish.yml | 140 +++++++++++---- .github/workflows/release-on-merge.yml | 162 ++++++++++++++++++ AGENTS.md | 4 +- s/README.md | 4 +- s/version++ | 40 ++--- spec/queries.md | 25 +++ spec/roadmap.md | 8 +- tests/distribution/test_shell_scripts.py | 15 ++ tests/distribution/test_validate_release.py | 160 +++++++++++++++++ 10 files changed, 669 insertions(+), 69 deletions(-) create mode 100644 .github/scripts/validate-release.py create mode 100644 .github/workflows/release-on-merge.yml create mode 100644 spec/queries.md create mode 100644 tests/distribution/test_shell_scripts.py create mode 100644 tests/distribution/test_validate_release.py diff --git a/.github/scripts/validate-release.py b/.github/scripts/validate-release.py new file mode 100644 index 0000000..3ea614b --- /dev/null +++ b/.github/scripts/validate-release.py @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 +"""Validate that a merged release PR is safe to publish.""" + +import argparse +import json +import re +import subprocess +from datetime import date +from pathlib import Path + + +ALLOWED_FILES = {"CITATION.cff", "pyproject.toml"} +SHA_PATTERN = re.compile(r"[0-9a-f]{40}") +VERSION_PATTERN = re.compile(r"(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)") + + +def fail(message): + raise ValueError(message) + + +def git(repo, *arguments): + return subprocess.run( + ["git", *arguments], + cwd=repo, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def toml_value(contents, section, key): + current_section = None + for line in contents.splitlines(): + section_match = re.fullmatch(r"\s*\[([^]]+)]\s*", line) + if section_match: + current_section = section_match.group(1) + continue + if current_section == section: + value_match = re.fullmatch( + rf'\s*{re.escape(key)}\s*=\s*"([^"]+)"\s*(?:#.*)?', line + ) + if value_match: + return value_match.group(1) + fail(f"Missing {key!r} in [{section}]") + + +def citation_value(contents, key): + match = re.search(rf"^{re.escape(key)}:\s*['\"]?([^'\"\s]+)['\"]?\s*$", contents, re.MULTILINE) + if not match: + fail(f"Missing {key!r} in CITATION.cff") + return match.group(1) + + +def parse_version(value): + match = VERSION_PATTERN.fullmatch(value) + if not match: + fail(f"Version is not supported semantic version X.Y.Z: {value!r}") + return tuple(int(part) for part in match.groups()) + + +def bump_kind(previous, release): + old = parse_version(previous) + new = parse_version(release) + supported = { + (old[0], old[1], old[2] + 1): "patch", + (old[0], old[1] + 1, 0): "minor", + (old[0] + 1, 0, 0): "major", + } + if new not in supported: + fail(f"Unsupported version bump: {previous} -> {release}") + return supported[new] + + +def validate_metadata(pyproject, citation, expected_version): + versions = { + "project.version": toml_value(pyproject, "project", "version"), + "tool.bumpversion.current_version": toml_value( + pyproject, "tool.bumpversion", "current_version" + ), + "CITATION.cff version": citation_value(citation, "version"), + } + mismatches = {name: value for name, value in versions.items() if value != expected_version} + if mismatches: + fail(f"Release metadata is not synchronized to {expected_version}: {mismatches}") + release_date = citation_value(citation, "date-released") + if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", release_date): + fail(f"CITATION.cff date-released is not YYYY-MM-DD: {release_date!r}") + try: + date.fromisoformat(release_date) + except ValueError: + fail(f"CITATION.cff date-released is not a valid date: {release_date!r}") + return release_date + + +def validate_pr(pr, files, repository): + if not pr.get("merged"): + fail("Release PR is not merged") + if pr.get("base", {}).get("ref") != "live": + fail("Release PR does not target live") + if pr.get("head", {}).get("repo", {}).get("full_name") != repository: + fail("Release PR must originate in the same repository") + + head_ref = pr.get("head", {}).get("ref", "") + branch_match = re.fullmatch(r"release/v(.+)", head_ref) + if not branch_match: + fail(f"Invalid release branch: {head_ref!r}") + version = branch_match.group(1) + parse_version(version) + if pr.get("title") != f"chore(release): v{version}": + fail("Release PR title does not match its version") + + changed_files = {item.get("filename") for item in files} + if changed_files != ALLOWED_FILES or len(files) != len(ALLOWED_FILES): + fail( + "Release PR must change exactly pyproject.toml and CITATION.cff; " + f"found {sorted(str(item) for item in changed_files)}" + ) + + merge_sha = pr.get("merge_commit_sha", "") + base_sha = pr.get("base", {}).get("sha", "") + head_sha = pr.get("head", {}).get("sha", "") + for name, value in (("merge", merge_sha), ("base", base_sha), ("head", head_sha)): + if not SHA_PATTERN.fullmatch(value): + fail(f"PR has an invalid {name} commit SHA") + return version, merge_sha, base_sha, head_sha + + +def validate_repository(repo, merge_sha, base_sha, head_sha, version): + head = git(repo, "rev-parse", "HEAD") + if head != merge_sha: + fail(f"Checked-out commit {head} is not PR merge commit {merge_sha}") + + commit = git(repo, "rev-list", "--parents", "-n", "1", merge_sha).split() + if len(commit) != 3: + fail("Release PR must use a two-parent merge commit") + if commit[1:] != [base_sha, head_sha]: + fail("Merge commit parents do not match the PR base and head commits") + + pyproject = git(repo, "show", f"{merge_sha}:pyproject.toml") + citation = git(repo, "show", f"{merge_sha}:CITATION.cff") + release_date = validate_metadata(pyproject, citation, version) + previous_pyproject = git(repo, "show", f"{base_sha}:pyproject.toml") + previous_version = toml_value(previous_pyproject, "project", "version") + kind = bump_kind(previous_version, version) + return previous_version, kind, release_date + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--pr-json", type=Path, required=True) + parser.add_argument("--files-json", type=Path, required=True) + parser.add_argument("--repository", required=True) + parser.add_argument("--repo", type=Path, default=Path.cwd()) + parser.add_argument("--github-output", type=Path) + args = parser.parse_args() + + pr = json.loads(args.pr_json.read_text()) + files = json.loads(args.files_json.read_text()) + version, merge_sha, base_sha, head_sha = validate_pr(pr, files, args.repository) + previous_version, kind, release_date = validate_repository( + args.repo, merge_sha, base_sha, head_sha, version + ) + result = { + "base_sha": base_sha, + "bump": kind, + "merge_sha": merge_sha, + "previous_version": previous_version, + "release_date": release_date, + "tag": f"v{version}", + "version": version, + } + if args.github_output: + with args.github_output.open("a") as output: + for name, value in result.items(): + output.write(f"{name}={value}\n") + print(json.dumps(result, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml index 9d23e8a..c12001d 100644 --- a/.github/workflows/python-publish.yml +++ b/.github/workflows/python-publish.yml @@ -1,56 +1,132 @@ -# This workflow will upload a Python Package using Twine when a release is created -# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries - -# Third-party GitHub Actions are pinned to a full 40-character commit SHA -# (with a `# vX.Y.Z` trailing comment for human readability). This protects -# against tag re-pointing supply-chain attacks: a tag like `v4` is mutable -# and can be silently rewritten to malicious code, but a commit SHA is not. -# Dependabot (see .github/dependabot.yml) recognises this pattern and keeps -# both the SHA and the version comment in sync when raising update PRs. -# -# Convention reference: -# https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-third-party-actions -# Tooling that automates the pin/unpin cycle: -# https://github.com/mheap/pin-github-action - -name: Upload Python Package +name: Build, verify, and publish Python package on: - release: - types: [created] + workflow_call: + inputs: + ref: + description: Exact commit to test and package + required: true + type: string + version: + description: Expected package version without the v prefix + required: true + type: string + publish: + description: Publish the verified artifacts to PyPI + required: false + default: false + type: boolean + +permissions: + contents: read + +concurrency: + group: python-package-${{ inputs.ref }}-${{ inputs.publish }} + cancel-in-progress: false jobs: - # Reuse the EXACT same test job that runs on every PR and push, so a release - # can never be published with a dependency/config the PR matrix did not test. test: - uses: ./.github/workflows/run-pytest-on-push-and-all-prs.yml + uses: $/.github/workflows/run-pytest-on-push-and-all-prs.yml + with: + ref: ${{ inputs.ref }} - deploy: + build: + name: Build and verify exact artifacts needs: test runs-on: ubuntu-latest - permissions: - # IMPORTANT: this permission is mandatory for Trusted Publishing - id-token: write - steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ inputs.ref }} + persist-credentials: false + + - name: Verify exact release commit + env: + EXPECTED_SHA: ${{ inputs.ref }} + EXPECTED_VERSION: ${{ inputs.version }} + run: | + test "$(git rev-parse HEAD)" = "$EXPECTED_SHA" + python - <<'PY' + import os + import pathlib + import re + + expected = os.environ["EXPECTED_VERSION"] + pyproject = pathlib.Path("pyproject.toml").read_text() + match = re.search(r'^version = "([^"]+)"$', pyproject, re.MULTILINE) + assert match and match.group(1) == expected + PY - name: Set up Python uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.13" - - name: Install build dependencies + - name: Install package tooling run: | python -m pip install --upgrade pip - pip install build twine + pip install .[dev] twine - - name: Stamp build commit SHA - run: | - echo "COMMIT = \"${GITHUB_SHA}\"" > rcpchgrowth/_build_info.py + - name: Stamp exact build commit + env: + RELEASE_SHA: ${{ inputs.ref }} + run: printf 'COMMIT = "%s"\n' "$RELEASE_SHA" > rcpchgrowth/_build_info.py - - name: Build (sdist & wheel) + - name: Build source distribution and wheel once run: python -m build + - name: Check and smoke-test exact artifacts + env: + EXPECTED_COMMIT: ${{ inputs.ref }} + EXPECTED_VERSION: ${{ inputs.version }} + run: | + sdists=(dist/*.tar.gz) + wheels=(dist/*.whl) + test "${#sdists[@]}" -eq 1 + test "${#wheels[@]}" -eq 1 + [[ "${sdists[0]}" == "dist/rcpchgrowth-${EXPECTED_VERSION}.tar.gz" ]] + [[ "${wheels[0]}" == "dist/rcpchgrowth-${EXPECTED_VERSION}-"*.whl ]] + python -m twine check dist/* + python tests/distribution/check-wheel.py "${wheels[0]}" + python -m venv /tmp/release-wheel-venv + wheel_path="$PWD/${wheels[0]}" + ( + cd /tmp + env -u PYTHONPATH /tmp/release-wheel-venv/bin/python -m pip install "$wheel_path" + ) + ( + cd /tmp + env -u PYTHONPATH /tmp/release-wheel-venv/bin/python "$GITHUB_WORKSPACE/tests/distribution/installed-wheel-smoke.py" + ) + sha256sum dist/* > artifact-sha256.txt + + - name: Retain verified artifacts + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: python-package-${{ inputs.ref }} + path: | + dist/* + artifact-sha256.txt + if-no-files-found: error + retention-days: 7 + + publish: + name: Publish verified artifacts to PyPI + if: inputs.publish + needs: build + runs-on: ubuntu-latest + permissions: + id-token: write + steps: + - name: Download verified artifacts + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: python-package-${{ inputs.ref }} + + - name: Verify retained artifact checksums + run: sha256sum --check artifact-sha256.txt + - name: Publish to PyPI uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 + with: + skip-existing: true diff --git a/.github/workflows/release-on-merge.yml b/.github/workflows/release-on-merge.yml new file mode 100644 index 0000000..0bd5a75 --- /dev/null +++ b/.github/workflows/release-on-merge.yml @@ -0,0 +1,162 @@ +name: Release merged version PR + +on: + pull_request: + branches: [live] + types: [closed] + workflow_dispatch: + inputs: + pr_number: + description: Historical merged release PR to validate without publishing + required: true + type: string + +permissions: + contents: read + pull-requests: read + +concurrency: + group: release-cascade-${{ github.event.pull_request.number || inputs.pr_number }} + cancel-in-progress: false + +jobs: + validate: + name: Validate merged release PR + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.pull_request.merged == true && startsWith(github.head_ref, 'release/v')) + runs-on: ubuntu-latest + outputs: + merge_sha: ${{ steps.validate.outputs.merge_sha }} + tag: ${{ steps.validate.outputs.tag }} + version: ${{ steps.validate.outputs.version }} + steps: + - name: Resolve merged PR + env: + EVENT_PR_NUMBER: ${{ github.event.pull_request.number }} + GH_TOKEN: ${{ github.token }} + INPUT_PR_NUMBER: ${{ inputs.pr_number }} + REPOSITORY: ${{ github.repository }} + run: | + pr_number="${EVENT_PR_NUMBER:-$INPUT_PR_NUMBER}" + [[ "$pr_number" =~ ^[1-9][0-9]*$ ]] + gh api "repos/${REPOSITORY}/pulls/${pr_number}" > "$RUNNER_TEMP/pr.json" + gh api --paginate "repos/${REPOSITORY}/pulls/${pr_number}/files?per_page=100" | jq -s add > "$RUNNER_TEMP/files.json" + + - name: Read merge commit + id: merge + run: echo "sha=$(jq -r .merge_commit_sha "$RUNNER_TEMP/pr.json")" >> "$GITHUB_OUTPUT" + + - name: Check out release validator + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + path: validator + persist-credentials: false + + - name: Check out exact release merge + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ steps.merge.outputs.sha }} + fetch-depth: 0 + path: release-source + persist-credentials: false + + - name: Validate release policy and metadata + id: validate + env: + REPOSITORY: ${{ github.repository }} + run: | + python validator/.github/scripts/validate-release.py \ + --pr-json "$RUNNER_TEMP/pr.json" \ + --files-json "$RUNNER_TEMP/files.json" \ + --repository "$REPOSITORY" \ + --repo release-source \ + --github-output "$GITHUB_OUTPUT" + + - name: Summarize dry run + if: github.event_name == 'workflow_dispatch' + env: + MERGE_SHA: ${{ steps.validate.outputs.merge_sha }} + TAG: ${{ steps.validate.outputs.tag }} + run: | + { + echo "## Release cascade dry run" + echo "Validated \`$TAG\` at exact merge commit \`$MERGE_SHA\`." + echo "Tag creation, GitHub Release creation, and PyPI publication are disabled for workflow dispatch." + } >> "$GITHUB_STEP_SUMMARY" + + release: + name: Create or verify tag and GitHub Release + if: github.event_name == 'pull_request' + needs: validate + runs-on: ubuntu-latest + permissions: + contents: write + outputs: + url: ${{ steps.release.outputs.url }} + steps: + - name: Create or verify annotated tag + env: + GH_TOKEN: ${{ github.token }} + MERGE_SHA: ${{ needs.validate.outputs.merge_sha }} + REPOSITORY: ${{ github.repository }} + TAG: ${{ needs.validate.outputs.tag }} + run: | + if tag_ref="$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}" 2>/dev/null)"; then + test "$(jq -r .object.type <<< "$tag_ref")" = "tag" + tag_object_sha="$(jq -r .object.sha <<< "$tag_ref")" + existing_sha="$(gh api "repos/${REPOSITORY}/git/tags/${tag_object_sha}" --jq .object.sha)" + if [[ "$existing_sha" != "$MERGE_SHA" ]]; then + echo "Refusing to overwrite $TAG at $existing_sha; expected $MERGE_SHA." >&2 + exit 1 + fi + echo "Reusing annotated $TAG at $MERGE_SHA." + else + tag_object_sha="$(gh api --method POST "repos/${REPOSITORY}/git/tags" \ + -f tag="$TAG" -f message="$TAG" -f object="$MERGE_SHA" -f type=commit --jq .sha)" + gh api --method POST "repos/${REPOSITORY}/git/refs" \ + -f ref="refs/tags/${TAG}" -f sha="$tag_object_sha" >/dev/null + echo "Created annotated $TAG at $MERGE_SHA." + fi + + - name: Create or verify GitHub Release + id: release + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + TAG: ${{ needs.validate.outputs.tag }} + run: | + if release="$(gh api "repos/${REPOSITORY}/releases/tags/${TAG}" 2>/dev/null)"; then + test "$(jq -r .draft <<< "$release")" = "false" + test "$(jq -r .prerelease <<< "$release")" = "false" + url="$(jq -r .html_url <<< "$release")" + echo "Reusing $url." + else + url="$(gh release create "$TAG" --repo "$REPOSITORY" --verify-tag --title "$TAG" --generate-notes)" + fi + echo "url=$url" >> "$GITHUB_OUTPUT" + + publish-production: + name: Test, build, and publish + if: github.event_name == 'pull_request' + needs: [validate, release] + permissions: + contents: read + id-token: write + uses: $/.github/workflows/python-publish.yml + with: + ref: ${{ needs.validate.outputs.merge_sha }} + version: ${{ needs.validate.outputs.version }} + publish: true + + verify-dry-run: + name: Test and build without publishing + if: github.event_name == 'workflow_dispatch' + needs: validate + permissions: + contents: read + uses: $/.github/workflows/python-publish.yml + with: + ref: ${{ needs.validate.outputs.merge_sha }} + version: ${{ needs.validate.outputs.version }} + publish: false diff --git a/AGENTS.md b/AGENTS.md index 72be424..1d3d495 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,9 +75,9 @@ This collection-time filtering replaced 1,700 runtime skips (1,655 out-of-range Use `s/version++ [patch|minor|major]` to prepare a release version bump; it defaults to `patch`. The script is optional, but it is the canonical automated path: from a clean and up-to-date `live` branch it runs the full suite, creates `release/vX.Y.Z`, synchronizes `pyproject.toml` and `CITATION.cff`, validates the package build, commits `chore(release): vX.Y.Z`, pushes the branch, and opens a PR. Use `--dry-run` to preview without changing anything. -The script deliberately does not tag or publish because `live` is protected and the release tag must point to the reviewed PR's exact merge commit. After the PR is merged, follow the terminal runbook printed by the script to resolve that commit and create the GitHub Release; that release event triggers `.github/workflows/python-publish.yml`. Do not push directly to `live` or create the release before the PR merges. See [`s/README.md`](s/README.md) for usage. +The script deliberately does not tag or publish because `live` is protected and the release tag must point to the reviewed PR's exact merge commit. Review CI and merge the release PR using a merge commit; that merge is the final human release action. `.github/workflows/release-on-merge.yml` validates the PR and exact merge commit, creates or reuses the annotated tag and GitHub Release without overwriting conflicts, and directly invokes the reusable `.github/workflows/python-publish.yml` path. Do not push directly to `live`, manually create release tags, or use squash/rebase merging for release PRs. See [`s/README.md`](s/README.md) for usage and the historical non-publishing dry run. -The manual post-merge tag and GitHub Release are a documented project-specific exception to the house-style CI auto-tag cascade. They remain necessary while the existing publisher is triggered by `release: created`; do not introduce a second release command or automatic tag path without migrating the publishing workflow as one coherent change. +Release PRs may change only `pyproject.toml` and `CITATION.cff`, must carry exactly one supported patch, minor, or major bump, and must be merged from the same repository into `live` with a merge commit. Repository settings may disable squash/rebase globally, but the release workflow enforces merge-commit policy independently. Keep PyPI Trusted Publishing bound to `.github/workflows/python-publish.yml`; introducing a protected environment requires a coordinated PyPI trusted-publisher update. ## Key Code Locations diff --git a/s/README.md b/s/README.md index 54f43fa..74aaad5 100644 --- a/s/README.md +++ b/s/README.md @@ -30,6 +30,6 @@ The `s/` directory contains convenience scripts that standardise repeated develo - `s/version++ major` - prepare a major bump. - `s/version++ minor --dry-run` - preview the proposed version and workflow without changing anything. -The script does not tag or publish. After the release PR is merged, it prints a runbook containing commands that resolve the PR's exact merge commit and create the GitHub Release from that commit. The release event triggers the existing PyPI workflow; the runbook also provides links for checking GitHub Actions, PyPI, and Zenodo. +The script does not tag or publish. Review CI and merge the release PR using a merge commit; that merge is the final human release action. `.github/workflows/release-on-merge.yml` then validates the same-repository PR, its permitted files, supported version bump, synchronized metadata, and exact two-parent merge commit before creating or reusing the annotated tag and GitHub Release. It directly invokes `.github/workflows/python-publish.yml`, avoiding `GITHUB_TOKEN` event anti-recursion, and publishes only the exact commit-stamped artifacts that passed package checks and an isolated installed-wheel smoke test. -This is a project-specific exception to the house-style CI auto-tag cascade. The existing publisher is intentionally triggered by a manually reviewed GitHub Release, so the post-merge tag and Release remain explicit runbook steps until the publishing workflow is migrated to an auto-tag and `workflow_call` design. +To exercise the complete validation, Python 3.10-3.13 test matrix, build, and artifact checks without creating a tag, GitHub Release, or PyPI publication, manually run the `Release merged version PR` workflow with the number of a merged release PR. Manual dispatch is always non-publishing. The first end-to-end historical dispatch can only run after this workflow has merged to `live`; use a release PR whose exact commit satisfies the current quality gates. diff --git a/s/version++ b/s/version++ index 9b9b668..65fa414 100755 --- a/s/version++ +++ b/s/version++ @@ -5,8 +5,8 @@ # Optionally prepare a release version bump from a clean, up-to-date live branch. # The script runs the full test suite, updates every configured version field, # validates a source/wheel build, commits the result, and opens a release PR. -# It never tags or publishes: after review and merge it prints the exact manual -# tag and GitHub Release commands that trigger the existing PyPI workflow. +# It never tags or publishes. Merging the validated release PR with a merge +# commit triggers the release cascade, so the merge is the final human action. # # Requires Docker, GitHub CLI authentication, and current dev dependencies in # the container. Run `s/rebuild` if the bump/build tool preflight fails. @@ -22,8 +22,7 @@ usage() { Usage: s/version++ [patch|minor|major] [--dry-run] Prepares a release PR from live. The bump defaults to patch. -After the PR is merged, the script's terminal runbook explains how to create -the GitHub Release that triggers publication to PyPI. +Merging the validated PR with a merge commit triggers tagging and publication. Options: --dry-run Show the proposed version and workflow without changing anything. @@ -73,7 +72,7 @@ An actual run will: 4. Update pyproject.toml and CITATION.cff with bump-my-version. 5. Validate synchronized metadata and build the sdist and wheel. 6. Commit chore(release): v${new_version}, push, and open a PR to live. - 7. Print the manual post-merge GitHub Release and verification runbook. + 7. Print the final merge command and automated-release verification links. EOF exit 0 fi @@ -196,7 +195,7 @@ pr_url="$(gh pr create \ --base live \ --head "$release_branch" \ --title "chore(release): v${new_version}" \ - --body "Prepare v${new_version}: synchronize package/citation metadata, pass the full local test suite, and validate the sdist and wheel build. After merge, tag the PR's exact merge commit and create the GitHub Release to trigger PyPI publication.")" + --body "Prepare v${new_version}: synchronize package/citation metadata, pass the full local test suite, and validate the sdist and wheel build. Merge this PR with a merge commit to trigger the validated tag, GitHub Release, artifact checks, and PyPI publication cascade.")" trap - ERR @@ -205,34 +204,17 @@ cat < Date: Fri, 4 Sep 2026 10:39:53 +0100 Subject: [PATCH 09/11] chore: prune gitignore to repository tooling --- .gitignore | 152 ++++++----------------------------------------------- 1 file changed, 17 insertions(+), 135 deletions(-) diff --git a/.gitignore b/.gitignore index a863883..57a04a6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,149 +1,31 @@ -# Byte-compiled / optimized / DLL files +# Python __pycache__/ *.py[cod] -*$py.class - -# C extensions -*.so - -# Distribution / packaging -.Python +*.egg-info/ build/ -develop-eggs/ dist/ -downloads/ -eggs/ -.eggs/ -lib/ -lib64/ -parts/ -sdist/ -var/ -wheels/ -pip-wheel-metadata/ -share/python-wheels/ -*.egg-info/ -.installed.cfg -*.egg -MANIFEST - -# PyInstaller -# Usually these files are written by a python script from a template -# before PyInstaller builds the exe, so as to inject date/other infos into it. -*.manifest -*.spec - -# Installer logs -pip-log.txt -pip-delete-this-directory.txt - -# Unit test / coverage reports -htmlcov/ -.tox/ -.nox/ -.coverage -.coverage.* -.cache -nosetests.xml -coverage.xml -*.cover -*.py,cover -.hypothesis/ -.pytest_cache/ - -# Translations -*.mo -*.pot - -# Django stuff: -*.log -local_settings.py -db.sqlite3 -db.sqlite3-journal - -# Flask stuff: -instance/ -.webassets-cache - -# Scrapy stuff: -.scrapy - -# Sphinx documentation -docs/_build/ - -# PyBuilder -target/ - -# Jupyter Notebook -.ipynb_checkpoints - -# IPython -profile_default/ -ipython_config.py - -# pyenv -.python-version - -# pipenv -# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. -# However, in case of collaboration, if having platform-specific dependencies or dependencies -# having no cross-platform support, pipenv may install dependencies that don't work, or not -# install all needed dependencies. -#Pipfile.lock - -# PEP 582; used by e.g. github.com/David-OConnor/pyflow -__pypackages__/ - -# Celery stuff -celerybeat-schedule -celerybeat.pid - -# SageMath parsed files -*.sage.py # Environments .env -.venv -env/ -venv/ -ENV/ -env.bak/ -venv.bak/ - -# Spyder project settings -.spyderproject -.spyproject +.venv/ -# Rope project settings -.ropeproject - -# mkdocs documentation -/site - -# mypy -.mypy_cache/ -.dmypy.json -dmypy.json - -# Pyre type checker -.pyre/ +# Testing and linting +.pytest_cache/ +.coverage +.ruff_cache/ -.vscode -.pytest/cache +# Jupyter +.ipynb_checkpoints/ -# MacOS DS_Store files +# Editor / OS +.vscode/ .DS_Store -# Fenton data is licenced and not to be shared -rcpchgrowth/data_tables/fenton/*.* - -# Private files -.private - -# Disposable agent and browser-automation output +# Agent tooling (Claude Code, etc.) .claude/ -.playwright-mcp/ +.private/ .playwright/ -playwright-report/ -blob-report/ -test-results/ +.playwright-mcp/ + +# Licensed data - Fenton chart data must not be shared +rcpchgrowth/data_tables/fenton/*.* From 0f185a04a73565a78e475f91f2bf9ca10e9d7251 Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Fri, 4 Sep 2026 12:57:06 +0100 Subject: [PATCH 10/11] chore(deps): raise maintained dependency floors --- pyproject.toml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index f435d0e..61767f3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ maintainers = [ requires-python = ">=3.10" dependencies = [ "python-dateutil", - "scipy", + "scipy>=1.15", ] classifiers = [ "Development Status :: 5 - Production/Stable", @@ -38,8 +38,8 @@ Documentation = "https://growth.rcpch.ac.uk/products/python-library/" [project.optional-dependencies] notebook = [ - "pandas>=1.5", - "matplotlib>=3.7", + "pandas>=2.2", + "matplotlib>=3.10", "jupyterlab", "ipykernel", "nbconvert==7.17.1", @@ -50,7 +50,7 @@ notebook = [ # drift apart. `pandas` is required by rcpchgrowth/tests/test_who.py. dev = [ "pytest", - "pandas>=1.5", + "pandas>=2.2", "bump-my-version", "build", "ruff>=0.16.5", From db7d1d6bf93dec24d09adff0ecd7c851904a12b8 Mon Sep 17 00:00:00 2001 From: Marcus Baw Date: Sat, 5 Sep 2026 18:53:55 +0100 Subject: [PATCH 11/11] docs: roadmap data-derived reference boundaries --- spec/roadmap.md | 1 + 1 file changed, 1 insertion(+) diff --git a/spec/roadmap.md b/spec/roadmap.md index 257f1d9..f633488 100644 --- a/spec/roadmap.md +++ b/spec/roadmap.md @@ -37,6 +37,7 @@ Complete these items before broad formatting, import cleanup, packaging reductio ## Medium priority +- [ ] **R25 - Derive accepted age domains from reference metadata and table extents.** Replace duplicated age-limit literals with a central domain model for every reference segment, measurement, and sex. Distinguish observed table extent from clinically supported applicability, including endpoint inclusivity and overlap precedence for hybrid references such as UK-WHO and WHO. Add build-time checks against canonical source material in `rcpch/growth-references`, prohibit extrapolation beyond selected series, and test every first/last accepted endpoint plus `math.nextafter()` immediately outside it. Track implementation in issue #109. - [ ] **R3 - Decide the future of the unusable `BoneAge` class.** `rcpchgrowth/bone_age.py:12-48` returns a dictionary from `__init__`, so construction raises `TypeError: __init__() should return None`. The class has no internal consumers and is not exported from the package root, although related fields are supported by `Measurement`. Either implement and test a coherent object API or remove the module after checking external compatibility. - [ ] **R4 - Define an explicit runtime package-data allowlist.** The published installation occupies roughly 15 MB and ships spreadsheets, a PDF, VBA source, validation CSVs, old WHO tables, experimental LMS tables, and other provenance material because `pyproject.toml:68-69` broadly includes file extensions beneath `data_tables`. Identify the JSON and correlation resources required at runtime, package only those files, and verify wheel and sdist contents in CI. - [ ] **R5 - Exclude the test package from release wheels.** `pyproject.toml:60-62` discovers `rcpchgrowth*`, which includes 19 modules beneath `rcpchgrowth/tests` in the published wheel. Exclude `rcpchgrowth.tests` and its subpackages while retaining all tests in the source repository and sdist if deliberately required there.