From bbb15b2a2e8f0af60f062ac6445db0e730e78212 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 14 Jun 2026 00:47:05 +0200 Subject: [PATCH 1/3] Release 0.2.0 (#36) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(data): shared download manifest for artefacts + binaries (#16) * feat(data): shared download manifest for artefacts and binaries Introduce a single, language-agnostic manifest (data/manifest.schema.json) that lists every downloadable data artefact and external-binary bundle with a SHA256, consumed by both raven-python and (via the same JSON) MATLAB RAVEN. The manifest is a superset of the two runtime registries: * manifest["data"] -> raven_python.data._DATA_REGISTRY * manifest["binaries"] -> raven_python.binaries._REGISTRY Added: * data/manifest.schema.json (JSON Schema) + data/manifest.example.json (worked example) + data/manifest.json (empty, the live source of truth until assets are published). * raven_python.manifest — load_manifest / to_*_registry / load_into_registries. * Lazy autoload: data.ensure_* and binaries.ensure_binary populate themselves from $RAVEN_PYTHON_MANIFEST on first use when their registry is still empty (guarded; no effect when a registry is passed explicitly or the env var is unset). * scripts/make_registry_snippet.py: a `manifest` subcommand that computes url+sha256+bytes and writes/updates manifest.json. * tests/test_manifest.py (round-trip, converters, lazy autoload via file:// URLs, repo manifests valid). * docs/maintenance/data_manifest.md — format, Python + MATLAB consumers, GitHub-Releases vs Zenodo hosting (incl. a release→Zenodo GitHub Action), and per-asset recommendations. * docs(data): host assets on existing-repo releases; KEGG redistribution permitted Reflect the chosen distribution model: GitHub release assets live outside the git tree, so a separate data repository is optional — attach assets to dedicated tags (e.g. kegg-kegg116, diamond-2.1.9) on an existing RAVEN repo and reuse the same URLs across raven-python and MATLAB RAVEN. Use Zenodo only for DOIs or files >2 GB. KEGG artefacts are redistributed with permission, so the prior 'confirm rights' caveat is removed. Example/schema URLs repointed from a hypothetical raven-data repo to raven-python. * Add yeast-GEM-derived shared modules (diff, annotation, conditions, biomass, curation) (#17) * Add diff_models, annotation, and conditions modules for yeast-GEM port Lands the upstream-shareable pieces that yeast-GEM has been implementing locally during its Python port (see yeast-GEM/code/python/PORTING_PLAN.md and UPSTREAM_CANDIDATES.md). These are organism-agnostic; yeast-GEM will consume them via a Python dependency on raven-python. New modules ----------- raven_python.comparison.diff diff_models(a, b, ...) -> DiffReport — strict two-model semantic- equality diff. Complements the existing compare_models (N-model presence-matrix overview). Used as a CI gate to verify that two toolchains (e.g. MATLAB RAVEN vs raven_python, pre/post refactor of one toolchain) produce equivalent models. Includes a python -m raven_python.comparison.diff CLI. raven_python.annotation.sbo add_sbo_terms — SBO term assignment with "fill" semantic. Default parameter set reproduces yeast-GEM's behaviour; biomass metabolite names, biomass/NGAM reaction names, and pseudoreaction substrings are overridable. Transport detection is pluggable (default: same- met-name in two compartments). Includes an `only_last_reaction_ for_pseudo` legacy bug-compat flag for yeast-GEM's lock-step migration; off by default for any new caller. raven_python.annotation.delta_g load_delta_g_csv / save_delta_g_csv — generic side-car CSV persistence for scalar notes (ΔG by default, but the note key, column names, and id/value mapping are all configurable). raven_python.conditions.apply apply_condition(model, yaml_or_dict) — generic "apply this YAML condition" loader. Schema: prelude (reset_exchanges), cofactor_pseudoreaction (remove_mets + charge_balance_met), biomass_stoichiometry_delta, per-rxn bounds, expected_uptake_count. Project-specific extensions (e.g. yeast-GEM's amino_acid_ratio) are handled by the caller before/after this function — kept upstream-narrow on purpose. Also exposes set_reaction_bounds helper that bypasses cobra's lb<=ub validator for the (legitimate) cases where a condition lands on an infeasible bound state. Tests ----- 46 new tests across the three modules; full pre-existing raven-python suite still passes (519 passed; 1 unrelated pre-existing openpyxl ImportError in tests/test_io_git.py; 2 skipped). ruff clean. Not in this commit ------------------ The biomass / GAM / chemostat / fit_gam modules are still tracked as upstream candidates in yeast-GEM/code/python/UPSTREAM_CANDIDATES.md and remain local in yeast-GEM until phase 4 of the port (which would ideally land them directly here). * Add raven_python.biomass — sum / scale / rescale / set_gam Generic biomass-equation manipulation, extracted from yeast-GEM's sumBioMass / scaleBioMass / rescalePseudoReaction / changeGAM as yeast-GEM moves to depend on raven-python (yeast-GEM phase 4 of the porting plan). Module layout ------------- raven_python.biomass.config BiomassConfig — biomass_rxn id + proton_met id + ordered tuple of BiomassComponent (per-component pseudoreaction name + mass- computation strategy). raven_python.biomass.scale sum_biomass(model, config) → {component: g/gDW, ..., "total": X} rescale_pseudoreaction(model, config, name, factor) — multiply the pseudoreaction's substrate coefs by factor and rebalance H+ to keep ionic charge at zero. scale_biomass(model, config, name, new_value, balance_out=None) — rescale a component to a target fraction, optionally balancing a second component so the total stays at 1 g/gDW. raven_python.biomass.gam set_gam(model, value, *, biomass_rxn, cofactor_met_names, ngam_rxn=None, ngam_value=None) — scales every metabolite in the biomass pseudoreaction whose `name` is in the supplied cofactor set, preserving its sign; optionally fixes the NGAM rxn bounds. Mass strategies (per BiomassComponent.mass_strategy): "mw" plain MW from chemical formula (carbohydrate / ion / cofactor) "mw_minus_2h" MW − 2.016 g/mol per substrate (protein — charged tRNAs release two protons) "mw_minus_water" MW − 18.015 g/mol per substrate (RNA / DNA — polymerisation releases one water) "grams" stoichiometry already in g/gDW (lipid backbone) Tests: 19 new tests over a synthetic toy model that exercises every mass strategy, the H+ charge rebalance, scale_biomass with and without balance_out, set_gam on cofactor mets (and the NGAM bound path). * Add raven_python.manipulation.find_duplicate_reactions (detection variant) Detection-only counterpart to remove_duplicate_reactions. Returns duplicate groups instead of mutating the model. Ignores bounds / GPR / objective — only stoichiometry is compared, mirroring the typical curation use case ("find reactions that could be merged"). A new ``ignore_direction=True`` default (yeast-GEM convention) treats A→B and B→A as duplicates. Set False to require identical orientation. Used by yeast-GEM's modelTests port (Tier 3 / phase 5) to flag duplicate reactions during curation review. * Add raven_python.curation — batch_curate / batch_curate_from_tsv Generic batch curation engine extracted from yeast-GEM's MATLAB curateMetsRxnsGenes (yeast-GEM phase 6). Adds or updates metabolites, reactions and genes from pandas DataFrames; a batch_curate_from_tsv convenience wrapper reads the equivalent TSVs. Schema (matches yeast-GEM's data/modelCuration/template/ layout): mets_df metNames, comps, formula, charge, inchi, metNotes + any number of MIRIAM-namespace columns genes_df genes, geneShortNames + MIRIAM columns rxns_df rxnNames, grRules, lb, ub, rev, subSystems, eccodes, rxnNotes, rxnReferences, rxnConfidenceScores + MIRIAM columns rxns_coeffs_df rxnNames, metNames, comps, coefficient (one row per (reaction, metabolite) pair) Match keys: metabolites — (name, compartment) tuple genes — gene id reactions — stoichiometric signature Existing entities get their annotations overwritten (warning emitted); new entities are added with fresh ids generated from the supplied ``met_id_prefix`` / ``rxn_id_prefix`` (defaults M_ / R_ per the BiGG convention; yeast-GEM passes s_ / r_). Width of the existing zero-padded suffix is preserved so s_0001 → s_0002, not s_2. "Everything after the core columns is MIRIAM" — the header of any extra column becomes the annotation namespace key. Matches MATLAB behaviour exactly so yeast-GEM's existing TSVs work unchanged, and projects with different MIRIAM column sets need no code change. CurationResult dataclass records what was added vs updated so callers can verify in tests / CI. Tests: 13 new (add/update mets, add/update genes, add/update rxns by stoichiometry, miriam auto-detect, id-width preservation, combined mets+rxns in one call, missing-metabolite error, batch_curate_from_tsv round trip). * io.yaml: byte-compatible round-trip with cobrapy + RAVEN MATLAB Three things this fixes: 1. write_yaml_model dropped the !!omap tags entirely. _to_plain was flattening cobra's OrderedDict to plain dict, which causes ruamel to emit ordinary block mappings. RAVEN MATLAB's reader is a line-based parser keyed on !!omap and therefore could not load any file we wrote. _to_plain now returns OrderedDict so ruamel re-emits the !!omap tag. 2. eccodes was lost on round-trip — it wasn't in _RXN_FIELDS, so read_yaml_model didn't capture it into .notes and write_yaml_model couldn't lift it back. Added. 3. RAVEN MATLAB writes reaction notes as 'rxnNotes'; cobrapy and this writer use 'notes'. Added a read-time alias so existing yeast-GEM YAML files (which still say 'rxnNotes') load cleanly. Writes go out as 'notes' (cobrapy-canonical). Top-level layout now matches RAVEN MATLAB: metaData first, then metabolites / reactions / genes / compartments, then optional gecko_light + ec-rxns + ec-enzymes. id/name/version live inside metaData (RAVEN convention) — cobrapy reading these files still works, but cobra_model.id ends up None because cobrapy doesn't know about metaData. raven_python.read_yaml_model lifts both metaData.id/name/version onto model.id / model.name / model.notes['version'] so the rest of the codebase doesn't care which layout the file used. Empty-name genes are no longer emitted as — that's a cobrapy quirk that drifts yeast-GEM YAML files away from RAVEN MATLAB's output. Verified end-to-end: * cobra.io.load_yaml_model reads every file the new writer produces (yeast-GEM and a synthetic fixture). * RAVEN MATLAB readYAMLmodel reads every file the new writer produces. * Round-tripping yeast-GEM through raven_python preserves 2748/2748 metabolites, 4102/4102 reactions, 1143/1143 genes, 2411 eccodes, 3984 reaction deltaG, 2696 metabolite deltaG, 1788 SMILES, 1443 rxn-notes — no semantic drift. Tests ----- * tests/test_io_yaml_parity.py is new: covers every RAVEN extension, the rxnNotes legacy alias, the SMILES YAML-special character case, metaData-first layout, and cobra readability. * tests/test_io_yaml.py::test_output_is_cobra_readable adjusts for the metaData layout (cobra recovers mets/rxns/annotation but not model.id, by design). * conditions: switch from PyYAML to ruamel.yaml PyYAML is not a project dependency; raven-python uses ruamel.yaml (already pulled in via cobra) everywhere else. The conditions module and its tests still imported PyYAML, which broke pytest collection on clean CI runners with 'No module named yaml'. Both apply.py and the test now use a YAML(typ='safe') instance from ruamel.yaml — same plain-dict semantics as PyYAML's safe_load / safe_dump, no new dependency. * io.yaml: document the format + accept legacy geckoLight-in-metaData Adds docs/reference/yaml_format.md as the canonical schema reference for the cross-toolchain YAML format (cobrapy / raven-python / RAVEN MATLAB). Covers the full document shape, per-entry field order, RAVEN extensions, the GECKO ec-* sections, the metaData provenance block, number / string / quoting rules, and the cross-reader interoperability matrix. Linked from docs/reference/index.md and the I/O guide. Reader fix: pre-shim RAVEN MATLAB writes emitted GECKO models with geckoLight: "true" inside the metaData block (not as a top-level gecko_light). The reader now lifts that legacy key out of metaData so model.ec.gecko_light is populated whichever placement the file used. Round-trip writes always use the new top-level form. Regression tests: test_pre_shim_format_loads — synthetic fixture covering every legacy quirk we know about (--- doc marker, plain metaData, geckoLight inside metaData, top-level metabolite smiles, rxnNotes reaction key, integer bounds, double-quoted strings). Each quirk has its own assertion + comment. test_pre_shim_yeast_gem_loads_if_available — sanity-loads the real yeast-GEM.yml (2748 mets, 4102 rxns, 1143 genes) and asserts the documented preserved-counts table from the format reference. Skipped on CI runners where the working copy isn't mounted. * Cobra-aligned hardening pass from full code review (#18) * Add diff_models, annotation, and conditions modules for yeast-GEM port Lands the upstream-shareable pieces that yeast-GEM has been implementing locally during its Python port (see yeast-GEM/code/python/PORTING_PLAN.md and UPSTREAM_CANDIDATES.md). These are organism-agnostic; yeast-GEM will consume them via a Python dependency on raven-python. New modules ----------- raven_python.comparison.diff diff_models(a, b, ...) -> DiffReport — strict two-model semantic- equality diff. Complements the existing compare_models (N-model presence-matrix overview). Used as a CI gate to verify that two toolchains (e.g. MATLAB RAVEN vs raven_python, pre/post refactor of one toolchain) produce equivalent models. Includes a python -m raven_python.comparison.diff CLI. raven_python.annotation.sbo add_sbo_terms — SBO term assignment with "fill" semantic. Default parameter set reproduces yeast-GEM's behaviour; biomass metabolite names, biomass/NGAM reaction names, and pseudoreaction substrings are overridable. Transport detection is pluggable (default: same- met-name in two compartments). Includes an `only_last_reaction_ for_pseudo` legacy bug-compat flag for yeast-GEM's lock-step migration; off by default for any new caller. raven_python.annotation.delta_g load_delta_g_csv / save_delta_g_csv — generic side-car CSV persistence for scalar notes (ΔG by default, but the note key, column names, and id/value mapping are all configurable). raven_python.conditions.apply apply_condition(model, yaml_or_dict) — generic "apply this YAML condition" loader. Schema: prelude (reset_exchanges), cofactor_pseudoreaction (remove_mets + charge_balance_met), biomass_stoichiometry_delta, per-rxn bounds, expected_uptake_count. Project-specific extensions (e.g. yeast-GEM's amino_acid_ratio) are handled by the caller before/after this function — kept upstream-narrow on purpose. Also exposes set_reaction_bounds helper that bypasses cobra's lb<=ub validator for the (legitimate) cases where a condition lands on an infeasible bound state. Tests ----- 46 new tests across the three modules; full pre-existing raven-python suite still passes (519 passed; 1 unrelated pre-existing openpyxl ImportError in tests/test_io_git.py; 2 skipped). ruff clean. Not in this commit ------------------ The biomass / GAM / chemostat / fit_gam modules are still tracked as upstream candidates in yeast-GEM/code/python/UPSTREAM_CANDIDATES.md and remain local in yeast-GEM until phase 4 of the port (which would ideally land them directly here). * Add raven_python.biomass — sum / scale / rescale / set_gam Generic biomass-equation manipulation, extracted from yeast-GEM's sumBioMass / scaleBioMass / rescalePseudoReaction / changeGAM as yeast-GEM moves to depend on raven-python (yeast-GEM phase 4 of the porting plan). Module layout ------------- raven_python.biomass.config BiomassConfig — biomass_rxn id + proton_met id + ordered tuple of BiomassComponent (per-component pseudoreaction name + mass- computation strategy). raven_python.biomass.scale sum_biomass(model, config) → {component: g/gDW, ..., "total": X} rescale_pseudoreaction(model, config, name, factor) — multiply the pseudoreaction's substrate coefs by factor and rebalance H+ to keep ionic charge at zero. scale_biomass(model, config, name, new_value, balance_out=None) — rescale a component to a target fraction, optionally balancing a second component so the total stays at 1 g/gDW. raven_python.biomass.gam set_gam(model, value, *, biomass_rxn, cofactor_met_names, ngam_rxn=None, ngam_value=None) — scales every metabolite in the biomass pseudoreaction whose `name` is in the supplied cofactor set, preserving its sign; optionally fixes the NGAM rxn bounds. Mass strategies (per BiomassComponent.mass_strategy): "mw" plain MW from chemical formula (carbohydrate / ion / cofactor) "mw_minus_2h" MW − 2.016 g/mol per substrate (protein — charged tRNAs release two protons) "mw_minus_water" MW − 18.015 g/mol per substrate (RNA / DNA — polymerisation releases one water) "grams" stoichiometry already in g/gDW (lipid backbone) Tests: 19 new tests over a synthetic toy model that exercises every mass strategy, the H+ charge rebalance, scale_biomass with and without balance_out, set_gam on cofactor mets (and the NGAM bound path). * Add raven_python.manipulation.find_duplicate_reactions (detection variant) Detection-only counterpart to remove_duplicate_reactions. Returns duplicate groups instead of mutating the model. Ignores bounds / GPR / objective — only stoichiometry is compared, mirroring the typical curation use case ("find reactions that could be merged"). A new ``ignore_direction=True`` default (yeast-GEM convention) treats A→B and B→A as duplicates. Set False to require identical orientation. Used by yeast-GEM's modelTests port (Tier 3 / phase 5) to flag duplicate reactions during curation review. * Add raven_python.curation — batch_curate / batch_curate_from_tsv Generic batch curation engine extracted from yeast-GEM's MATLAB curateMetsRxnsGenes (yeast-GEM phase 6). Adds or updates metabolites, reactions and genes from pandas DataFrames; a batch_curate_from_tsv convenience wrapper reads the equivalent TSVs. Schema (matches yeast-GEM's data/modelCuration/template/ layout): mets_df metNames, comps, formula, charge, inchi, metNotes + any number of MIRIAM-namespace columns genes_df genes, geneShortNames + MIRIAM columns rxns_df rxnNames, grRules, lb, ub, rev, subSystems, eccodes, rxnNotes, rxnReferences, rxnConfidenceScores + MIRIAM columns rxns_coeffs_df rxnNames, metNames, comps, coefficient (one row per (reaction, metabolite) pair) Match keys: metabolites — (name, compartment) tuple genes — gene id reactions — stoichiometric signature Existing entities get their annotations overwritten (warning emitted); new entities are added with fresh ids generated from the supplied ``met_id_prefix`` / ``rxn_id_prefix`` (defaults M_ / R_ per the BiGG convention; yeast-GEM passes s_ / r_). Width of the existing zero-padded suffix is preserved so s_0001 → s_0002, not s_2. "Everything after the core columns is MIRIAM" — the header of any extra column becomes the annotation namespace key. Matches MATLAB behaviour exactly so yeast-GEM's existing TSVs work unchanged, and projects with different MIRIAM column sets need no code change. CurationResult dataclass records what was added vs updated so callers can verify in tests / CI. Tests: 13 new (add/update mets, add/update genes, add/update rxns by stoichiometry, miriam auto-detect, id-width preservation, combined mets+rxns in one call, missing-metabolite error, batch_curate_from_tsv round trip). * io.yaml: byte-compatible round-trip with cobrapy + RAVEN MATLAB Three things this fixes: 1. write_yaml_model dropped the !!omap tags entirely. _to_plain was flattening cobra's OrderedDict to plain dict, which causes ruamel to emit ordinary block mappings. RAVEN MATLAB's reader is a line-based parser keyed on !!omap and therefore could not load any file we wrote. _to_plain now returns OrderedDict so ruamel re-emits the !!omap tag. 2. eccodes was lost on round-trip — it wasn't in _RXN_FIELDS, so read_yaml_model didn't capture it into .notes and write_yaml_model couldn't lift it back. Added. 3. RAVEN MATLAB writes reaction notes as 'rxnNotes'; cobrapy and this writer use 'notes'. Added a read-time alias so existing yeast-GEM YAML files (which still say 'rxnNotes') load cleanly. Writes go out as 'notes' (cobrapy-canonical). Top-level layout now matches RAVEN MATLAB: metaData first, then metabolites / reactions / genes / compartments, then optional gecko_light + ec-rxns + ec-enzymes. id/name/version live inside metaData (RAVEN convention) — cobrapy reading these files still works, but cobra_model.id ends up None because cobrapy doesn't know about metaData. raven_python.read_yaml_model lifts both metaData.id/name/version onto model.id / model.name / model.notes['version'] so the rest of the codebase doesn't care which layout the file used. Empty-name genes are no longer emitted as — that's a cobrapy quirk that drifts yeast-GEM YAML files away from RAVEN MATLAB's output. Verified end-to-end: * cobra.io.load_yaml_model reads every file the new writer produces (yeast-GEM and a synthetic fixture). * RAVEN MATLAB readYAMLmodel reads every file the new writer produces. * Round-tripping yeast-GEM through raven_python preserves 2748/2748 metabolites, 4102/4102 reactions, 1143/1143 genes, 2411 eccodes, 3984 reaction deltaG, 2696 metabolite deltaG, 1788 SMILES, 1443 rxn-notes — no semantic drift. Tests ----- * tests/test_io_yaml_parity.py is new: covers every RAVEN extension, the rxnNotes legacy alias, the SMILES YAML-special character case, metaData-first layout, and cobra readability. * tests/test_io_yaml.py::test_output_is_cobra_readable adjusts for the metaData layout (cobra recovers mets/rxns/annotation but not model.id, by design). * conditions: switch from PyYAML to ruamel.yaml PyYAML is not a project dependency; raven-python uses ruamel.yaml (already pulled in via cobra) everywhere else. The conditions module and its tests still imported PyYAML, which broke pytest collection on clean CI runners with 'No module named yaml'. Both apply.py and the test now use a YAML(typ='safe') instance from ruamel.yaml — same plain-dict semantics as PyYAML's safe_load / safe_dump, no new dependency. * io.yaml: document the format + accept legacy geckoLight-in-metaData Adds docs/reference/yaml_format.md as the canonical schema reference for the cross-toolchain YAML format (cobrapy / raven-python / RAVEN MATLAB). Covers the full document shape, per-entry field order, RAVEN extensions, the GECKO ec-* sections, the metaData provenance block, number / string / quoting rules, and the cross-reader interoperability matrix. Linked from docs/reference/index.md and the I/O guide. Reader fix: pre-shim RAVEN MATLAB writes emitted GECKO models with geckoLight: "true" inside the metaData block (not as a top-level gecko_light). The reader now lifts that legacy key out of metaData so model.ec.gecko_light is populated whichever placement the file used. Round-trip writes always use the new top-level form. Regression tests: test_pre_shim_format_loads — synthetic fixture covering every legacy quirk we know about (--- doc marker, plain metaData, geckoLight inside metaData, top-level metabolite smiles, rxnNotes reaction key, integer bounds, double-quoted strings). Each quirk has its own assertion + comment. test_pre_shim_yeast_gem_loads_if_available — sanity-loads the real yeast-GEM.yml (2748 mets, 4102 rxns, 1143 genes) and asserts the documented preserved-counts table from the format reference. Skipped on CI runners where the working copy isn't mounted. * Cobra-aligned hardening pass from full code review No behaviour change on well-formed inputs. Highlights: - Packaging: derive __version__ from package metadata (was a stale hard-coded "0.0.1" that the docs site reported); pin ruff==0.15.15 in the dev extra and CI; fix two lint errors unpinned ruff started flagging. - Errors: solver/feasibility failures in run_init, run_ftinit, fill_tasks and random_sampling now raise cobra.exceptions.OptimizationError instead of bare RuntimeError (consistent with the rest of the package). - Consistency: single utils.parse.subsystem_to_str coerces reaction subsystem to cobra's canonical str across io.excel / comparison.compare / curation.batch / manipulation.add (fixes a crash on non-string items and the silent drop of multi-subsystem reactions); shared GPR score aggregators in utils.gpr used by init.score and init.genes; KEGG-download progress uses a module logger instead of print. - Robustness: zip path-traversal guard in binaries.py; penalty>0 check in connect_blocked_reactions; NaN-sample guard in random_sampling; all-zero ec coupling warning; optional verify= SHA256 re-check on data cache hits; non-finite z-score guard in reporter. Regression tests added for each. * io.yaml: reaction EC codes as cobra annotation ec-code (#19) * Add diff_models, annotation, and conditions modules for yeast-GEM port Lands the upstream-shareable pieces that yeast-GEM has been implementing locally during its Python port (see yeast-GEM/code/python/PORTING_PLAN.md and UPSTREAM_CANDIDATES.md). These are organism-agnostic; yeast-GEM will consume them via a Python dependency on raven-python. New modules ----------- raven_python.comparison.diff diff_models(a, b, ...) -> DiffReport — strict two-model semantic- equality diff. Complements the existing compare_models (N-model presence-matrix overview). Used as a CI gate to verify that two toolchains (e.g. MATLAB RAVEN vs raven_python, pre/post refactor of one toolchain) produce equivalent models. Includes a python -m raven_python.comparison.diff CLI. raven_python.annotation.sbo add_sbo_terms — SBO term assignment with "fill" semantic. Default parameter set reproduces yeast-GEM's behaviour; biomass metabolite names, biomass/NGAM reaction names, and pseudoreaction substrings are overridable. Transport detection is pluggable (default: same- met-name in two compartments). Includes an `only_last_reaction_ for_pseudo` legacy bug-compat flag for yeast-GEM's lock-step migration; off by default for any new caller. raven_python.annotation.delta_g load_delta_g_csv / save_delta_g_csv — generic side-car CSV persistence for scalar notes (ΔG by default, but the note key, column names, and id/value mapping are all configurable). raven_python.conditions.apply apply_condition(model, yaml_or_dict) — generic "apply this YAML condition" loader. Schema: prelude (reset_exchanges), cofactor_pseudoreaction (remove_mets + charge_balance_met), biomass_stoichiometry_delta, per-rxn bounds, expected_uptake_count. Project-specific extensions (e.g. yeast-GEM's amino_acid_ratio) are handled by the caller before/after this function — kept upstream-narrow on purpose. Also exposes set_reaction_bounds helper that bypasses cobra's lb<=ub validator for the (legitimate) cases where a condition lands on an infeasible bound state. Tests ----- 46 new tests across the three modules; full pre-existing raven-python suite still passes (519 passed; 1 unrelated pre-existing openpyxl ImportError in tests/test_io_git.py; 2 skipped). ruff clean. Not in this commit ------------------ The biomass / GAM / chemostat / fit_gam modules are still tracked as upstream candidates in yeast-GEM/code/python/UPSTREAM_CANDIDATES.md and remain local in yeast-GEM until phase 4 of the port (which would ideally land them directly here). * Add raven_python.biomass — sum / scale / rescale / set_gam Generic biomass-equation manipulation, extracted from yeast-GEM's sumBioMass / scaleBioMass / rescalePseudoReaction / changeGAM as yeast-GEM moves to depend on raven-python (yeast-GEM phase 4 of the porting plan). Module layout ------------- raven_python.biomass.config BiomassConfig — biomass_rxn id + proton_met id + ordered tuple of BiomassComponent (per-component pseudoreaction name + mass- computation strategy). raven_python.biomass.scale sum_biomass(model, config) → {component: g/gDW, ..., "total": X} rescale_pseudoreaction(model, config, name, factor) — multiply the pseudoreaction's substrate coefs by factor and rebalance H+ to keep ionic charge at zero. scale_biomass(model, config, name, new_value, balance_out=None) — rescale a component to a target fraction, optionally balancing a second component so the total stays at 1 g/gDW. raven_python.biomass.gam set_gam(model, value, *, biomass_rxn, cofactor_met_names, ngam_rxn=None, ngam_value=None) — scales every metabolite in the biomass pseudoreaction whose `name` is in the supplied cofactor set, preserving its sign; optionally fixes the NGAM rxn bounds. Mass strategies (per BiomassComponent.mass_strategy): "mw" plain MW from chemical formula (carbohydrate / ion / cofactor) "mw_minus_2h" MW − 2.016 g/mol per substrate (protein — charged tRNAs release two protons) "mw_minus_water" MW − 18.015 g/mol per substrate (RNA / DNA — polymerisation releases one water) "grams" stoichiometry already in g/gDW (lipid backbone) Tests: 19 new tests over a synthetic toy model that exercises every mass strategy, the H+ charge rebalance, scale_biomass with and without balance_out, set_gam on cofactor mets (and the NGAM bound path). * Add raven_python.manipulation.find_duplicate_reactions (detection variant) Detection-only counterpart to remove_duplicate_reactions. Returns duplicate groups instead of mutating the model. Ignores bounds / GPR / objective — only stoichiometry is compared, mirroring the typical curation use case ("find reactions that could be merged"). A new ``ignore_direction=True`` default (yeast-GEM convention) treats A→B and B→A as duplicates. Set False to require identical orientation. Used by yeast-GEM's modelTests port (Tier 3 / phase 5) to flag duplicate reactions during curation review. * Add raven_python.curation — batch_curate / batch_curate_from_tsv Generic batch curation engine extracted from yeast-GEM's MATLAB curateMetsRxnsGenes (yeast-GEM phase 6). Adds or updates metabolites, reactions and genes from pandas DataFrames; a batch_curate_from_tsv convenience wrapper reads the equivalent TSVs. Schema (matches yeast-GEM's data/modelCuration/template/ layout): mets_df metNames, comps, formula, charge, inchi, metNotes + any number of MIRIAM-namespace columns genes_df genes, geneShortNames + MIRIAM columns rxns_df rxnNames, grRules, lb, ub, rev, subSystems, eccodes, rxnNotes, rxnReferences, rxnConfidenceScores + MIRIAM columns rxns_coeffs_df rxnNames, metNames, comps, coefficient (one row per (reaction, metabolite) pair) Match keys: metabolites — (name, compartment) tuple genes — gene id reactions — stoichiometric signature Existing entities get their annotations overwritten (warning emitted); new entities are added with fresh ids generated from the supplied ``met_id_prefix`` / ``rxn_id_prefix`` (defaults M_ / R_ per the BiGG convention; yeast-GEM passes s_ / r_). Width of the existing zero-padded suffix is preserved so s_0001 → s_0002, not s_2. "Everything after the core columns is MIRIAM" — the header of any extra column becomes the annotation namespace key. Matches MATLAB behaviour exactly so yeast-GEM's existing TSVs work unchanged, and projects with different MIRIAM column sets need no code change. CurationResult dataclass records what was added vs updated so callers can verify in tests / CI. Tests: 13 new (add/update mets, add/update genes, add/update rxns by stoichiometry, miriam auto-detect, id-width preservation, combined mets+rxns in one call, missing-metabolite error, batch_curate_from_tsv round trip). * io.yaml: byte-compatible round-trip with cobrapy + RAVEN MATLAB Three things this fixes: 1. write_yaml_model dropped the !!omap tags entirely. _to_plain was flattening cobra's OrderedDict to plain dict, which causes ruamel to emit ordinary block mappings. RAVEN MATLAB's reader is a line-based parser keyed on !!omap and therefore could not load any file we wrote. _to_plain now returns OrderedDict so ruamel re-emits the !!omap tag. 2. eccodes was lost on round-trip — it wasn't in _RXN_FIELDS, so read_yaml_model didn't capture it into .notes and write_yaml_model couldn't lift it back. Added. 3. RAVEN MATLAB writes reaction notes as 'rxnNotes'; cobrapy and this writer use 'notes'. Added a read-time alias so existing yeast-GEM YAML files (which still say 'rxnNotes') load cleanly. Writes go out as 'notes' (cobrapy-canonical). Top-level layout now matches RAVEN MATLAB: metaData first, then metabolites / reactions / genes / compartments, then optional gecko_light + ec-rxns + ec-enzymes. id/name/version live inside metaData (RAVEN convention) — cobrapy reading these files still works, but cobra_model.id ends up None because cobrapy doesn't know about metaData. raven_python.read_yaml_model lifts both metaData.id/name/version onto model.id / model.name / model.notes['version'] so the rest of the codebase doesn't care which layout the file used. Empty-name genes are no longer emitted as — that's a cobrapy quirk that drifts yeast-GEM YAML files away from RAVEN MATLAB's output. Verified end-to-end: * cobra.io.load_yaml_model reads every file the new writer produces (yeast-GEM and a synthetic fixture). * RAVEN MATLAB readYAMLmodel reads every file the new writer produces. * Round-tripping yeast-GEM through raven_python preserves 2748/2748 metabolites, 4102/4102 reactions, 1143/1143 genes, 2411 eccodes, 3984 reaction deltaG, 2696 metabolite deltaG, 1788 SMILES, 1443 rxn-notes — no semantic drift. Tests ----- * tests/test_io_yaml_parity.py is new: covers every RAVEN extension, the rxnNotes legacy alias, the SMILES YAML-special character case, metaData-first layout, and cobra readability. * tests/test_io_yaml.py::test_output_is_cobra_readable adjusts for the metaData layout (cobra recovers mets/rxns/annotation but not model.id, by design). * conditions: switch from PyYAML to ruamel.yaml PyYAML is not a project dependency; raven-python uses ruamel.yaml (already pulled in via cobra) everywhere else. The conditions module and its tests still imported PyYAML, which broke pytest collection on clean CI runners with 'No module named yaml'. Both apply.py and the test now use a YAML(typ='safe') instance from ruamel.yaml — same plain-dict semantics as PyYAML's safe_load / safe_dump, no new dependency. * io.yaml: document the format + accept legacy geckoLight-in-metaData Adds docs/reference/yaml_format.md as the canonical schema reference for the cross-toolchain YAML format (cobrapy / raven-python / RAVEN MATLAB). Covers the full document shape, per-entry field order, RAVEN extensions, the GECKO ec-* sections, the metaData provenance block, number / string / quoting rules, and the cross-reader interoperability matrix. Linked from docs/reference/index.md and the I/O guide. Reader fix: pre-shim RAVEN MATLAB writes emitted GECKO models with geckoLight: "true" inside the metaData block (not as a top-level gecko_light). The reader now lifts that legacy key out of metaData so model.ec.gecko_light is populated whichever placement the file used. Round-trip writes always use the new top-level form. Regression tests: test_pre_shim_format_loads — synthetic fixture covering every legacy quirk we know about (--- doc marker, plain metaData, geckoLight inside metaData, top-level metabolite smiles, rxnNotes reaction key, integer bounds, double-quoted strings). Each quirk has its own assertion + comment. test_pre_shim_yeast_gem_loads_if_available — sanity-loads the real yeast-GEM.yml (2748 mets, 4102 rxns, 1143 genes) and asserts the documented preserved-counts table from the format reference. Skipped on CI runners where the working copy isn't mounted. * io.yaml: represent reaction EC codes as cobra annotation['ec-code'] EC numbers are a standard MIRIAM cross-reference, so the cobra-native representation is annotation['ec-code'] (a list) -- exactly where cobrapy and geckopy read them. raven-python was instead routing RAVEN's legacy top-level `eccodes` key into model.notes['eccodes'], so reaction EC codes written by RAVEN-MATLAB never reached the annotation['ec-code'] location geckopy reads from. - Drop `eccodes` from _RXN_FIELDS (it is not a RAVEN-only notes field). - Add _lift_eccodes_to_annotation: a legacy top-level `eccodes` (a ;-joined string or a list) is lifted into annotation['ec-code'] on read, mirroring the existing _lift_smiles_to_annotation; a native annotation['ec-code'] wins. - On write, EC codes serialise via cobra's annotation block; no top-level `eccodes` is emitted. - Update test_io_yaml_parity expectations to the cobra-aligned location (verified against the real yeast-GEM.yml: 2411 reactions). * docs: update RAVEN cross-references for the post-reorg folder layout (#20) RAVEN moved its functions out of the core/ catch-all into purpose-based top-level folders (SysBioChalmers/RAVEN#614). Repoint every RAVEN file path in the cross-reference docs (IMPROVEMENTS.md, docs/reference/matlab_raven_backports.md): - FSEOF / randomSampling / reporterMetabolites -> analysis/ - parseTaskList / checkTasks -> tasks/ - fillGaps -> gapfilling/ - addRxns / changeRxns / standardizeGrRules -> manipulation/ - getIndexes / checkModelStruct / getElementalBalance -> queries/ - getModelFromHomology -> reconstruction/homology/ - getKEGGModelForOrganism -> reconstruction/kegg/ - runINIT / ftINIT -> INIT/ Also corrects references that were stale even before the reorg (getKEGGModelForOrganism was in external/kegg/) and points the proposed GPR-lint back-port findPotentialErrors at manipulation/, alongside standardizeGrRules. Doc-only: raven-python's module layout already matches RAVEN's new structure (it was the template the reorg mirrored), so no code changes are needed. * refactor: rename plotting subpackage to visualization (#21) Align the (stub) plotting subpackage with RAVEN's folder layout, where pathway + plotting were unified into visualization/ (SysBioChalmers/RAVEN#614). - src/raven_python/plotting/ -> src/raven_python/visualization/ - pyproject optional-dependency extra [plotting] -> [visualization] (matplotlib); CI (.github/workflows/ci.yml) and ReadTheDocs (.readthedocs.yaml) install lines updated to match so the renamed extra still resolves. - docs updated: README, CHANGELOG, installation, docs/README, api/index, todo. The subpackage is an unimplemented stub (empty __init__), so nothing imports it and there is no behaviour change. Generic uses of the word "plotting" (seaborn / heatmap prose) are left as-is. * Ship type information and enforce it; make gpr_to_dnf public (#22) Three related "make the package's contracts real" changes: - Add the PEP 561 py.typed marker so the package's extensive type hints are visible to downstream type checkers (geckopy included). The hatchling wheel ships raven_python/py.typed. - Add mypy to the dev extra, a lenient [tool.mypy] config (ignore_missing_imports for the un-stubbed cobra/optlang/scipy/ruamel), and a mypy CI job. Fix the 36 type errors this surfaced -- all type-only (Path vs str annotations, None-guards that match existing behaviour, optlang Variable typing, isinstance/cast narrowing). No runtime behaviour changes; the full test suite stays green. - Promote manipulation.expand._gpr_to_dnf to a public gpr_to_dnf (re-exported from raven_python.manipulation). geckopy's call sites switch to it in lockstep (separate PR), so no deprecated alias is kept. * Harden curation, EC-data and archive-handling modules (#23) Tier-2 audit of the post-review modules surfaced four targeted fixes: - curation/batch.py: new reactions coerce a list-valued subSystems via subsystem_to_str (";"-joined) instead of str(list), matching the update path. - io/ec_data.py: _eccodes_to_yaml strips stray separators in the single-EC case so a trailing ";" never leaks into the written YAML. - binaries.py: _safe_extract_zip rejects symlink members, defence-in-depth alongside the existing path-traversal guard. - binaries.py / data.py: archive and dataset downloads pass a socket timeout to urlopen so a stalled server cannot hang the process. Adds regression tests for each fix. * Surgical performance pass on hot paths (#24) Targeted, behaviour-preserving optimisations from the review: - manipulation/add.py + change.py: resolve equation tokens through a shared (name, compartment) -> metabolite index (_build_met_index) instead of re-scanning model.metabolites per token. Bulk reaction add/change by name drops from O(R*k*M) to O(R*k); the index is updated as new mets are created so cross-token and cross-reaction dedup is preserved. - reconstruction/homology/homology.py: replace DataFrame.apply(axis=1) in the ortholog filter with a comprehension over the columns (membership is already O(1); avoids per-row Series construction). - analysis/sampling.py: build the random objective with optlang add() instead of sum(), which re-canonicalises the expression on every term (O(n^2)). Adds a cross-reaction metabolite-dedup regression test for the add path. * Robustness and polish fixes (#25) Tier 4 of the review: small, targeted hardening, no behaviour change on valid input. - gapfilling/fill.py: clamp the connectivity gap-fill big-M to the largest finite bound magnitude, so a template reaction with an infinite bound no longer puts an infinite coefficient into the MILP (which broke the solver). - reconstruction/kegg/download.py: a malformed or unreadable .netrc now raises a ValueError explaining how to fix it, instead of a raw NetrcParseError/OSError. - io/excel.py: always write the metabolite formula to the METS COMPOSITION column; it was dropped whenever an InChI was present. - visualization: the empty stub package raises a clear NotImplementedError (with a roadmap pointer) on attribute access, via a PEP 562 module __getattr__. A regression test per fix. * Add code-built-model YAML round-trip test (covers the objective) (#26) The existing YAML round-trip and parity tests originate their model from a parsed doc; none builds a model directly from cobra objects, and none asserts the objective coefficient survives (the parity fixture pins it to 0). Add one round-trip test that builds a model in code with a non-zero objective and asserts metabolites, reactions, bounds, stoichiometry, GPR, subsystem, formula, annotation and the objective all survive write -> read. * Share the linear-chain INIT model fixture via tests/conftest.py (#27) test_init.py, test_init_build.py and test_init_solvers.py each built the same linear-chain INIT model (EX_A -> A -> B -> C -> D) independently, differing only in the model id and whether gene rules were attached. Move that construction into a new tests/conftest.py as linear_chain_model / linear_chain_model_with_genes fixtures; the three files now reuse it (test bodies unchanged). The bespoke _toy_ftinit_model stays local. No behaviour change. * Publish kegg116 KEGG artefacts (v0.1.0) (#28) * Publish kegg116 KEGG artefacts as gzip, version-prefixed assets (v0.1.0) First downloadable KEGG artefact set, wired into the runtime resolvers: - All artefacts are gzip and version-prefixed (kegg116_.gz) so MATLAB and Windows read them with the built-in gunzip, no external tool. organism_gene_ko moves from xz to gzip for the same reason. - HMM libraries ship as one gzip concatenated flatfile per domain; ensure_kegg_hmm_library decompresses and hmmpresses on first use, ~10x smaller than the pressed index and portable across HMMER versions. - Add a version-prefix-tolerant artefact resolver (_resolve_artefact) used by the organism/sequence entry points; parse_kegg_dump and build_kegg_artefacts.py gain an opt-in --version. - Populate data/manifest.json and _DATA_REGISTRY with the kegg116 release assets (real SHA256 + bytes); refresh the maintainer docs and manifest example. - Bump version to 0.1.0 and update CHANGELOG. * Add KEGG taxonomy artefact and phyl_dist (RAVEN getPhylDist port) Publish kegg116_taxonomy.gz and regenerate RAVEN's keggPhylDist from it, so GECKO's organism-distance kcat selection needs no MATLAB .mat file: - reconstruction.kegg.phyl_dist + PhylDist faithfully reproduce RAVEN getPhylDist's (asymmetric, occasionally negative) distance metric; parse_taxonomy_records exposes ids/names/lineages and reads .gz transparently. - data.ensure_kegg_taxonomy fetches the artefact; build_kegg_artefacts.py emits it. - Register kegg116_taxonomy.gz in data/manifest.json and _DATA_REGISTRY (8 files). - Tests for phyl_dist (hand-checked against RAVEN) and the taxonomy fetch; update migration/IMPROVEMENTS/maintainer docs and CHANGELOG. * Publish kegg116 KEGG artefacts as gzip, version-prefixed assets (v0.1.0) (#29) First downloadable KEGG artefact set, wired into the runtime resolvers: - All artefacts are gzip and version-prefixed (kegg116_.gz) so MATLAB and Windows read them with the built-in gunzip, no external tool. organism_gene_ko moves from xz to gzip for the same reason. - HMM libraries ship as one gzip concatenated flatfile per domain; ensure_kegg_hmm_library decompresses and hmmpresses on first use, ~10x smaller than the pressed index and portable across HMMER versions. - Add a version-prefix-tolerant artefact resolver (_resolve_artefact) used by the organism/sequence entry points; parse_kegg_dump and build_kegg_artefacts.py gain an opt-in --version. - Populate data/manifest.json and _DATA_REGISTRY with the kegg116 release assets (real SHA256 + bytes); refresh the maintainer docs and manifest example. - Bump version to 0.1.0 and update CHANGELOG. Add KEGG taxonomy artefact and phyl_dist (RAVEN getPhylDist port) Publish kegg116_taxonomy.gz and regenerate RAVEN's keggPhylDist from it, so GECKO's organism-distance kcat selection needs no MATLAB .mat file: - reconstruction.kegg.phyl_dist + PhylDist faithfully reproduce RAVEN getPhylDist's (asymmetric, occasionally negative) distance metric; parse_taxonomy_records exposes ids/names/lineages and reads .gz transparently. - data.ensure_kegg_taxonomy fetches the artefact; build_kegg_artefacts.py emits it. - Register kegg116_taxonomy.gz in data/manifest.json and _DATA_REGISTRY (8 files). - Tests for phyl_dist (hand-checked against RAVEN) and the taxonomy fetch; update migration/IMPROVEMENTS/maintainer docs and CHANGELOG. Bundle core KEGG artefacts into kegg116_core.tar.gz Combine the five core model files (reference model + KO/reaction/organism-gene/ rxn-flag tables) into one kegg116_core.tar.gz; HMM libraries and taxonomy stay separate. The release drops from 8 assets to 4. - ensure_kegg_data now fetches the single bundle, SHA-verifies it, and extracts the version-prefixed members into the cache once (safe extraction, matching download.py). - build_kegg_artefacts.py groups the core files into the bundle after the HMM step. - Regenerate data/manifest.json and _DATA_REGISTRY (4 entries); update manifest.example, tests (bundle fixture), and docs. * Remove the visualization stub and [visualization] extra (#30) Mirror MATLAB RAVEN removing its pathway-map / omics-overlay plotting functions (drawMap, colorPathway, drawPathway, markPathwayWith*, setOmicDataToRxns, ...) as obsolete/low-value (SysBioChalmers/RAVEN #618). raven-python only had a not-implemented `visualization` stub reserving that domain; drop it and its scaffolding. cobrapy + Escher cover pathway/omics visualization externally. - Delete src/raven_python/visualization/ and tests/test_visualization.py. - Drop the [visualization] (matplotlib) extra; remove it from CI, ReadTheDocs, and the installation / README / api-index / todo docs. - CHANGELOG: record the removal. The other functions RAVEN removed (MetaCyc, xml_toolbox, Excel-import wrappers, solveQP) were never ported to raven-python, so no further changes are needed. * Auto-resolve the taxonomy artefact in domain-mode from_artefacts (#31) get_kegg_model_for_organism_from_artefacts("prokaryotes"/"eukaryotes") builds a whole-domain model, which needs the KEGG taxonomy file. Taxonomy is a separate artefact (not part of the core set ensure_kegg_data fetches), so the call raised "Domain mode needs the KEGG taxonomy file; pass taxonomy=." unless the caller supplied a path by hand. It now auto-resolves taxonomy for domain mode: from the artefact directory if present, else via ensure_kegg_taxonomy(version). An explicit taxonomy= still wins; species mode is unchanged. Adds a regression test. * Use hmmsearch (not hmmscan) for the de-novo KEGG query (#32) get_kegg_model_from_sequences now runs one hmmsearch over the concatenated KO library instead of an hmmscan against a pressed database: - run_hmmsearch / parse_hmmsearch_tblout replace run_hmmscan / parse_hmmscan_tblout. hmmsearch is HMMER's faster, better-parallelising direction (profiles as the query) and needs no hmmpress. -Z is fixed to the profile count so per-hit E-values (and thus assign_kos output) are identical to the previous hmmscan path — verified on real HMMs (same hits, same E-values, same assignments). - ensure_kegg_hmm_library just gunzips the library (no hmmpress, no .h3* sidecars). - build_hmm_library concatenates the per-KO HMMs without pressing; the published .hmm.gz artefact is unchanged. - Docs / IMPROVEMENTS (K7) / CHANGELOG updated. * Replace the on-disk KEGG test fixture with a synthetic in-code dump (#33) tests/data/kegg_dump contained real KEGG records (e.g. reaction R00010 and KO K01194 with their EC/RHEA/ChEBI cross-references) which the project is not licensed to redistribute. Remove the directory and instead generate an equivalent, fully fictional KEGG-format dump at test time via a new session-scoped fixture in tests/conftest.py. The synthetic dump mimics the flat-file format so it still exercises the parser (reaction flags, overview-map skipping, InChI/formula handling, mapformula irreversibility, KO/gene grouping, taxonomy lineages) but all identifiers, names, sequences and cross-references are invented. The four dependent test modules (parse, query, hmm, organism) consume the fixture and assert against the fictional ids. No real KEGG content is committed and coverage is unchanged. * Rename project and import package: raven-python -> raven-toolbox (#34) * Rename project and import package: raven-python -> raven-toolbox Rename the distribution (raven-python -> raven-toolbox) and the import package (raven_python -> raven_toolbox) across all source, tests, scripts, docs, and packaging metadata. Project URLs now point to SysBioChalmers/raven-toolbox. * Complete the rename: remaining raven-python/raven_python -> raven-toolbox/raven_toolbox The package/distribution rename left occurrences behind after the rebase: - import statements () in the reconstruction.kegg modules and data.py, which would have failed at import time; - monkeypatch string targets and the cache-path assertions in the tests; - the wheel/package and mypy paths in pyproject.toml (still pointing at the now-removed src/raven_python), plus the distribution name and project URLs; - docs, data manifests and GitHub URLs. Replace them so the import package is consistently raven_toolbox and all distribution/repo references point to raven-toolbox. Also drop the empty src/raven_python directory left behind by the rebase. * Wrap homology imports to satisfy ruff isort after the rename raven_python -> raven_toolbox widened the homology hits import past the 100-char line length, so ruff isort (I001) wanted it split across lines. Format it as a multiline import block; ruff check . is clean again. * CI: bump actions to Node 24 versions (checkout v5, setup-python v6) (#35) actions/checkout@v4 and actions/setup-python@v5 run on the deprecated Node.js 20 runtime. Bump to actions/checkout@v5 and actions/setup-python@v6, both of which run on Node.js 24, to clear the GitHub Actions deprecation warning. * Prepare 0.2.0 release Bump version 0.1.0 -> 0.2.0 and complete the CHANGELOG 0.2.0 section (raven-toolbox rename, hmmsearch de-novo KEGG query, domain-mode taxonomy auto-resolve, synthetic KEGG test fixture, visualization stub removal, Node 24 CI). --- .github/workflows/ci.yml | 38 +- .readthedocs.yaml | 1 - CHANGELOG.md | 135 +++-- IMPROVEMENTS.md | 200 ++++---- README.md | 20 +- data/manifest.example.json | 31 ++ data/manifest.json | 35 ++ data/manifest.schema.json | 82 ++++ docs/README.md | 2 +- docs/archive/ftinit_review_and_plan.md | 4 +- docs/archive/localization_design.md | 12 +- docs/conf.py | 10 +- docs/guide/analysis_and_comparison.md | 16 +- docs/guide/context_specific.md | 18 +- docs/guide/index.md | 4 +- docs/guide/io_and_manipulation.md | 38 +- docs/guide/localization.md | 14 +- docs/guide/omics.md | 14 +- docs/guide/quickstart.md | 16 +- docs/guide/reconstruction.md | 22 +- docs/guide/tasks_and_gapfilling.md | 12 +- docs/index.md | 10 +- docs/installation.md | 15 +- docs/maintenance/data_manifest.md | 155 ++++++ docs/maintenance/index.md | 4 + docs/maintenance/kegg_data_format.md | 62 +-- docs/maintenance/maintaining_binaries.md | 28 +- docs/maintenance/maintaining_kegg_data.md | 68 ++- docs/reference/api/analysis.md | 4 +- docs/reference/api/comparison.md | 4 +- docs/reference/api/gapfilling.md | 4 +- docs/reference/api/index.md | 7 +- docs/reference/api/init.md | 4 +- docs/reference/api/io.md | 4 +- docs/reference/api/localization.md | 6 +- docs/reference/api/manipulation.md | 6 +- docs/reference/api/omics.md | 4 +- docs/reference/api/reconstruction.md | 6 +- docs/reference/api/resolvers.md | 20 +- docs/reference/api/tasks.md | 4 +- docs/reference/api/utils.md | 6 +- docs/reference/index.md | 12 +- docs/reference/matlab_raven_backports.md | 56 +-- docs/reference/migration.md | 114 ++--- docs/reference/todo.md | 19 +- docs/reference/yaml_format.md | 332 +++++++++++++ docs/requirements.txt | 2 +- docs/studies/humangem_validation.md | 34 +- docs/studies/index.md | 4 +- docs/studies/init_param_calibration.md | 4 +- docs/studies/init_solver_benchmark.md | 6 +- docs/studies/kegg_hmm_cutoff_calibration.md | 4 +- docs/studies/yeast_localization_benchmark.md | 6 +- pyproject.toml | 24 +- scripts/README.md | 10 +- scripts/analyze_hmm_cutoffs.py | 6 +- scripts/analyze_init_params.py | 14 +- scripts/analyze_init_robustness.py | 4 +- scripts/analyze_init_solvers.py | 2 +- scripts/benchmark_localization_yeast.py | 6 +- scripts/build_kegg_artefacts.py | 70 ++- scripts/make_registry_snippet.py | 122 ++++- src/raven_python/__init__.py | 10 - src/raven_python/comparison/__init__.py | 7 - src/raven_python/data.py | 135 ----- src/raven_python/plotting/__init__.py | 1 - .../reconstruction/kegg/taxonomy.py | 71 --- src/raven_toolbox/__init__.py | 15 + .../analysis/__init__.py | 6 +- .../analysis/fseof.py | 0 .../analysis/reporter.py | 2 + .../analysis/sampling.py | 17 +- src/raven_toolbox/annotation/__init__.py | 24 + src/raven_toolbox/annotation/delta_g.py | 115 +++++ src/raven_toolbox/annotation/sbo.py | 180 +++++++ .../binaries.py | 45 +- src/raven_toolbox/biomass/__init__.py | 50 ++ src/raven_toolbox/biomass/config.py | 95 ++++ src/raven_toolbox/biomass/gam.py | 67 +++ src/raven_toolbox/biomass/scale.py | 207 ++++++++ src/raven_toolbox/comparison/__init__.py | 23 + .../comparison/compare.py | 25 +- src/raven_toolbox/comparison/diff.py | 285 +++++++++++ src/raven_toolbox/conditions/__init__.py | 25 + src/raven_toolbox/conditions/apply.py | 192 ++++++++ src/raven_toolbox/curation/__init__.py | 54 ++ src/raven_toolbox/curation/batch.py | 460 ++++++++++++++++++ src/raven_toolbox/data.py | 215 ++++++++ .../gapfilling/__init__.py | 2 +- .../gapfilling/fill.py | 24 +- .../init/__init__.py | 18 +- .../init/build.py | 10 +- .../init/ftinit.py | 30 +- .../init/genes.py | 16 +- .../init/init.py | 6 +- .../init/merge.py | 0 .../init/prep.py | 6 +- .../init/score.py | 15 +- .../init/steps.py | 2 +- .../init/taskfill.py | 15 +- .../io/__init__.py | 10 +- .../io/ec_data.py | 13 +- .../io/excel.py | 16 +- src/{raven_python => raven_toolbox}/io/git.py | 10 +- src/{raven_python => raven_toolbox}/io/sif.py | 2 +- .../io/yaml.py | 155 +++++- .../localization/__init__.py | 4 +- .../localization/predict.py | 16 +- .../localization/scores.py | 4 +- src/raven_toolbox/manifest.py | 123 +++++ .../manipulation/__init__.py | 5 +- .../manipulation/add.py | 51 +- .../manipulation/change.py | 8 +- .../manipulation/compartments.py | 4 +- .../manipulation/expand.py | 12 +- .../manipulation/irreversible.py | 2 +- .../manipulation/merge.py | 0 .../manipulation/parameters.py | 0 .../manipulation/remove.py | 2 +- .../manipulation/simplify.py | 50 +- .../manipulation/transfer.py | 2 +- .../manipulation/transport.py | 14 +- .../omics/__init__.py | 4 +- .../omics/hpa.py | 8 +- src/raven_toolbox/py.typed | 0 .../reconstruction/__init__.py | 0 .../reconstruction/homology/__init__.py | 10 +- .../reconstruction/homology/blast.py | 16 +- .../reconstruction/homology/hits.py | 0 .../reconstruction/homology/homology.py | 14 +- .../reconstruction/kegg/__init__.py | 26 +- .../reconstruction/kegg/assemble.py | 0 .../reconstruction/kegg/download.py | 24 +- .../reconstruction/kegg/hmm.py | 32 +- .../reconstruction/kegg/organism.py | 35 +- .../reconstruction/kegg/parse.py | 80 ++- .../reconstruction/kegg/query.py | 83 ++-- .../reconstruction/kegg/taxonomy.py | 176 +++++++ .../tasks/__init__.py | 4 +- .../tasks/check.py | 16 +- .../tasks/tasklist.py | 2 +- .../utils/__init__.py | 8 +- .../utils/balance.py | 0 .../utils/gpr.py | 39 +- .../utils/parse.py | 28 +- .../utils/sort.py | 0 .../utils/validate.py | 0 tests/conftest.py | 210 ++++++++ tests/data/kegg_dump/compound | 34 -- tests/data/kegg_dump/compound.inchi | 1 - tests/data/kegg_dump/genes.pep | 12 - tests/data/kegg_dump/ko | 14 - tests/data/kegg_dump/reaction | 31 -- tests/data/kegg_dump/reaction_mapformula.lst | 3 - tests/data/kegg_dump/taxonomy | 10 - tests/test_analysis_fseof.py | 4 +- tests/test_analysis_reporter.py | 2 +- tests/test_analysis_sampling.py | 2 +- tests/test_annotation.py | 204 ++++++++ tests/test_binaries.py | 79 ++- tests/test_biomass.py | 246 ++++++++++ tests/test_change_grrules.py | 2 +- tests/test_comparison.py | 4 +- tests/test_comparison_diff.py | 146 ++++++ tests/test_conditions.py | 211 ++++++++ tests/test_curation.py | 270 ++++++++++ tests/test_data.py | 130 +++-- tests/test_gapfilling.py | 18 +- tests/test_init.py | 40 +- tests/test_init_build.py | 21 +- tests/test_init_ftinit.py | 4 +- tests/test_init_genes.py | 2 +- tests/test_init_merge.py | 2 +- tests/test_init_oracles.py | 2 +- tests/test_init_pipeline.py | 8 +- tests/test_init_solvers.py | 24 +- tests/test_init_taskfill.py | 4 +- tests/test_io_excel.py | 8 +- tests/test_io_git.py | 8 +- tests/test_io_sif.py | 6 +- tests/test_io_yaml.py | 51 +- tests/test_io_yaml_ec_data.py | 27 +- tests/test_io_yaml_parity.py | 364 ++++++++++++++ tests/test_localization.py | 4 +- tests/test_manifest.py | 125 +++++ tests/test_manipulation_add.py | 25 +- tests/test_manipulation_change.py | 4 +- tests/test_manipulation_compartments.py | 2 +- tests/test_manipulation_expand.py | 6 +- tests/test_manipulation_find_duplicate.py | 85 ++++ tests/test_manipulation_irreversible.py | 2 +- tests/test_manipulation_merge.py | 2 +- tests/test_manipulation_remove.py | 4 +- tests/test_manipulation_simplify.py | 2 +- tests/test_manipulation_transfer.py | 2 +- tests/test_manipulation_transport.py | 2 +- tests/test_omics_hpa.py | 2 +- tests/test_parameters.py | 2 +- tests/test_reconstruction_blast.py | 4 +- tests/test_reconstruction_homology.py | 4 +- tests/test_reconstruction_kegg_download.py | 10 +- tests/test_reconstruction_kegg_hmm.py | 134 ++--- tests/test_reconstruction_kegg_organism.py | 140 +++--- tests/test_reconstruction_kegg_parse.py | 150 +++--- tests/test_reconstruction_kegg_phyl_dist.py | 70 +++ tests/test_reconstruction_kegg_query.py | 62 +-- tests/test_tasks.py | 2 +- tests/test_tasks_essential.py | 2 +- tests/test_utils_balance.py | 2 +- tests/test_utils_gpr.py | 4 +- tests/test_utils_parse.py | 57 +++ tests/test_utils_sort.py | 6 +- tests/test_utils_validate.py | 4 +- tests/tinit_oracles.py | 2 +- 214 files changed, 7105 insertions(+), 1595 deletions(-) create mode 100644 data/manifest.example.json create mode 100644 data/manifest.json create mode 100644 data/manifest.schema.json create mode 100644 docs/maintenance/data_manifest.md create mode 100644 docs/reference/yaml_format.md delete mode 100644 src/raven_python/__init__.py delete mode 100644 src/raven_python/comparison/__init__.py delete mode 100644 src/raven_python/data.py delete mode 100644 src/raven_python/plotting/__init__.py delete mode 100644 src/raven_python/reconstruction/kegg/taxonomy.py create mode 100644 src/raven_toolbox/__init__.py rename src/{raven_python => raven_toolbox}/analysis/__init__.py (73%) rename src/{raven_python => raven_toolbox}/analysis/fseof.py (100%) rename src/{raven_python => raven_toolbox}/analysis/reporter.py (97%) rename src/{raven_python => raven_toolbox}/analysis/sampling.py (91%) create mode 100644 src/raven_toolbox/annotation/__init__.py create mode 100644 src/raven_toolbox/annotation/delta_g.py create mode 100644 src/raven_toolbox/annotation/sbo.py rename src/{raven_python => raven_toolbox}/binaries.py (73%) create mode 100644 src/raven_toolbox/biomass/__init__.py create mode 100644 src/raven_toolbox/biomass/config.py create mode 100644 src/raven_toolbox/biomass/gam.py create mode 100644 src/raven_toolbox/biomass/scale.py create mode 100644 src/raven_toolbox/comparison/__init__.py rename src/{raven_python => raven_toolbox}/comparison/compare.py (87%) create mode 100644 src/raven_toolbox/comparison/diff.py create mode 100644 src/raven_toolbox/conditions/__init__.py create mode 100644 src/raven_toolbox/conditions/apply.py create mode 100644 src/raven_toolbox/curation/__init__.py create mode 100644 src/raven_toolbox/curation/batch.py create mode 100644 src/raven_toolbox/data.py rename src/{raven_python => raven_toolbox}/gapfilling/__init__.py (81%) rename src/{raven_python => raven_toolbox}/gapfilling/fill.py (85%) rename src/{raven_python => raven_toolbox}/init/__init__.py (63%) rename src/{raven_python => raven_toolbox}/init/build.py (92%) rename src/{raven_python => raven_toolbox}/init/ftinit.py (93%) rename src/{raven_python => raven_toolbox}/init/genes.py (84%) rename src/{raven_python => raven_toolbox}/init/init.py (97%) rename src/{raven_python => raven_toolbox}/init/merge.py (100%) rename src/{raven_python => raven_toolbox}/init/prep.py (97%) rename src/{raven_python => raven_toolbox}/init/score.py (82%) rename src/{raven_python => raven_toolbox}/init/steps.py (96%) rename src/{raven_python => raven_toolbox}/init/taskfill.py (94%) rename src/{raven_python => raven_toolbox}/io/__init__.py (56%) rename src/{raven_python => raven_toolbox}/io/ec_data.py (96%) rename src/{raven_python => raven_toolbox}/io/excel.py (92%) rename src/{raven_python => raven_toolbox}/io/git.py (91%) rename src/{raven_python => raven_toolbox}/io/sif.py (97%) rename src/{raven_python => raven_toolbox}/io/yaml.py (68%) rename src/{raven_python => raven_toolbox}/localization/__init__.py (87%) rename src/{raven_python => raven_toolbox}/localization/predict.py (97%) rename src/{raven_python => raven_toolbox}/localization/scores.py (96%) create mode 100644 src/raven_toolbox/manifest.py rename src/{raven_python => raven_toolbox}/manipulation/__init__.py (90%) rename src/{raven_python => raven_toolbox}/manipulation/add.py (89%) rename src/{raven_python => raven_toolbox}/manipulation/change.py (92%) rename src/{raven_python => raven_toolbox}/manipulation/compartments.py (97%) rename src/{raven_python => raven_toolbox}/manipulation/expand.py (94%) rename src/{raven_python => raven_toolbox}/manipulation/irreversible.py (97%) rename src/{raven_python => raven_toolbox}/manipulation/merge.py (100%) rename src/{raven_python => raven_toolbox}/manipulation/parameters.py (100%) rename src/{raven_python => raven_toolbox}/manipulation/remove.py (99%) rename src/{raven_python => raven_toolbox}/manipulation/simplify.py (82%) rename src/{raven_python => raven_toolbox}/manipulation/transfer.py (98%) rename src/{raven_python => raven_toolbox}/manipulation/transport.py (91%) rename src/{raven_python => raven_toolbox}/omics/__init__.py (75%) rename src/{raven_python => raven_toolbox}/omics/hpa.py (96%) create mode 100644 src/raven_toolbox/py.typed rename src/{raven_python => raven_toolbox}/reconstruction/__init__.py (100%) rename src/{raven_python => raven_toolbox}/reconstruction/homology/__init__.py (55%) rename src/{raven_python => raven_toolbox}/reconstruction/homology/blast.py (91%) rename src/{raven_python => raven_toolbox}/reconstruction/homology/hits.py (100%) rename src/{raven_python => raven_toolbox}/reconstruction/homology/homology.py (95%) rename src/{raven_python => raven_toolbox}/reconstruction/kegg/__init__.py (75%) rename src/{raven_python => raven_toolbox}/reconstruction/kegg/assemble.py (100%) rename src/{raven_python => raven_toolbox}/reconstruction/kegg/download.py (90%) rename src/{raven_python => raven_toolbox}/reconstruction/kegg/hmm.py (95%) rename src/{raven_python => raven_toolbox}/reconstruction/kegg/organism.py (78%) rename src/{raven_python => raven_toolbox}/reconstruction/kegg/parse.py (88%) rename src/{raven_python => raven_toolbox}/reconstruction/kegg/query.py (70%) create mode 100644 src/raven_toolbox/reconstruction/kegg/taxonomy.py rename src/{raven_python => raven_toolbox}/tasks/__init__.py (86%) rename src/{raven_python => raven_toolbox}/tasks/check.py (96%) rename src/{raven_python => raven_toolbox}/tasks/tasklist.py (99%) rename src/{raven_python => raven_toolbox}/utils/__init__.py (50%) rename src/{raven_python => raven_toolbox}/utils/balance.py (100%) rename src/{raven_python => raven_toolbox}/utils/gpr.py (73%) rename src/{raven_python => raven_toolbox}/utils/parse.py (52%) rename src/{raven_python => raven_toolbox}/utils/sort.py (100%) rename src/{raven_python => raven_toolbox}/utils/validate.py (100%) create mode 100644 tests/conftest.py delete mode 100644 tests/data/kegg_dump/compound delete mode 100644 tests/data/kegg_dump/compound.inchi delete mode 100644 tests/data/kegg_dump/genes.pep delete mode 100644 tests/data/kegg_dump/ko delete mode 100644 tests/data/kegg_dump/reaction delete mode 100644 tests/data/kegg_dump/reaction_mapformula.lst delete mode 100644 tests/data/kegg_dump/taxonomy create mode 100644 tests/test_annotation.py create mode 100644 tests/test_biomass.py create mode 100644 tests/test_comparison_diff.py create mode 100644 tests/test_conditions.py create mode 100644 tests/test_curation.py create mode 100644 tests/test_io_yaml_parity.py create mode 100644 tests/test_manifest.py create mode 100644 tests/test_manipulation_find_duplicate.py create mode 100644 tests/test_reconstruction_kegg_phyl_dist.py create mode 100644 tests/test_utils_parse.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b5b7f125..99445c3a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,16 +16,34 @@ jobs: name: ruff runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 + - uses: actions/checkout@v5 + - uses: actions/setup-python@v6 with: python-version: "3.12" cache: pip cache-dependency-path: pyproject.toml - run: pip install --upgrade pip - - run: pip install ruff + # Pinned so the lint result is reproducible (unpinned ruff silently + # adds rules over time). Keep in sync with the ``dev`` extra in pyproject. + - run: pip install ruff==0.15.15 - run: ruff check . + typecheck: + name: mypy + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: pyproject.toml + - run: pip install --upgrade pip + # Installs runtime deps + the dev extra (mypy). Third-party stubs are + # absent for cobra/optlang/scipy; ignore_missing_imports handles that. + - run: pip install -e ".[dev]" + - run: mypy + test: name: pytest (py${{ matrix.python }}) runs-on: ubuntu-latest @@ -34,32 +52,32 @@ jobs: matrix: python: ["3.11", "3.12", "3.13"] steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 + - uses: actions/checkout@v5 + - uses: actions/setup-python@v6 with: python-version: ${{ matrix.python }} cache: pip cache-dependency-path: pyproject.toml - run: pip install --upgrade pip - # ``-e .[dev,plotting,excel]`` so every optional extra is exercised. + # ``-e .[dev,excel]`` so every optional extra is exercised. # Gurobi is not installable on free runners; the relevant tests # skip themselves when ``optlang.gurobi_interface`` cannot import. - - run: pip install -e ".[dev,plotting,excel]" + - run: pip install -e ".[dev,excel]" - run: pytest -q --maxfail=5 --durations=20 docs: name: docs runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 + - uses: actions/checkout@v5 + - uses: actions/setup-python@v6 with: python-version: "3.12" cache: pip cache-dependency-path: pyproject.toml - run: pip install --upgrade pip # The package (with autodoc's import-time extras) plus the docs toolchain. - - run: pip install -e ".[excel,plotting]" -r docs/requirements.txt + - run: pip install -e ".[excel]" -r docs/requirements.txt # ``-W --keep-going`` keeps the docs build warning-clean; mirrors the # ReadTheDocs build (docs/conf.py + .readthedocs.yaml). - run: sphinx-build -b html -W --keep-going docs docs/_build/html diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 996ad6bd..c8f3257e 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -21,7 +21,6 @@ python: path: . extra_requirements: - excel - - plotting - requirements: docs/requirements.txt formats: diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d9bd98a..51547b6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,77 @@ # Changelog -Milestones in the raven-python port. For function-level status see -[docs/raven_migration.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/reference/migration.md); for open work see -[docs/todo.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/reference/todo.md). +Milestones in the raven-toolbox port. For function-level status see +[docs/raven_migration.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/migration.md); for open work see +[docs/todo.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/todo.md). + +## 0.2.0 — 2026-06-14 + +Project rename plus KEGG-reconstruction and CI improvements. + +* **Project renamed `raven-python` → `raven-toolbox`.** The import package is now + `raven_toolbox` (was `raven_python`) and the repository moved to + `SysBioChalmers/raven-toolbox`. Update imports and any `raven-python` git/URL + references accordingly. (#34) +* **De-novo KEGG query uses `hmmsearch` instead of `hmmscan`.** + `get_kegg_model_from_sequences` now runs one `hmmsearch` over the concatenated KO + library (`-Z` set to the profile count, so E-values and KO assignments are identical + to the previous `hmmscan` path) — the faster, more parallel search direction. + `ensure_kegg_hmm_library` no longer runs `hmmpress` (just gunzips); the published + `.hmm.gz` artefact is unchanged. (#32) +* **Domain-mode `get_kegg_model_for_organism_from_artefacts` auto-resolves the + taxonomy artefact** from the artefact directory, so `"prokaryotes"` / + `"eukaryotes"` no longer require an explicit `taxonomy=` path. (#31) +* **Test data no longer ships real KEGG records.** The on-disk + `tests/data/kegg_dump` is replaced by a session fixture that generates a fully + fictional KEGG-format dump at runtime, so no KEGG-derived data is redistributed. (#33) +* **Removed the visualization stub and the `[visualization]` extra** — an + unimplemented placeholder. (#30) +* **CI on Node 24** — `actions/checkout@v5`, `actions/setup-python@v6`. (#35) + +## 0.1.0 — 2026-06-10 + +First release with **published, downloadable KEGG artefacts**, plus a cobra-aligned +hardening pass (no behaviour change on well-formed inputs). Highlights: + +* **KEGG artefacts published (`kegg116`):** `ensure_kegg_data` / + `ensure_kegg_hmm_library` fetch version-pinned, SHA256-verified files from the + GitHub release. Every artefact is **gzip + version-prefixed** + (`kegg116_.gz`) so MATLAB and Windows read them with the built-in `gunzip` + (no external tool) — `organism_gene_ko` moved from xz to gzip for this. The core + model files (reference model + KO/reaction tables) ship as a single + `kegg116_core.tar.gz` that `ensure_kegg_data` extracts on first use; the HMM + libraries and `taxonomy` are separate assets. The **HMM + libraries ship as one gzip concatenated flatfile per domain** + (`kegg116_.hmm.gz`); the client decompresses and `hmmpress`-es once on + first use, cutting the download ~10× versus the pressed index and letting the + same artefact serve MATLAB RAVEN. +* **Taxonomy + phylogenetic distance:** publish `kegg116_taxonomy.gz` and add + `reconstruction.kegg.phyl_dist` (with `PhylDist`), a faithful port of RAVEN's + `getPhylDist` that regenerates the `keggPhylDist` distance matrix from the + taxonomy file — so GECKO's organism-distance kcat selection needs no MATLAB + `.mat`. `ensure_kegg_taxonomy` fetches the artefact. +* **Packaging:** `raven_toolbox.__version__` now derives from the installed package + metadata (`importlib.metadata`) instead of a hard-coded literal that had drifted + to `0.0.1`; the docs site reported the wrong version. Pinned `ruff==0.15.15` in + both the `dev` extra and CI so the lint result is reproducible, and fixed two + lint errors the unpinned ruff had started flagging. +* **Errors aligned to cobra:** solver/feasibility failures in `run_init`, + `run_ftinit`, `fill_tasks` and `random_sampling` now raise + `cobra.exceptions.OptimizationError` (already used elsewhere in the package) + instead of a bare `RuntimeError`. +* **Consistency:** a single `utils.parse.subsystem_to_str` coerces a reaction + `subsystem` to cobra's canonical `str` everywhere it is rendered/compared + (`io.excel`, `comparison.compare`, `curation.batch`, `manipulation.add`) — fixes + a crash on non-string subsystem items and the silent drop of multi-subsystem + reactions. GPR score-aggregation (`AGGREGATORS` / `resolve_aggregators`) is now + shared by `init.score` and `init.genes`. Maintainer-side KEGG-download progress + uses a module logger instead of `print`. +* **Robustness:** path-traversal guard on bundled-ZIP extraction (`binaries.py`, + matching the tarfile `filter="data"` precedent); `connect_blocked_reactions` + rejects a non-positive `penalty`; `random_sampling` refuses a NaN-contaminated + sample matrix; `ec_data` warns on an all-zero reaction↔enzyme coupling; optional + `verify=` SHA256 re-check on `ensure_data_file` cache hits; reporter p-value + guarded against non-finite z-scores. Regression tests added for each. ## 0.1.0a1 — 2026-05-30 @@ -15,15 +84,15 @@ RAVEN-style I/O formats (YAML / SIF / Excel). Validated against MATLAB RAVEN on * **Licensing:** released under the **MIT** license (previously GPL-3.0-or-later). * **Docs:** Sphinx + MyST documentation site (sources under `docs/`). -* Not yet implemented: visualization (`plotting/`), metabolomics-based (f)tINIT scoring, +* Not yet implemented: visualization (`visualization/`), metabolomics-based (f)tINIT scoring, and published binary / KEGG-artefact release bundles. See the README and - [docs/todo.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/reference/todo.md). + [docs/todo.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/todo.md). The milestone sections below record the incremental development history leading to this release. ## Infrastructure -* **GitHub Actions CI** ([.github/workflows/ci.yml](https://github.com/SysBioChalmers/raven-python/blob/develop/.github/workflows/ci.yml)) — +* **GitHub Actions CI** ([.github/workflows/ci.yml](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/.github/workflows/ci.yml)) — ruff + pytest matrix over Python 3.11/3.12/3.13. Tests that require Gurobi auto-skip (no Gurobi on free runners); the known HiGHS upstream blocker (`hybrid_interface.Configuration` rejects `lp_method='primal'`) is marked @@ -50,7 +119,7 @@ fixes with matching MATLAB back-port proposals in IMPROVEMENTS.md (FS4, B2). bug to `check_model`; the actual code is in `balance.py`. Two new regression tests (F3 in `test_analysis_fseof.py`, F5 in -`test_utils_balance.py`). [docs/known_issues.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/reference/known_issues.md) now +`test_utils_balance.py`). [docs/known_issues.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/known_issues.md) now fully closed (all sections A–F). ## Quality sweep — known-issues sections C / D / E @@ -85,7 +154,7 @@ download (defensive, needs urlopen mocking). ## Quality sweep — known-issues section B -Closed all four "silent misbehaviour" items from [docs/known_issues.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/reference/known_issues.md): +Closed all four "silent misbehaviour" items from [docs/known_issues.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/known_issues.md): * `merge_models` warns on `formula` / `charge` conflicts when two source models share a name[comp] but disagree (used to silently keep the first-seen). * `add_reactions_from_equations` warns when creating a metabolite in an @@ -99,7 +168,7 @@ Four new regression tests cover them. ## Quality sweep — known-issues section A -Closed all six "latent edge-case bug" items from [docs/known_issues.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/reference/known_issues.md): +Closed all six "latent edge-case bug" items from [docs/known_issues.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/known_issues.md): * `add_reactions_from_equations` no longer misparses `"2 oxoglutarate"` (or any leading-number metabolite name) — the resolver tries the full token before splitting off a coefficient. @@ -118,45 +187,45 @@ Six new regression tests cover the user-reachable cases. ## Phase 7 — Localization -* **Sub-cellular localisation by MILP.** [`localization.predict_localization`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/predict.py) - + [`apply_localization`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/predict.py). Deterministic (not simulated +* **Sub-cellular localisation by MILP.** [`localization.predict_localization`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/predict.py) + + [`apply_localization`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/predict.py). Deterministic (not simulated annealing); caller-passed `reactions_to_relocate` set with everything else pinned; incomplete-model tolerant (no silent reaction removal); `apply=False` returns a diff preview; multi-compartment by default with primary-free, extras-penalised scoring. -* **Predictor loaders.** [`load_wolfpsort`, `load_deeploc`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/scores.py), +* **Predictor loaders.** [`load_wolfpsort`, `load_deeploc`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/scores.py), with the `gene × compartment` DataFrame contract open for any predictor. -* **Compartment helpers** ([`manipulation/compartments.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/compartments.py)): +* **Compartment helpers** ([`manipulation/compartments.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/compartments.py)): `merge_compartments`, `copy_to_compartment` — useful standalone for model curation. -* **Real-data validation on yeast-GEM** ([docs/yeast_localization_benchmark.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/studies/yeast_localization_benchmark.md)) +* **Real-data validation on yeast-GEM** ([docs/yeast_localization_benchmark.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/studies/yeast_localization_benchmark.md)) — accuracy 0.72 → 0.39 on 298 GPR'd reactions as confident predictor mis-scoring rises from 0 % to 50 %; perfect on compartments with disjoint gene sets (c/g/lp/p/v/vm), and surfaces a `transport_cost` calibration insight for soft-probability score tables. ## Phase 5 — Data integration & analysis -* **Reporter metabolites, FSEOF, random sampling** ([`analysis/`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/analysis/)). -* **HPA omics ingestion** ([`omics.parse_hpa`, `parse_hpa_rna`, `hpa_gene_scores`, `rna_gene_scores`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/omics/hpa.py)) +* **Reporter metabolites, FSEOF, random sampling** ([`analysis/`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/analysis/)). +* **HPA omics ingestion** ([`omics.parse_hpa`, `parse_hpa_rna`, `hpa_gene_scores`, `rna_gene_scores`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/omics/hpa.py)) — pandas-tidy DataFrames replace RAVEN's sparse-matrix layout; scoring adapters reuse the existing GPR walk. -* **N-model comparison** ([`comparison.compare_models`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/comparison/compare.py)). +* **N-model comparison** ([`comparison.compare_models`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/comparison/compare.py)). * **Dynamic FBA** is **not ported** — established Python packages cover it (`dfba`, `reframed`, `mewpy`). ## Phase 4d — ftINIT -* **ftINIT pipeline** ([`init.ftinit`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/ftinit.py)) — staged MILP, linear merge, +* **ftINIT pipeline** ([`init.ftinit`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/ftinit.py)) — staged MILP, linear merge, task-aware gap-filling, gene pruning. * **Validated against MATLAB RAVEN on Human-GEM.** 5 Hart2015 cell-line models; Jaccard 0.973–0.977 (no-task) and 0.978–0.980 (task-constrained). See - [docs/humangem_validation.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/studies/humangem_validation.md). -* **Parameter calibration & input-robustness study** ([docs/init_param_calibration.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/studies/init_param_calibration.md)) + [docs/humangem_validation.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/studies/humangem_validation.md). +* **Parameter calibration & input-robustness study** ([docs/init_param_calibration.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/studies/init_param_calibration.md)) — `mip_gap=0.01` is the genome-scale full-pipeline sweet spot (~37% faster than 0.001 at Jaccard 0.995); pipeline is robust to expression noise (Jaccard 0.92–0.95) but sensitive to sparsity (50–70% dropout → Jaccard 0.59–0.71); the task + gap-fill layer keeps the essential-task pass-rate at 67–69/69 across the gradient, whereas tINIT-without-it passes only 35/69 even on clean data. -* **Cross-solver portability** ([docs/init_solver_benchmark.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/studies/init_solver_benchmark.md)) - + [`tests/test_init_solvers.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/tests/test_init_solvers.py): Gurobi and GLPK pass at toy +* **Cross-solver portability** ([docs/init_solver_benchmark.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/studies/init_solver_benchmark.md)) + + [`tests/test_init_solvers.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/tests/test_init_solvers.py): Gurobi and GLPK pass at toy scale; only Gurobi is viable at genome scale today (HiGHS hits an upstream optlang `clone()` bug; GLPK ignores `configuration.timeout` on MIP). * **Engineering wins surfaced by the genome-scale work:** `check_tasks` and @@ -166,25 +235,25 @@ Six new regression tests cover the user-reachable cases. ## Phase 4c — tINIT -* **INIT MILP and the tINIT pipeline** ([`init.run_init`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/init.py), - [`init.get_init_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/build.py)). Clean optlang reformulation; +* **INIT MILP and the tINIT pipeline** ([`init.run_init`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/init.py), + [`init.get_init_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/build.py)). Clean optlang reformulation; RNA-seq scoring via `5·ln(level/ref)`-clamped. ## Phase 4b — Gap-filling -* **Connectivity gap-filling** ([`gapfilling.connect_blocked_reactions`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/gapfilling/fill.py)) +* **Connectivity gap-filling** ([`gapfilling.connect_blocked_reactions`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/gapfilling/fill.py)) — MILP. Targeted (toward objective) mode delegates to `cobra.gapfill`. ## Phase 4a — Metabolic tasks -* **Task list parsing + `check_tasks`** ([`tasks/`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/tasks/)). +* **Task list parsing + `check_tasks`** ([`tasks/`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/tasks/)). ## Phase 3 — Reconstruction * **Homology-based draft** from a template GEM + BLAST/DIAMOND wrappers - ([`reconstruction/homology/`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/reconstruction/homology/)) — with structured + ([`reconstruction/homology/`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/homology/)) — with structured improvements over RAVEN's `getModelFromHomology` (see IMPROVEMENTS H1–H6). -* **KEGG five-step pipeline** ([`reconstruction/kegg/`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/reconstruction/kegg/)): +* **KEGG five-step pipeline** ([`reconstruction/kegg/`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/kegg/)): dump → parser → HMM library builder → species model → HMM-query draft. * **MetaCyc reconstruction** **not ported** (and flagged for removal from MATLAB RAVEN — see IMPROVEMENTS R-MetaCyc). @@ -193,17 +262,17 @@ Six new regression tests cover the user-reachable cases. * **YAML** aligned to cobra's `!!omap` writer + RAVEN-only fields preserved into `.notes`, plus geckopy `ec-*` for enzyme-constrained models - ([`io/yaml.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/io/yaml.py)). + ([`io/yaml.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/io/yaml.py)). * **SIF**, **Excel export**, and **Standard-GEM `model//…` git layout** - ([`io/`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/io/)). Excel import intentionally excluded. + ([`io/`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/io/)). Excel import intentionally excluded. ## Phase 1 — Foundation -* **GPR / balance / validation / parsing helpers** ([`utils/`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/)) — +* **GPR / balance / validation / parsing helpers** ([`utils/`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/)) — cobra-absent bits only; the rest are cheatsheeted. -* **Manipulation ergonomic layer** ([`manipulation/`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/)) — +* **Manipulation ergonomic layer** ([`manipulation/`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/)) — add/change/remove/transport/transfer/merge/simplify/variance + adopted transforms. -* **External-binary resolver** ([`binaries.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/binaries.py)) — version-pinned +* **External-binary resolver** ([`binaries.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/binaries.py)) — version-pinned release-ZIP registry, SHA256-verified cache. ## Phase 0 — Scaffold diff --git a/IMPROVEMENTS.md b/IMPROVEMENTS.md index 825d551d..4cee38b7 100644 --- a/IMPROVEMENTS.md +++ b/IMPROVEMENTS.md @@ -14,14 +14,14 @@ Categories: - **REMOVAL** — functionality that should be dropped (here *and* in MATLAB RAVEN) because it does more harm than good. -Status legend: 💡 proposed · 🔨 implemented in raven-python · ⬆️ upstreamed to MATLAB RAVEN · +Status legend: 💡 proposed · 🔨 implemented in raven-toolbox · ⬆️ upstreamed to MATLAB RAVEN · 🗑️ dropped (and to remove from MATLAB RAVEN) --- ## R-MetaCyc: drop MetaCyc reconstruction (REMOVAL — also remove from MATLAB RAVEN) -**Decision 2026-05-24:** MetaCyc-based reconstruction is **not ported to raven-python** and should be +**Decision 2026-05-24:** MetaCyc-based reconstruction is **not ported to raven-toolbox** and should be **removed from MATLAB RAVEN**. Status: 🗑️. **What RAVEN does:** `getMetaCycModelForOrganism` builds a draft by BLAST/DIAMOND of the query @@ -62,18 +62,18 @@ it is only incidentally in the MetaCyc folder.) ## getModelFromHomology (Phase 3a — implemented) -Design + rationale in [docs/plan_get_model_from_homology.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/archive/plan_get_model_from_homology.md); +Design + rationale in [docs/plan_get_model_from_homology.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/archive/plan_get_model_from_homology.md); implemented in `reconstruction/homology/homology.py`. *Logic* improvements over RAVEN's algorithm (RAVEN's own comments flag several of these spots as uncertain). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| H1 | ERGONOMICS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | Split the overloaded `strictness` 1/2/3 into two orthogonal params: `bidirectional` (reciprocal hits) and `best_hits_only`. RBH = both true. `strictness=` kept as a compat alias. | -| H2 | EFFICIENCY (robustness) | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | Rewrite GPRs on the **cobra GPR AST**, not `regexprep` string substitution — eliminates partial-match hazards and the `OLD_… or` regex cleanup pass RAVEN needs. | -| H3 | ERGONOMICS (correctness) | raven-python 🔨 | 🔨 | Explicit `complex_policy` (default **`flag`** = RAVEN-compatible `OLD_`; plus `keep`/`drop`) for AND-subunits lacking an ortholog, via correct OR/AND **AST** semantics. | +| H1 | ERGONOMICS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | Split the overloaded `strictness` 1/2/3 into two orthogonal params: `bidirectional` (reciprocal hits) and `best_hits_only`. RBH = both true. `strictness=` kept as a compat alias. | +| H2 | EFFICIENCY (robustness) | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | Rewrite GPRs on the **cobra GPR AST**, not `regexprep` string substitution — eliminates partial-match hazards and the `OLD_… or` regex cleanup pass RAVEN needs. | +| H3 | ERGONOMICS (correctness) | raven-toolbox 🔨 | 🔨 | Explicit `complex_policy` (default **`flag`** = RAVEN-compatible `OLD_`; plus `keep`/`drop`) for AND-subunits lacking an ortholog, via correct OR/AND **AST** semantics. | | H4 | (correctness) | both 🔨/💡 | 🔨 | Best-hit selection by **bitscore** (db-size-independent, the RBH standard); `score="evalue"` optional. | -| H5 | EFFICIENCY | raven-python 🔨 | 🔨 | DataFrame ortholog map (pandas merge + dict) replaces `allGenes`/`allTo`/`allFrom` sparse-matrix `sub2ind` index juggling. | -| H6 | NEW | raven-python 🔨 | 🔨 | Structured provenance: `HomologyResult.gene_map` + per-reaction `notes['homology_source']`. | +| H5 | EFFICIENCY | raven-toolbox 🔨 | 🔨 | DataFrame ortholog map (pandas merge + dict) replaces `allGenes`/`allTo`/`allFrom` sparse-matrix `sub2ind` index juggling. | +| H6 | NEW | raven-toolbox 🔨 | 🔨 | Structured provenance: `HomologyResult.gene_map` + per-reaction `notes['homology_source']`. | ## KEGG download / dump parsing / HMM build (Phase 3b.1 / 3b.2 / 3b.3 — implemented) @@ -85,37 +85,37 @@ and `taxonomy.py` (3b.3). Maintainer-side, build-time tooling (PLAN.md §2.3b). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| K1 | EFFICIENCY (robustness) | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | Read each reaction's equation from its **own `EQUATION` field**, not from `reaction.lst` matched by line order. RAVEN reads `reaction.lst` line *i* into reaction *i*, assuming the two files stay perfectly aligned — brittle. **MATLAB back-port:** parse the `EQUATION` field already present in `reaction`. | -| K2 | ERGONOMICS (correctness) | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | Undefined-stoichiometry terms (`n C00001`, `(n+1) C00002`) keep their **real compound id** with coefficient 1 and the reaction is *flagged*, instead of minting `"n C00001"` pseudo-metabolites later renamed `undefined_N`. Cleaner metabolite graph; flag still drives the `keep*` filters. | -| K3 | ERGONOMICS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | Reaction quality labels become a tidy boolean **`rxn_flags` table** (spontaneous/undefined-stoich/incomplete/general) instead of free-text appended to `rxnNotes`, so downstream filters join on a column rather than substring-matching notes. | -| K4 | EFFICIENCY | raven-python 🔨 | 🔨 | **Gene-free reference model** + separate `organism_gene_ko` table (the big one), instead of RAVEN's giant `rxnGeneMat` baked into the global model. Per-organism GPRs are built only at runtime (3b.4/3b.5), keeping the published artefact small. | -| K5 | EFFICIENCY (portability) | raven-python 🔨 | 🔨 | **KEGG download in pure Python stdlib** (`urllib`/`tarfile`/`gzip`/`netrc`), porting `fetch_keggdb.sh`. Drops the script's `wget`/`tar`/`gunzip` (and Cygwin-on-Windows) requirement, so it runs unchanged on Linux/macOS/Windows; tar extraction uses the `data` filter (no path traversal); same `~/.netrc` credential hygiene. The arrange step is split out (`extract_kegg_dump`) so it's network-free and unit-tested. | -| K6 | EFFICIENCY | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Per-KO multi-FASTA via a stdlib offset index** (`_index_fasta` → seek), replacing `constructMultiFasta`'s Java-`Hashtable` byte scan with 5M-element preallocation. One streaming pass, only wanted ids retained; no MATLAB/Java heap tuning. | -| K7 | EFFICIENCY | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Concatenate per-KO HMMs and `hmmpress` into one pressed library**, so the query path (3b.5) runs a single `hmmscan` against the database instead of RAVEN's thousands of per-KO `hmmsearch` invocations. | -| K8 | EFFICIENCY (scope) | raven-python 🔨 | 🔨 | **Drop the `getPhylDist` distance matrix.** Its only uses in RAVEN were per-organism HMM-sequence subsampling (`maxPhylDist`/`nSequences`) and the kingdom filter. Our fixed prok90/euk90 libraries (3b.3) remove the subsampling rationale, and domain mode (3b.4) uses the taxonomy domain classification directly — so the O(n²) matrix is never built. Simpler, faster, less code. | -| K9 | EFFICIENCY (memory) | raven-python 🔨 | 🔨 | **Stream `organism_gene_ko` to disk** in `parse_kegg_dump` instead of building it in memory. Real KEGG has **9.05M** gene↔KO associations; the in-memory DataFrame build OOMs in a few GB. Streaming (now via the external merge sort of K14) runs the full parse with flat, bounded peak memory. (Found by validating against a real KEGG FTP dump.) | -| K10 | EFFICIENCY (size) | raven-python 🔨 | 🔨 | **Reference model as gzipped RAVEN/cobra YAML** (`reference_model.yml.gz`) rather than SBML: RAVEN-native, MATLAB-readable, and ~1.1 MB vs ~30 MB SBML for the real 12k-reaction model. Made `io/yaml.py` gzip-aware on a `.gz` suffix (general-purpose). | -| K11 | ERGONOMICS | raven-python 🔨 | 🔨 | **`ensure_data`** (`data.py`): version-pinned registry that fetches/verifies/caches the published KEGG artefacts under `~/.cache/raven-python/data/`, mirroring `ensure_binary`. End users get a draft model with no KEGG access and no manual data handling — the `…_from_artefacts` entry points auto-fetch when no local dir is supplied. | -| K12 | EFFICIENCY | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Fast MAFFT (FFT-NS-2) for HMM training** instead of RAVEN's `--auto`, which selects slow iterative refinement (`dvtditr`) on medium/large KOs — observed ~2.5 min/KO (days for a domain) on real KEGG 118. FFT-NS-2 (`--retree 2 --maxiterate 0`) is seconds/KO and ample for profile-HMM building. **PartTree cutover is residue-based and memory-auto-tuned**: MAFFT memory tracks residues (count × length), not sequence count, so a count threshold let long-protein KOs (K00901: 2,788 seqs, 2.55 M residues) OOM under FFT-NS-2 — measured ~5 GB MAFFT RSS with FFT-NS-2 vs **0.69 GB with PartTree** for the same alignment. The cutover is **length-aware and memory-auto-tuned**: FFT-NS-2 memory is driven by the progressive-alignment **DP cost ≈ n_seqs × mean_len²** (= residues²/n_seqs), *not* residue count — a few hundred long proteins cost far more than the same residues in many short ones. (First tried a residue-only model `RSS≈1.32R²+1.84R`; it then OOM'd on K12047 — 452 seqs but mean length 2082, 0.94 M residues — because long proteins blow the per-residue cost.) Calibrated `RSS_GB ≈ 4.2e-9 × (n_seqs × mean_len²)` across real KEGG KOs (250k/266→0.67 GB … 1.5M/1624→5.73 GB; K12047 cost 1.96e9 = the largest, hence its OOM). `_auto_cost_budget` switches to PartTree when the DP cost exceeds `0.65 × (total − 2.5 GB overhead) / 4.2e-9` (≈7.9e8 on a 7.6 GB box), **warns on low-memory hosts**, and `parttree_residues` overrides with a manual residue cutoff. Back-portable to RAVEN. | -| K13 | EFFICIENCY | raven-python 🗑️ | 🗑️ | ~~Per-KO sequence cap (`max_sequences`)~~ — **removed.** Briefly added as a count-based cap, but the residue-based PartTree cutover (K12) bounds MAFFT memory without dropping any sequences, so the cap was redundant complexity. All deduplicated sequences are kept. | -| K14 | EFFICIENCY (size) | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Sort `organism_gene_ko` by `(organism, gene)` and store it xz-compressed** (`organism_gene_ko.tsv.xz`), cutting the dominant artefact **≈78 → 27 MB (2.9×)**. Gene IDs within an organism share long prefixes (locus tags, numeric runs), so sorting makes them adjacent and far more compressible (sort alone: 78→48 MB; xz vs gzip captures the cross-row redundancy gzip's 32 KB window misses: →27 MB). The sort is an **external merge sort** bounded to `chunk_rows` rows in memory (sorted runs spooled to gzipped temp files, merged with `heapq.merge`), so it keeps K9's flat memory profile. Both `lzma` and `gzip` are Python stdlib (native on Windows/macOS/Linux, no extra binary); small tables stay gzipped TSV (MATLAB-native), only the big one is xz (MATLAB needs an external `unxz`). Sorted order also matches the by-organism query in `get_kegg_model_for_organism`, enabling a future `searchsorted` slice instead of loading all 9M rows. Back-portable to RAVEN. | -| K15 | ERGONOMICS (correctness) | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Recalibrate the HMM-query KO-assignment defaults** (`assign_kos`): cut-off `1e-50 → 1e-30`, `min_score_ratio_g 0.8 → 0.9`; `min_score_ratio_ko` left at 0.3 but **documented as empirically inert**. Cross-validated the full 3b.5 pipeline against the true KEGG gene→KO annotation of four organisms across both libraries and the well-/lesser-studied axis — *S. cerevisiae*, *Cyanidioschyzon merolae* (red alga), *E. coli* K-12, *Mycoplasma genitalium* (minimal genome). Real annotations score overwhelmingly (median E ≈ 1e-100…1e-155; even the weakest 1% ≈ 1e-15…1e-36) while spurious hits cluster at ≈1e-8 — a ~20-order-of-magnitude gap. RAVEN's `1e-50` therefore sits **inside the true-positive tail** and silently drops real-but-divergent hits for no noise-rejection gain: gene→KO recall on *M. genitalium* was only 0.84 (reaction recall 0.87). At `1e-30` + `ratio_g=0.9`: *M. genitalium* recall **0.84→0.94** (rxn 0.87→0.97), *E. coli* 0.95→0.97 with **fewer** unannotated reactions (198→173, the tighter gene-ratio prunes spurious multi-KO genes), *S. cerevisiae*/*C. merolae* held or improved. The three sweep tables showed `min_score_ratio_ko` produced identical output at 0.0/0.3/0.5 across all four organisms — a magic-number knob that does nothing; `min_score_ratio_g` is the real precision lever. Full numbers in [docs/kegg_hmm_cutoff_calibration.md](https://github.com/SysBioChalmers/raven-python/blob/develop/docs/studies/kegg_hmm_cutoff_calibration.md) (reproduce with `scripts/analyze_hmm_cutoffs.py`). Back-portable to RAVEN. | +| K1 | EFFICIENCY (robustness) | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | Read each reaction's equation from its **own `EQUATION` field**, not from `reaction.lst` matched by line order. RAVEN reads `reaction.lst` line *i* into reaction *i*, assuming the two files stay perfectly aligned — brittle. **MATLAB back-port:** parse the `EQUATION` field already present in `reaction`. | +| K2 | ERGONOMICS (correctness) | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | Undefined-stoichiometry terms (`n C00001`, `(n+1) C00002`) keep their **real compound id** with coefficient 1 and the reaction is *flagged*, instead of minting `"n C00001"` pseudo-metabolites later renamed `undefined_N`. Cleaner metabolite graph; flag still drives the `keep*` filters. | +| K3 | ERGONOMICS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | Reaction quality labels become a tidy boolean **`rxn_flags` table** (spontaneous/undefined-stoich/incomplete/general) instead of free-text appended to `rxnNotes`, so downstream filters join on a column rather than substring-matching notes. | +| K4 | EFFICIENCY | raven-toolbox 🔨 | 🔨 | **Gene-free reference model** + separate `organism_gene_ko` table (the big one), instead of RAVEN's giant `rxnGeneMat` baked into the global model. Per-organism GPRs are built only at runtime (3b.4/3b.5), keeping the published artefact small. | +| K5 | EFFICIENCY (portability) | raven-toolbox 🔨 | 🔨 | **KEGG download in pure Python stdlib** (`urllib`/`tarfile`/`gzip`/`netrc`), porting `fetch_keggdb.sh`. Drops the script's `wget`/`tar`/`gunzip` (and Cygwin-on-Windows) requirement, so it runs unchanged on Linux/macOS/Windows; tar extraction uses the `data` filter (no path traversal); same `~/.netrc` credential hygiene. The arrange step is split out (`extract_kegg_dump`) so it's network-free and unit-tested. | +| K6 | EFFICIENCY | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Per-KO multi-FASTA via a stdlib offset index** (`_index_fasta` → seek), replacing `constructMultiFasta`'s Java-`Hashtable` byte scan with 5M-element preallocation. One streaming pass, only wanted ids retained; no MATLAB/Java heap tuning. | +| K7 | EFFICIENCY | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Concatenate per-KO HMMs into one library**, so the query path (3b.5) runs a single `hmmsearch` over the multi-profile library instead of RAVEN's thousands of per-KO `hmmsearch` invocations — same fast search direction, one process, no `hmmpress`. | +| K8 | EFFICIENCY (scope) | raven-toolbox 🔨 | 🔨 | **Drop the `getPhylDist` distance matrix.** Its only uses in RAVEN were per-organism HMM-sequence subsampling (`maxPhylDist`/`nSequences`) and the kingdom filter. Our fixed prok90/euk90 libraries (3b.3) remove the subsampling rationale, and domain mode (3b.4) uses the taxonomy domain classification directly — so the *reconstruction* path never builds the O(n²) matrix. The matrix itself remains available on demand via `taxonomy.phyl_dist`, which regenerates RAVEN's `keggPhylDist` from the published `taxonomy` artefact for GECKO's organism-distance kcat selection (no `.mat` needed). | +| K9 | EFFICIENCY (memory) | raven-toolbox 🔨 | 🔨 | **Stream `organism_gene_ko` to disk** in `parse_kegg_dump` instead of building it in memory. Real KEGG has **9.05M** gene↔KO associations; the in-memory DataFrame build OOMs in a few GB. Streaming (now via the external merge sort of K14) runs the full parse with flat, bounded peak memory. (Found by validating against a real KEGG FTP dump.) | +| K10 | EFFICIENCY (size) | raven-toolbox 🔨 | 🔨 | **Reference model as gzipped RAVEN/cobra YAML** (`reference_model.yml.gz`) rather than SBML: RAVEN-native, MATLAB-readable, and ~1.1 MB vs ~30 MB SBML for the real 12k-reaction model. Made `io/yaml.py` gzip-aware on a `.gz` suffix (general-purpose). | +| K11 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **`ensure_data`** (`data.py`): version-pinned registry that fetches/verifies/caches the published KEGG artefacts under `~/.cache/raven-toolbox/data/`, mirroring `ensure_binary`. End users get a draft model with no KEGG access and no manual data handling — the `…_from_artefacts` entry points auto-fetch when no local dir is supplied. | +| K12 | EFFICIENCY | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Fast MAFFT (FFT-NS-2) for HMM training** instead of RAVEN's `--auto`, which selects slow iterative refinement (`dvtditr`) on medium/large KOs — observed ~2.5 min/KO (days for a domain) on real KEGG 118. FFT-NS-2 (`--retree 2 --maxiterate 0`) is seconds/KO and ample for profile-HMM building. **PartTree cutover is residue-based and memory-auto-tuned**: MAFFT memory tracks residues (count × length), not sequence count, so a count threshold let long-protein KOs (K00901: 2,788 seqs, 2.55 M residues) OOM under FFT-NS-2 — measured ~5 GB MAFFT RSS with FFT-NS-2 vs **0.69 GB with PartTree** for the same alignment. The cutover is **length-aware and memory-auto-tuned**: FFT-NS-2 memory is driven by the progressive-alignment **DP cost ≈ n_seqs × mean_len²** (= residues²/n_seqs), *not* residue count — a few hundred long proteins cost far more than the same residues in many short ones. (First tried a residue-only model `RSS≈1.32R²+1.84R`; it then OOM'd on K12047 — 452 seqs but mean length 2082, 0.94 M residues — because long proteins blow the per-residue cost.) Calibrated `RSS_GB ≈ 4.2e-9 × (n_seqs × mean_len²)` across real KEGG KOs (250k/266→0.67 GB … 1.5M/1624→5.73 GB; K12047 cost 1.96e9 = the largest, hence its OOM). `_auto_cost_budget` switches to PartTree when the DP cost exceeds `0.65 × (total − 2.5 GB overhead) / 4.2e-9` (≈7.9e8 on a 7.6 GB box), **warns on low-memory hosts**, and `parttree_residues` overrides with a manual residue cutoff. Back-portable to RAVEN. | +| K13 | EFFICIENCY | raven-toolbox 🗑️ | 🗑️ | ~~Per-KO sequence cap (`max_sequences`)~~ — **removed.** Briefly added as a count-based cap, but the residue-based PartTree cutover (K12) bounds MAFFT memory without dropping any sequences, so the cap was redundant complexity. All deduplicated sequences are kept. | +| K14 | EFFICIENCY (size) | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Sort `organism_gene_ko` by `(organism, gene)` and store it gzipped** (`organism_gene_ko.tsv.gz`), cutting the dominant artefact **≈78 → 48 MB** by sorting alone. Gene IDs within an organism share long prefixes (locus tags, numeric runs), so sorting makes them adjacent and far more compressible. The sort is an **external merge sort** bounded to `chunk_rows` rows in memory (sorted runs spooled to gzipped temp files, merged with `heapq.merge`), so it keeps K9's flat memory profile. We first xz-compressed this file (≈27 MB, 2.9×) but switched to **gzip** (≈74 MB) so MATLAB reads it with built-in `gunzip` and no external `unxz`: the artefacts are shared with MATLAB RAVEN, and the once-per-release size cut wasn't worth a MATLAB toolchain dependency. All tables are now gzipped TSV, native on Windows/macOS/Linux. Sorted order also matches the by-organism query in `get_kegg_model_for_organism`, enabling a future `searchsorted` slice instead of loading all 9M rows. Back-portable to RAVEN. | +| K15 | ERGONOMICS (correctness) | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Recalibrate the HMM-query KO-assignment defaults** (`assign_kos`): cut-off `1e-50 → 1e-30`, `min_score_ratio_g 0.8 → 0.9`; `min_score_ratio_ko` left at 0.3 but **documented as empirically inert**. Cross-validated the full 3b.5 pipeline against the true KEGG gene→KO annotation of four organisms across both libraries and the well-/lesser-studied axis — *S. cerevisiae*, *Cyanidioschyzon merolae* (red alga), *E. coli* K-12, *Mycoplasma genitalium* (minimal genome). Real annotations score overwhelmingly (median E ≈ 1e-100…1e-155; even the weakest 1% ≈ 1e-15…1e-36) while spurious hits cluster at ≈1e-8 — a ~20-order-of-magnitude gap. RAVEN's `1e-50` therefore sits **inside the true-positive tail** and silently drops real-but-divergent hits for no noise-rejection gain: gene→KO recall on *M. genitalium* was only 0.84 (reaction recall 0.87). At `1e-30` + `ratio_g=0.9`: *M. genitalium* recall **0.84→0.94** (rxn 0.87→0.97), *E. coli* 0.95→0.97 with **fewer** unannotated reactions (198→173, the tighter gene-ratio prunes spurious multi-KO genes), *S. cerevisiae*/*C. merolae* held or improved. The three sweep tables showed `min_score_ratio_ko` produced identical output at 0.0/0.3/0.5 across all four organisms — a magic-number knob that does nothing; `min_score_ratio_g` is the real precision lever. Full numbers in [docs/kegg_hmm_cutoff_calibration.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/studies/kegg_hmm_cutoff_calibration.md) (reproduce with `scripts/analyze_hmm_cutoffs.py`). Back-portable to RAVEN. | ## FSEOF (Phase 5 — implemented, redesigned) -RAVEN `core/FSEOF.m` → `analysis/fseof.py` (`fseof`). User was unhappy with RAVEN's +RAVEN `analysis/FSEOF.m` → `analysis/fseof.py` (`fseof`). User was unhappy with RAVEN's output; redesigned substantially. | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| FS1 | CORRECTNESS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Robust trend via linear regression** (slope + correlation) over the whole scan, instead of RAVEN's strict step-by-step monotonicity that discards a target on a single noisy step (LP alternative optima). |r| is a quality score for filtering. pFBA per step keeps the scan stable. | -| FS2 | NEW | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Reports knockdown/knockout targets**, not just amplification. RAVEN only flags reactions whose \|flux\| *rises* with the enforced product; reactions driven *toward zero* — the down-regulation/deletion candidates, arguably the most actionable — are classified here (`knockdown`/`knockout`). | -| FS3 | ERGONOMICS | raven-python 🔨 | 🔨 | **Gene-level aggregation** (`gene_targets`) mapping reaction targets to genes, plus the **full flux scan** retained — all as DataFrames, vs RAVEN's printed TSV + endpoint-only slope. | -| FS4 | CORRECTNESS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | Slope is the **regression slope** consistent with the selection criterion, not RAVEN's endpoint difference that disagreed with its own monotonicity filter. **Proposed back-port:** record the enforced target-flux level at each iteration into a vector and replace both the reported slope `abs(fseof.results(num,iterations) - fseof.results(num,1)) / abs(targetMax - targetMax/iterations)` and the `targets.slope` field with `polyfit(enforcedFlux, fseof.results(num,:), 1)` slope across all iterations. The label classification (amplify/knockdown/knockout, see FS2) should also use the slope of `|flux|` instead of comparing endpoints. | +| FS1 | CORRECTNESS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Robust trend via linear regression** (slope + correlation) over the whole scan, instead of RAVEN's strict step-by-step monotonicity that discards a target on a single noisy step (LP alternative optima). |r| is a quality score for filtering. pFBA per step keeps the scan stable. | +| FS2 | NEW | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Reports knockdown/knockout targets**, not just amplification. RAVEN only flags reactions whose \|flux\| *rises* with the enforced product; reactions driven *toward zero* — the down-regulation/deletion candidates, arguably the most actionable — are classified here (`knockdown`/`knockout`). | +| FS3 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Gene-level aggregation** (`gene_targets`) mapping reaction targets to genes, plus the **full flux scan** retained — all as DataFrames, vs RAVEN's printed TSV + endpoint-only slope. | +| FS4 | CORRECTNESS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | Slope is the **regression slope** consistent with the selection criterion, not RAVEN's endpoint difference that disagreed with its own monotonicity filter. **Proposed back-port:** record the enforced target-flux level at each iteration into a vector and replace both the reported slope `abs(fseof.results(num,iterations) - fseof.results(num,1)) / abs(targetMax - targetMax/iterations)` and the `targets.slope` field with `polyfit(enforcedFlux, fseof.results(num,:), 1)` slope across all iterations. The label classification (amplify/knockdown/knockout, see FS2) should also use the slope of `|flux|` instead of comparing endpoints. | ## randomSampling (Phase 5 — implemented) -RAVEN `core/randomSampling.m` → `analysis/sampling.py` (`random_sampling`). The +RAVEN `analysis/randomSampling.m` → `analysis/sampling.py` (`random_sampling`). The random-objective / extreme-point method of Bordel et al. (2010) — **not** what `cobra.sampling` (OptGP/ACHR) does (those draw a near-uniform MCMC sample of the polytope interior), so it is a genuine addition, and it was wrongly listed as @@ -124,19 +124,19 @@ combination of reactions. | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| SAMP1 | EFFICIENCY | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **`good_reactions` (loop-free, flux-carrying objective candidates) via one `cobra` FVA pass** rather than RAVEN's hand-rolled per-reaction `parfor` that solves a separate LP maximising and minimising every reaction. FVA computes the same min/max ranges, optimised and optionally `loopless` (`cycleFreeFlux`), in far less code. | -| SAMP2 | ERGONOMICS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Reproducible** (`seed`) and **`n_objectives` is a parameter** — RAVEN has no seed and hard-codes 2 objective reactions (its docstring even claims 3, a transcription bug worth fixing upstream). | -| SAMP3 | ERGONOMICS | raven-python 🔨 | 🔨 | **Output is a samples × reactions DataFrame** (the `cobra.sampling` layout, directly usable with pandas/`analyzeSampling`-style stats) plus the reusable `good_reactions` list — instead of a reactions × samples matrix and a parallel `goodRxns` index vector that the caller must re-thread. | -| SAMP4 | CORRECTNESS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **`replace_max_bound` (RAVEN's `replaceBoundsWithInf`) defaults off and is scoped to sampling only.** It applies *after* `good_reactions` is found (FVA cannot evaluate `inf` bounds — it errors as 'unbounded'), and it can open unbounded loop directions; documented to pair with `min_flux`. RAVEN ran goodRxns detection on the inf-replaced model, conflating "loop reaction" with "unbounded objective". | +| SAMP1 | EFFICIENCY | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **`good_reactions` (loop-free, flux-carrying objective candidates) via one `cobra` FVA pass** rather than RAVEN's hand-rolled per-reaction `parfor` that solves a separate LP maximising and minimising every reaction. FVA computes the same min/max ranges, optimised and optionally `loopless` (`cycleFreeFlux`), in far less code. | +| SAMP2 | ERGONOMICS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Reproducible** (`seed`) and **`n_objectives` is a parameter** — RAVEN has no seed and hard-codes 2 objective reactions (its docstring even claims 3, a transcription bug worth fixing upstream). | +| SAMP3 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Output is a samples × reactions DataFrame** (the `cobra.sampling` layout, directly usable with pandas/`analyzeSampling`-style stats) plus the reusable `good_reactions` list — instead of a reactions × samples matrix and a parallel `goodRxns` index vector that the caller must re-thread. | +| SAMP4 | CORRECTNESS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **`replace_max_bound` (RAVEN's `replaceBoundsWithInf`) defaults off and is scoped to sampling only.** It applies *after* `good_reactions` is found (FVA cannot evaluate `inf` bounds — it errors as 'unbounded'), and it can open unbounded loop directions; documented to pair with `min_flux`. RAVEN ran goodRxns detection on the inf-replaced model, conflating "loop reaction" with "unbounded objective". | ## reporterMetabolites (Phase 5 — implemented) -RAVEN `core/reporterMetabolites.m` → `analysis/reporter.py` (`reporter_metabolites`). +RAVEN `analysis/reporterMetabolites.m` → `analysis/reporter.py` (`reporter_metabolites`). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| RM1 | EFFICIENCY + CORRECTNESS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Exact closed-form background correction** instead of RAVEN's 100 000-random-set Monte Carlo *per neighbour-count*. RAVEN samples with replacement from the scored-gene Z pool, so a random aggregate `Σz/√n` provably has mean `√n·μ` and std `σ` — the corrected score is exactly `(metZ − √n·μ)/σ`. Removes the slow sampling **and** its run-to-run randomness (deterministic results); back-portable to RAVEN. | -| RM2 | ERGONOMICS | raven-python 🔨 | 🔨 | Returns a sorted **DataFrame** per test (`all`/`up`/`down`) and takes gene→p-value / gene→fold-change **dicts**, vs RAVEN's parallel arrays + struct array + print/file side-effects. Neighbour genes come from cobra's metabolite→reaction→gene graph (no `rxnGeneMat`). | +| RM1 | EFFICIENCY + CORRECTNESS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Exact closed-form background correction** instead of RAVEN's 100 000-random-set Monte Carlo *per neighbour-count*. RAVEN samples with replacement from the scored-gene Z pool, so a random aggregate `Σz/√n` provably has mean `√n·μ` and std `σ` — the corrected score is exactly `(metZ − √n·μ)/σ`. Removes the slow sampling **and** its run-to-run randomness (deterministic results); back-portable to RAVEN. | +| RM2 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | Returns a sorted **DataFrame** per test (`all`/`up`/`down`) and takes gene→p-value / gene→fold-change **dicts**, vs RAVEN's parallel arrays + struct array + print/file side-effects. Neighbour genes come from cobra's metabolite→reaction→gene graph (no `rxnGeneMat`). | ## runINIT (Phase 4c — MILP core implemented) @@ -144,11 +144,11 @@ RAVEN `INIT/runINIT.m` → `init/init.py` (`run_init`). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| I1 | ERGONOMICS | raven-python 🔨 | 🔨 | **Clean optlang reformulation** of the INIT MILP instead of RAVEN's hand-built sparse `prob.A`/`blc`/`buc`/`vartype` arrays + fake "FAKEFORPM" metabolites. Standard include-indicator form `eps·x ≤ v ≤ ub·x` with objective `max Σ score·x + prod_weight·Σ sink`. Far more readable/reviewable; functional equivalence is the bar (PLAN §0). | -| I2 | ERGONOMICS | raven-python 🔨 | 🔨 | **`no_rev_loops` as a single `x_fwd + x_rev ≤ 1`** per reversible reaction, replacing RAVEN's auxiliary A/B/C metabolites with int1/int2 reactions and `C ub=-1` construction. Same effect (no spurious forward/back connectivity loop), a fraction of the machinery. | -| I3 | ERGONOMICS | raven-python 🔨 | 🔨 | **`present_mets` producibility via a small LP feasibility test** (sum of compartment-form drains ≥ 1), instead of mutating the live MILP's RHS one metabolite at a time. | -| I4 | CORRECTNESS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **MILP big-M is each reaction's own `ub`** (`v ≤ ub·x`), not RAVEN's fixed 1000; and `eps`/`prod_weight` are exposed parameters. RAVEN's hard-coded 1/1000/0.1/0.5 only suit ±1000-bounded models with O(1) scores — flagged as scale-dependent and tunable (don't blindly trust them). | -| I5 | ERGONOMICS | raven-python 🔨 | 🔨 | **Predictor-agnostic scoring**: `get_init_model` takes gene *or* reaction scores; gene scoring is generic (`gene_scores_from_expression` for the common RNA-seq path), so single-cell/HPA are just alternative upstream sources feeding the same gene→score table — rather than RAVEN's HPA/array-specific structs baked into `getINITModel`. | +| I1 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Clean optlang reformulation** of the INIT MILP instead of RAVEN's hand-built sparse `prob.A`/`blc`/`buc`/`vartype` arrays + fake "FAKEFORPM" metabolites. Standard include-indicator form `eps·x ≤ v ≤ ub·x` with objective `max Σ score·x + prod_weight·Σ sink`. Far more readable/reviewable; functional equivalence is the bar (PLAN §0). | +| I2 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **`no_rev_loops` as a single `x_fwd + x_rev ≤ 1`** per reversible reaction, replacing RAVEN's auxiliary A/B/C metabolites with int1/int2 reactions and `C ub=-1` construction. Same effect (no spurious forward/back connectivity loop), a fraction of the machinery. | +| I3 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **`present_mets` producibility via a small LP feasibility test** (sum of compartment-form drains ≥ 1), instead of mutating the live MILP's RHS one metabolite at a time. | +| I4 | CORRECTNESS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **MILP big-M is each reaction's own `ub`** (`v ≤ ub·x`), not RAVEN's fixed 1000; and `eps`/`prod_weight` are exposed parameters. RAVEN's hard-coded 1/1000/0.1/0.5 only suit ±1000-bounded models with O(1) scores — flagged as scale-dependent and tunable (don't blindly trust them). | +| I5 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Predictor-agnostic scoring**: `get_init_model` takes gene *or* reaction scores; gene scoring is generic (`gene_scores_from_expression` for the common RNA-seq path), so single-cell/HPA are just alternative upstream sources feeding the same gene→score table — rather than RAVEN's HPA/array-specific structs baked into `getINITModel`. | ## ftINIT (Phase 4d — in progress) @@ -157,96 +157,96 @@ RAVEN `INIT/ftINITInternalAlg.m` (+ orchestration) → `init/ftinit.py` (`run_ft | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| FT1 | EFFICIENCY | raven-python 🔨 | 🔨 | **Essential-reaction discovery via one FVA pass** (`find_task_essential_reactions`) — a reaction is essential for a task iff its FVA range excludes 0 (= RAVEN's constrain-to-0→infeasible), restricted to the flux-carrying candidates of a `pfba` solution. Replaces `getEssentialRxns`' per-reaction knockout loop. | -| FT2 | ERGONOMICS | raven-python 🔨 | 🔨 | **Clean optlang reformulation** of the 6-category MILP instead of RAVEN's hand-built block `prob.a` with the `pi/ni/ei/vprb/vnrvm…` figure. Positive-score reactions keep the continuous-indicator trick (no binary — the ftINIT speedup), encoded as `net_flux ≥ force_on·y`; only negative scores and reversible-direction get binaries. | -| FT3 | CORRECTNESS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Big-M is each reaction's own bound**, not RAVEN's fixed 100/1000; `force_on`/`force_on_ess` exposed. Scale-dependent, calibrated in 4d.7 (cf. I4). | +| FT1 | EFFICIENCY | raven-toolbox 🔨 | 🔨 | **Essential-reaction discovery via one FVA pass** (`find_task_essential_reactions`) — a reaction is essential for a task iff its FVA range excludes 0 (= RAVEN's constrain-to-0→infeasible), restricted to the flux-carrying candidates of a `pfba` solution. Replaces `getEssentialRxns`' per-reaction knockout loop. | +| FT2 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Clean optlang reformulation** of the 6-category MILP instead of RAVEN's hand-built block `prob.a` with the `pi/ni/ei/vprb/vnrvm…` figure. Positive-score reactions keep the continuous-indicator trick (no binary — the ftINIT speedup), encoded as `net_flux ≥ force_on·y`; only negative scores and reversible-direction get binaries. | +| FT3 | CORRECTNESS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Big-M is each reaction's own bound**, not RAVEN's fixed 100/1000; `force_on`/`force_on_ess` exposed. Scale-dependent, calibrated in 4d.7 (cf. I4). | | FT4 | (caveat) | RAVEN parity | — | **No loopless constraint** (matches RAVEN): the bare MILP can "include" an internal thermodynamically-infeasible cycle if it carries positive net score. Loop-free models rely on the staged pipeline + exchange handling and, at genome scale, real exchanges making cycles non-optimal. A loopless option could be added later. | -| FT9 | CORRECTNESS (robustness) | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Per-reaction essential forcing, clamped to capacity** (4d.7 calibration). Staging fixes previous-step reactions as essential; RAVEN forces them at `min(0.99·\|prev flux\|, 0.1)` so none is forced above the flux it carried (`essential_force`). On top, the forced magnitude is **clamped to the reaction's bound** so a low-capacity essential never produces an `lb>ub` error — it is forced to its capacity instead (RAVEN would error). | +| FT9 | CORRECTNESS (robustness) | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Per-reaction essential forcing, clamped to capacity** (4d.7 calibration). Staging fixes previous-step reactions as essential; RAVEN forces them at `min(0.99·\|prev flux\|, 0.1)` so none is forced above the flux it carried (`essential_force`). On top, the forced magnitude is **clamped to the reaction's bound** so a low-capacity essential never produces an `lb>ub` error — it is forced to its capacity instead (RAVEN would error). | | FT-met | (deferred) | — | ⬜ | **Metabolomics production bonus (4d.6) deferred.** The linear merge eliminates degree-2 detected metabolites, so it needs RAVEN's producer-group-mapping + `mon`/`vnrbm`/`vnrvm`/`vnim` negative-producer force-flux block — the most intricate MILP in ftINIT, for its least-used input. `ftinit(metabolomics=…)` raises `NotImplementedError`. | -| FT8 | ERGONOMICS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **`remove_low_score_genes`** (port of `removeLowScoreGenes`): prune negative-scoring genes from GPRs via a recursive walk of **cobra's GPR AST** — isozymes (OR) dropped keeping ≥1 (least-negative if all negative), complex subunits (AND) kept intact, nested groups handled — instead of RAVEN's regex subset-extraction with `#n#` placeholders. The all-negative tie-break is **deterministic** (reproducible) vs RAVEN's random. Wired into `ftinit(gene_scores=…)` (with `fill_tasks` for gap-filling) to complete the pipeline. Matches RAVEN's three docstring example cases. | -| FT7 | EFFICIENCY | raven-python 🔨 | 🔨 | **Task gap-filling** (`fill_tasks`, port of `ftINITFillGapsForAllTasks` + the fill MILP): per task, only if it is **infeasible** in the current model (a cheap LP gates the MILP), add the minimum-cost set of reference reactions (cost = `−min(score, −0.1)`) to satisfy the task's ranged metabolite bounds; accumulate across tasks; exchange reactions excluded as candidates. Built on the shared `apply_task_constraints` (factored out of `check_tasks`) + an optlang on/off MILP, instead of RAVEN's hand-built `[S pos neg int b var]` block and the custom `rxnScores`-field-on-the-model hack. Matches RAVEN tinitTests T0003 (gap at R7 → R7 added back). | -| FT6 | ERGONOMICS | raven-python 🔨 | 🔨 | **Staged pipeline** (`prep_init_model` → `PrepData`, `get_init_steps`/`InitStep`, `ftinit`): one-time reaction classification (`classify_reactions` = the `toIgnore` masks) + essential discovery + linear merge bundled into `PrepData`, reused per sample; the staged `'1+1'`/`'2+1'`/`full` schedule runs `run_ftinit` per step with previous-step reactions fixed as essential **in their flux direction** (carried as `essential_directions`, no model-flipping). RAVEN's Gurobi-specific per-step MIPGap retry schedule is dropped (solver-agnostic; tuning → 4d.7). Matches RAVEN tinitTests T0001 (no tasks → {R1,R4,R6,R8,R9,R10}; with R7/R10 spontaneous → {R1,R2,R4,R6,R7,R8}) and T0002 (task → essentials {R1,R7}, model {R1,R2,R4,R6,R7,R8,R9,R10}). | -| FT5 | EFFICIENCY | raven-python 🔨 | 🔨 | **Linear merge** (`merge_linear` + `group_rxn_scores`, port of `mergeLinear`/`groupRxnScores`): contracts degree-2-metabolite reaction chains (one producer, one consumer; reversibles included) into single reactions, losslessly shrinking the MILP (~⅓ fewer reactions on Human-GEM). Clean Python on a working representation with reversibility ≡ `lb<0` (RAVEN's `rev1·rev2` falls out of the most-constraining bound recompute); drops genes and objective on the reduced model (it exists only to feed the MILP, which scores via `group_rxn_scores`). Matches RAVEN's tinitTests T0004 exactly (group ids, merged bounds/reversibility, flipped reactions, grouped scores incl. the 0→0.01 handling). | +| FT8 | ERGONOMICS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **`remove_low_score_genes`** (port of `removeLowScoreGenes`): prune negative-scoring genes from GPRs via a recursive walk of **cobra's GPR AST** — isozymes (OR) dropped keeping ≥1 (least-negative if all negative), complex subunits (AND) kept intact, nested groups handled — instead of RAVEN's regex subset-extraction with `#n#` placeholders. The all-negative tie-break is **deterministic** (reproducible) vs RAVEN's random. Wired into `ftinit(gene_scores=…)` (with `fill_tasks` for gap-filling) to complete the pipeline. Matches RAVEN's three docstring example cases. | +| FT7 | EFFICIENCY | raven-toolbox 🔨 | 🔨 | **Task gap-filling** (`fill_tasks`, port of `ftINITFillGapsForAllTasks` + the fill MILP): per task, only if it is **infeasible** in the current model (a cheap LP gates the MILP), add the minimum-cost set of reference reactions (cost = `−min(score, −0.1)`) to satisfy the task's ranged metabolite bounds; accumulate across tasks; exchange reactions excluded as candidates. Built on the shared `apply_task_constraints` (factored out of `check_tasks`) + an optlang on/off MILP, instead of RAVEN's hand-built `[S pos neg int b var]` block and the custom `rxnScores`-field-on-the-model hack. Matches RAVEN tinitTests T0003 (gap at R7 → R7 added back). | +| FT6 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Staged pipeline** (`prep_init_model` → `PrepData`, `get_init_steps`/`InitStep`, `ftinit`): one-time reaction classification (`classify_reactions` = the `toIgnore` masks) + essential discovery + linear merge bundled into `PrepData`, reused per sample; the staged `'1+1'`/`'2+1'`/`full` schedule runs `run_ftinit` per step with previous-step reactions fixed as essential **in their flux direction** (carried as `essential_directions`, no model-flipping). RAVEN's Gurobi-specific per-step MIPGap retry schedule is dropped (solver-agnostic; tuning → 4d.7). Matches RAVEN tinitTests T0001 (no tasks → {R1,R4,R6,R8,R9,R10}; with R7/R10 spontaneous → {R1,R2,R4,R6,R7,R8}) and T0002 (task → essentials {R1,R7}, model {R1,R2,R4,R6,R7,R8,R9,R10}). | +| FT5 | EFFICIENCY | raven-toolbox 🔨 | 🔨 | **Linear merge** (`merge_linear` + `group_rxn_scores`, port of `mergeLinear`/`groupRxnScores`): contracts degree-2-metabolite reaction chains (one producer, one consumer; reversibles included) into single reactions, losslessly shrinking the MILP (~⅓ fewer reactions on Human-GEM). Clean Python on a working representation with reversibility ≡ `lb<0` (RAVEN's `rev1·rev2` falls out of the most-constraining bound recompute); drops genes and objective on the reduced model (it exists only to feed the MILP, which scores via `group_rxn_scores`). Matches RAVEN's tinitTests T0004 exactly (group ids, merged bounds/reversibility, flipped reactions, grouped scores incl. the 0→0.01 handling). | ## parseTaskList / checkTasks (Phase 4a — implemented) -RAVEN `core/parseTaskList.m` + `core/checkTasks.m` → `tasks/tasklist.py` + `tasks/check.py`. +RAVEN `tasks/parseTaskList.m` + `tasks/checkTasks.m` → `tasks/tasklist.py` + `tasks/check.py`. | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| T1 | ERGONOMICS | raven-python 🔨 | 🔨 | **Structured `Task` dataclass + `TaskResult`** instead of RAVEN's parallel-array struct and a printed report; programmatic access to per-task pass/fail/feasibility/error. | -| T2 | ERGONOMICS | raven-python 🔨 | 🔨 | **TSV-first task files** (stdlib `csv`); `.xlsx` still supported but via the lazy `[excel]` extra — no hard Excel dependency just to read a task list. | -| T3 | EFFICIENCY | raven-python 🔨 | 🔨 | Inputs/outputs imposed directly on cobra's **metabolite mass-balance constraint bounds** (the analogue of RAVEN's two-column `model.b`), and existing boundary reactions are auto-closed — so a model with open exchanges is handled correctly (RAVEN assumes a closed model and silently misbehaves otherwise). | +| T1 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Structured `Task` dataclass + `TaskResult`** instead of RAVEN's parallel-array struct and a printed report; programmatic access to per-task pass/fail/feasibility/error. | +| T2 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **TSV-first task files** (stdlib `csv`); `.xlsx` still supported but via the lazy `[excel]` extra — no hard Excel dependency just to read a task list. | +| T3 | EFFICIENCY | raven-toolbox 🔨 | 🔨 | Inputs/outputs imposed directly on cobra's **metabolite mass-balance constraint bounds** (the analogue of RAVEN's two-column `model.b`), and existing boundary reactions are auto-closed — so a model with open exchanges is handled correctly (RAVEN assumes a closed model and silently misbehaves otherwise). | ## fillGaps (Phase 4b — implemented) -RAVEN `core/fillGaps.m`. Only the **connectivity** mode is ported, as -`connect_blocked_reactions` ([gapfilling/fill.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/gapfilling/fill.py)) — +RAVEN `gapfilling/fillGaps.m`. Only the **connectivity** mode is ported, as +`connect_blocked_reactions` ([gapfilling/fill.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/gapfilling/fill.py)) — MILP via cobra/optlang (GLPK). RAVEN's other mode (fill to make the objective feasible) is `cobra.flux_analysis.gapfill` and is **cheatsheeted, not re-wrapped** (PLAN §1). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| GF1 | NEW (vs cobra) | raven-python 🔨 | 🔨 | **Connectivity gap-fill has no cobra equivalent**: add the minimum-penalty set of template reactions so *blocked* draft reactions can carry flux (cobra's `gapfill` only fills toward the objective). Ported as `connect_blocked_reactions` — a name that avoids confusion with `cobra.gapfill` and says what it does, vs RAVEN's overloaded `fillGaps(useModelConstraints=...)` boolean. | -| GF2 | ERGONOMICS | raven-python 🔨 | 🔨 | **Templates matched by `name[comp]`** (via `add_reactions_from_model`), so a template in a different identifier namespace than the draft still contributes — as RAVEN's name-based merge does. (For the targeted `cobra.gapfill` path, ids must be aligned first, since cobra matches by id — noted in the cheatsheet.) | +| GF1 | NEW (vs cobra) | raven-toolbox 🔨 | 🔨 | **Connectivity gap-fill has no cobra equivalent**: add the minimum-penalty set of template reactions so *blocked* draft reactions can carry flux (cobra's `gapfill` only fills toward the objective). Ported as `connect_blocked_reactions` — a name that avoids confusion with `cobra.gapfill` and says what it does, vs RAVEN's overloaded `fillGaps(useModelConstraints=...)` boolean. | +| GF2 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Templates matched by `name[comp]`** (via `add_reactions_from_model`), so a template in a different identifier namespace than the draft still contributes — as RAVEN's name-based merge does. (For the targeted `cobra.gapfill` path, ids must be aligned first, since cobra matches by id — noted in the cheatsheet.) | ## addRxns -RAVEN `core/addRxns.m` — add reactions from equation strings (or mets+coeffs), auto-creating +RAVEN `manipulation/addRxns.m` — add reactions from equation strings (or mets+coeffs), auto-creating metabolites/genes. Ported as `add_reactions_from_equations` -([manipulation/add.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/add.py)). +([manipulation/add.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/add.py)). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| A1 | ERGONOMICS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Readable matching mode instead of the `eqnType` integer.** RAVEN takes `eqnType=1/2/3` (match by id / by name in a given compartment / `name[comp]`), which is opaque at call sites. raven-python uses `mets_by="id"\|"name"` and auto-detects `name[comp]` per token. **MATLAB back-port:** accept a string keyword. | -| A2 | ERGONOMICS (bug-class) | raven-python 🔨 | 🔨 | **Error on duplicate reaction IDs explicitly.** RAVEN errors; cobra's `add_reactions` *silently ignores* a duplicate. raven-python keeps RAVEN's stricter behaviour (raise) rather than cobra's silent drop. | -| A3 | EFFICIENCY (reuse) | raven-python 🔨 | 🔨 | **Delegate equation/arrow/coefficient parsing and gene/met creation to cobra** (`build_reaction_from_string` semantics, GPR auto-creation) instead of re-implementing RAVEN's `constructS`/`addGenesRaven`. Only the genuinely cobra-absent pieces (name matching, compartment for new mets, strict policies) are hand-written. | +| A1 | ERGONOMICS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Readable matching mode instead of the `eqnType` integer.** RAVEN takes `eqnType=1/2/3` (match by id / by name in a given compartment / `name[comp]`), which is opaque at call sites. raven-toolbox uses `mets_by="id"\|"name"` and auto-detects `name[comp]` per token. **MATLAB back-port:** accept a string keyword. | +| A2 | ERGONOMICS (bug-class) | raven-toolbox 🔨 | 🔨 | **Error on duplicate reaction IDs explicitly.** RAVEN errors; cobra's `add_reactions` *silently ignores* a duplicate. raven-toolbox keeps RAVEN's stricter behaviour (raise) rather than cobra's silent drop. | +| A3 | EFFICIENCY (reuse) | raven-toolbox 🔨 | 🔨 | **Delegate equation/arrow/coefficient parsing and gene/met creation to cobra** (`build_reaction_from_string` semantics, GPR auto-creation) instead of re-implementing RAVEN's `constructS`/`addGenesRaven`. Only the genuinely cobra-absent pieces (name matching, compartment for new mets, strict policies) are hand-written. | | A4 | NEW | both 💡 | 💡 | **Infer compartment from a structured metabolite ID** (e.g. `atp_c` → `c`) as an alternative to requiring `compartment`. Not yet implemented; would reduce boilerplate for SBML-style IDs. Revisit alongside `addMets`. | ## changeGrRules -Ported as `change_gene_reaction_rules` ([manipulation/change.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/change.py)). +Ported as `change_gene_reaction_rules` ([manipulation/change.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/change.py)). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| G8 | EFFICIENCY (reuse) | raven-python 🔨 | 🔨 | **changeGrRules: delegate gene creation + normalization to cobra.** RAVEN calls `getGenesFromGrRules` + `addGenesRaven` + `standardizeGrRules` + rebuilds `rxnGeneMat`; cobra does all of that automatically on `gene_reaction_rule =`. The port keeps only the batch loop and the append (`(old) or (new)`) option. | +| G8 | EFFICIENCY (reuse) | raven-toolbox 🔨 | 🔨 | **changeGrRules: delegate gene creation + normalization to cobra.** RAVEN calls `getGenesFromGrRules` + `addGenesRaven` + `standardizeGrRules` + rebuilds `rxnGeneMat`; cobra does all of that automatically on `gene_reaction_rule =`. The port keeps only the batch loop and the append (`(old) or (new)`) option. | ## simplifyModel -Gap modes ported in [manipulation/simplify.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/simplify.py). +Gap modes ported in [manipulation/simplify.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/simplify.py). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| S3 | EFFICIENCY (scope) | raven-python 🔨 | 🔨 | **Only the cobra-absent modes ported as focused functions**, not a monolithic 8-flag `simplifyModel`. `deleteMinMax`→`find_blocked_reactions`, `deleteZeroInterval`→filter+prune, `deleteUnconstrained`→moot are cheatsheeted. dead-end / duplicate / constrain-reversible / group-linear are standalone, composable functions. | +| S3 | EFFICIENCY (scope) | raven-toolbox 🔨 | 🔨 | **Only the cobra-absent modes ported as focused functions**, not a monolithic 8-flag `simplifyModel`. `deleteMinMax`→`find_blocked_reactions`, `deleteZeroInterval`→filter+prune, `deleteUnconstrained`→moot are cheatsheeted. dead-end / duplicate / constrain-reversible / group-linear are standalone, composable functions. | ## mergeModels -Ported as `merge_models` ([manipulation/merge.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/merge.py)). +Ported as `merge_models` ([manipulation/merge.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/merge.py)). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| M1 | EFFICIENCY (scope) | raven-python 🔨 | 🔨 | **~560 lines of struct field-padding + manual S-matrix assembly dropped.** On `cobra.Model` the merge is just: unify metabolites by `name[comp]`, re-add reactions remapped to the merged metabolites, let cobra rebuild S and create genes. | -| M2 | ERGONOMICS | raven-python 🔨 | 🔨 | **Provenance via `notes['origin']`** (one place) instead of three parallel `rxnFrom`/`metFrom`/`geneFrom` fields. `match_by="name"|"id"` keyword replaces RAVEN's `metParam` string. | +| M1 | EFFICIENCY (scope) | raven-toolbox 🔨 | 🔨 | **~560 lines of struct field-padding + manual S-matrix assembly dropped.** On `cobra.Model` the merge is just: unify metabolites by `name[comp]`, re-add reactions remapped to the merged metabolites, let cobra rebuild S and create genes. | +| M2 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **Provenance via `notes['origin']`** (one place) instead of three parallel `rxnFrom`/`metFrom`/`geneFrom` fields. `match_by="name"|"id"` keyword replaces RAVEN's `metParam` string. | ## checkModelStruct -Ported (curation subset) as `check_model` ([utils/validate.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/validate.py)). +Ported (curation subset) as `check_model` ([utils/validate.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/validate.py)). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| V1 | EFFICIENCY (scope) | raven-python 🔨 | 🔨 | **Drop the struct/type/duplicate-ID/`lb>ub`/`rev` checks** — cobra's object model enforces or precludes them (DictList forbids duplicate IDs, Reaction rejects `lb>ub`, no `rev` field). Only the curation checks cobra lacks survive. | -| V2 | ERGONOMICS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Return structured `ModelIssue`s, not printed warnings** (RAVEN prints / throws). Programmatically filterable by `category`. **MATLAB back-port:** return an issues struct array. | +| V1 | EFFICIENCY (scope) | raven-toolbox 🔨 | 🔨 | **Drop the struct/type/duplicate-ID/`lb>ub`/`rev` checks** — cobra's object model enforces or precludes them (DictList forbids duplicate IDs, Reaction rejects `lb>ub`, no `rev` field). Only the curation checks cobra lacks survive. | +| V2 | ERGONOMICS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Return structured `ModelIssue`s, not printed warnings** (RAVEN prints / throws). Programmatically filterable by `category`. **MATLAB back-port:** return an issues struct array. | ## setParam / getElementalBalance -Ported as `set_parameters` ([manipulation/parameters.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/parameters.py)) -and `get_elemental_balance` ([utils/balance.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/balance.py)). +Ported as `set_parameters` ([manipulation/parameters.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/parameters.py)) +and `get_elemental_balance` ([utils/balance.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/balance.py)). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| | ~~P1~~ | ERGONOMICS | — | ↩ revised | A 6-mode keyword `set_parameters` was built then **trimmed** (review: not Pythonic — it re-wrapped cobra one-liners for `lb`/`ub`/`eq`/`obj`/`unc`). Only the `var` ±% band, which cobra has no idiom for, is kept as `set_variance_bounds`; the rest are documented as cobra idioms in the §1 cheatsheet. | -| B1 | ERGONOMICS (correctness) | raven-python 🔨 | 🔨 | **getElementalBalance: report `unknown` for missing formulas.** cobra's `check_mass_balance` silently treats a metabolite with no formula as contributing nothing, so the reaction can read as (un)balanced on incomplete data. raven-python flags those as `unknown` rather than guessing — preserving RAVEN's distinction (its `-1` status). | -| B2 | CORRECTNESS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Empty-stoichiometry reactions report `unknown`, not vacuous `balanced`.** A reaction with no metabolites used to fall through to `balanced` (any-over-empty is False; the zero-imbalance dict is empty). Same bug in MATLAB `core/getElementalBalance.m`: with no entries in `model.S(:,j)`, `balanceStatus` stays NaN through both loops and the final `isnan→1` step labels it `balanced`. **Proposed back-port:** add an `emptyRxns = full(sum(model.S~=0,1))==0` mask and `balanceStatus(emptyRxns) = min(-1, balanceStatus(emptyRxns))` before the `isnan→1` line, so empty reactions are marked "missing information". | +| B1 | ERGONOMICS (correctness) | raven-toolbox 🔨 | 🔨 | **getElementalBalance: report `unknown` for missing formulas.** cobra's `check_mass_balance` silently treats a metabolite with no formula as contributing nothing, so the reaction can read as (un)balanced on incomplete data. raven-toolbox flags those as `unknown` rather than guessing — preserving RAVEN's distinction (its `-1` status). | +| B2 | CORRECTNESS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Empty-stoichiometry reactions report `unknown`, not vacuous `balanced`.** A reaction with no metabolites used to fall through to `balanced` (any-over-empty is False; the zero-imbalance dict is empty). Same bug in MATLAB `queries/getElementalBalance.m`: with no entries in `model.S(:,j)`, `balanceStatus` stays NaN through both loops and the final `isnan→1` step labels it `balanced`. **Proposed back-port:** add an `emptyRxns = full(sum(model.S~=0,1))==0` mask and `balanceStatus(emptyRxns) = min(-1, balanceStatus(emptyRxns))` before the `isnan→1` line, so empty reactions are marked "missing information". | ## getRxnsInComp / getMetsInComp — not ported @@ -256,52 +256,52 @@ if a downstream consumer needs the `include_partial` (fully-contained vs touchin several places — and ask before re-adding (see process note: argue pros/cons for marginal WRAPs). Ported as `remove_metabolites` / `remove_genes` -([manipulation/remove.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/remove.py)). `removeReactions` was **not** +([manipulation/remove.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/remove.py)). `removeReactions` was **not** ported: with orphan cleanup kept coupled (decision: don't separate metabolites from genes), it is identical to `cobra.Model.remove_reactions(remove_orphans=...)`. | # | Cat | Target | Status | Improvement | |---|---|---|---|---| | ~~R1~~ | ERGONOMICS | — | ❌ rejected | Separable orphan-met vs orphan-gene cleanup was considered, then **dropped** by decision — keep them coupled like cobra. (Removed the `remove_reactions` wrapper entirely as a result.) | -| R2 | EFFICIENCY (reuse) | raven-python 🔨 | 🔨 | **GPR rewriting delegated to cobra's AST**, not RAVEN's `eval` of a `&&`/`\|\|`-substituted rule string. cobra's `remove_genes` already gives correct AND/OR semantics (removing one gene of `A and B` empties the rule; of `A or B` keeps the other). **MATLAB back-port:** replace `canRxnCarryFlux`'s `eval` with a parsed boolean tree (safer, no eval). | -| R3 | ERGONOMICS | raven-python 🔨 | 🔨 | **`blocked_reactions` policy as a clear keyword** (`remove`/`constrain`/`keep`) instead of RAVEN's `removeBlockedRxns` boolean — and `keep` (rewrite GPR, leave bounds) is a third option RAVEN lacks. | -| R4 | (review) | raven-python ⚠️ | 💡 | **`remove_metabolites` is a deletion candidate.** Its only value over cobra is `by_name` cross-compartment deletion, likely rarely used; revisit and possibly drop the wrapper. | +| R2 | EFFICIENCY (reuse) | raven-toolbox 🔨 | 🔨 | **GPR rewriting delegated to cobra's AST**, not RAVEN's `eval` of a `&&`/`\|\|`-substituted rule string. cobra's `remove_genes` already gives correct AND/OR semantics (removing one gene of `A and B` empties the rule; of `A or B` keeps the other). **MATLAB back-port:** replace `canRxnCarryFlux`'s `eval` with a parsed boolean tree (safer, no eval). | +| R3 | ERGONOMICS | raven-toolbox 🔨 | 🔨 | **`blocked_reactions` policy as a clear keyword** (`remove`/`constrain`/`keep`) instead of RAVEN's `removeBlockedRxns` boolean — and `keep` (rewrite GPR, leave bounds) is a third option RAVEN lacks. | +| R4 | (review) | raven-toolbox ⚠️ | 💡 | **`remove_metabolites` is a deletion candidate.** Its only value over cobra is `by_name` cross-compartment deletion, likely rarely used; revisit and possibly drop the wrapper. | ## readYAMLmodel / writeYAMLmodel RAVEN `io/readYAMLmodel.m` + `writeYAMLmodel.m` (+ private legacy parser). Ported as -`read_yaml_model`/`write_yaml_model` ([io/yaml.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/io/yaml.py)). +`read_yaml_model`/`write_yaml_model` ([io/yaml.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/io/yaml.py)). **Lens correction (no separate legacy parser).** RAVEN ships a 462-line `parseYAMLLegacy.m` for the `!!omap` dialect, and geckopy refuses it ("re-save from MATLAB"). But `!!omap` is **cobra's own YAML format**: `cobra.io.load_yaml_model` reads a real yeast-GEM.yml (4102 rxns) directly. So the -raven-python-unique capability the PLAN imagined (a legacy reader) is unnecessary; the real cobra-absent +raven-toolbox-unique capability the PLAN imagined (a legacy reader) is unnecessary; the real cobra-absent value is preserving `metaData` identity and RAVEN-only per-entry fields, which is what was built. | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| Y1 | EFFICIENCY (scope) | raven-python 🔨 | 🔨 | **Drop the bespoke legacy YAML parser; delegate to cobra's `!!omap` loader.** ~460 lines of RAVEN parsing not reimplemented. raven-python reads old RAVEN/Human-GEM YAML in pure Python with no MATLAB needed (geckopy can't — it tells users to re-save from MATLAB). | -| Y2 | ERGONOMICS (data loss) | raven-python 🔨 | 🔨 | **Don't silently drop model identity/provenance or RAVEN-only fields.** A plain `cobra.io.load_yaml_model` of a RAVEN file yields `model.id is None` and discards `smiles`/`deltaG`/`confidence_score`/etc. raven-python preserves them. **Routed by meaning** (not blindly to notes): chemical-structure identifiers `smiles`/`inchis` → cobra `annotation` (the MIRIAM-style store other tools read); genuinely non-standard data (`deltaG`, `confidence_score`, `metFrom`/`rxnFrom`, `protein`) → `notes`. Not invented as attributes (`met.deltaG`), since cobra only persists `annotation`/`notes` through copy/SBML/JSON/YAML. | +| Y1 | EFFICIENCY (scope) | raven-toolbox 🔨 | 🔨 | **Drop the bespoke legacy YAML parser; delegate to cobra's `!!omap` loader.** ~460 lines of RAVEN parsing not reimplemented. raven-toolbox reads old RAVEN/Human-GEM YAML in pure Python with no MATLAB needed (geckopy can't — it tells users to re-save from MATLAB). | +| Y2 | ERGONOMICS (data loss) | raven-toolbox 🔨 | 🔨 | **Don't silently drop model identity/provenance or RAVEN-only fields.** A plain `cobra.io.load_yaml_model` of a RAVEN file yields `model.id is None` and discards `smiles`/`deltaG`/`confidence_score`/etc. raven-toolbox preserves them. **Routed by meaning** (not blindly to notes): chemical-structure identifiers `smiles`/`inchis` → cobra `annotation` (the MIRIAM-style store other tools read); genuinely non-standard data (`deltaG`, `confidence_score`, `metFrom`/`rxnFrom`, `protein`) → `notes`. Not invented as attributes (`met.deltaG`), since cobra only persists `annotation`/`notes` through copy/SBML/JSON/YAML. | | Y4 | NEW | both 💡 | 💡 | **Upstream candidate: a first-class thermodynamics/confidence field.** `deltaG` and `confidence_score` live in `notes` because neither cobra nor SBML core has a home; if a standard slot (e.g. SBML fbc/groups or a cobra attribute) emerges, migrate them there. Also applies to MATLAB RAVEN's `metDeltaG`/`rxnConfidenceScores` consistency. | -| Y3 | NEW | raven-python 🔨 | 🔨 | **Emit cobra-native `!!omap` output** (via cobra's own dumper) — done, matching RAVEN `fa281a1`. Verified `cobra.io.load_yaml_model` reads the output. | -| Y5 | ERGONOMICS (correctness) | raven-python 🔨 | 🔨 | **Field placement realigned to `fa281a1`:** `smiles`/`ec-code` are in the cobra-owned `annotation` block (not top-level), `inchis` is top-level, and the top-level `notes` *string* (metNotes/rxnNotes) is handled rather than crashing a notes-as-dict assumption. | +| Y3 | NEW | raven-toolbox 🔨 | 🔨 | **Emit cobra-native `!!omap` output** (via cobra's own dumper) — done, matching RAVEN `fa281a1`. Verified `cobra.io.load_yaml_model` reads the output. | +| Y5 | ERGONOMICS (correctness) | raven-toolbox 🔨 | 🔨 | **Field placement realigned to `fa281a1`:** `smiles`/`ec-code` are in the cobra-owned `annotation` block (not top-level), `inchis` is top-level, and the top-level `notes` *string* (metNotes/rxnNotes) is handled rather than crashing a notes-as-dict assumption. | ## changeRxns -RAVEN `core/changeRxns.m` — change reaction equations. Ported as -`change_reaction_equations` ([manipulation/change.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/change.py)). +RAVEN `manipulation/changeRxns.m` — change reaction equations. Ported as +`change_reaction_equations` ([manipulation/change.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/change.py)). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| C1 | EFFICIENCY | raven-python 🔨 | 🔨 | **Edit the reaction in place** instead of RAVEN's copy-all-fields → `removeReactions` → `addRxns` → `permuteModel` round-trip. cobra mutates the same `Reaction` object, so every other field and the model order are preserved for free, with no O(n) re-sort. (Not a MATLAB back-port — the round-trip is inherent to the struct layout there.) | +| C1 | EFFICIENCY | raven-toolbox 🔨 | 🔨 | **Edit the reaction in place** instead of RAVEN's copy-all-fields → `removeReactions` → `addRxns` → `permuteModel` round-trip. cobra mutates the same `Reaction` object, so every other field and the model order are preserved for free, with no O(n) re-sort. (Not a MATLAB back-port — the round-trip is inherent to the struct layout there.) | ## getIndexes -RAVEN `core/getIndexes.m` — resolve a list of IDs / logical mask / index vector into positional +RAVEN `queries/getIndexes.m` — resolve a list of IDs / logical mask / index vector into positional indexes (or a logical array) for `rxns` / `mets` / `genes` / `metNames` / `metcomps` (and GECKO `ec.*` fields). -**Decision (raven-python): do NOT port the function.** cobra is object-oriented, so the central +**Decision (raven-toolbox): do NOT port the function.** cobra is object-oriented, so the central index-resolver that RAVEN's struct-of-parallel-arrays design requires is largely unnecessary. cobra's `DictList` already covers the use cases more idiomatically — `get_by_any` (mixed id/object/index → objects), `get_by_id` (O(1)), `query` (name/substring/regex), `index` (position), @@ -313,11 +313,11 @@ as upstream-only back-port candidates. | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| G1 | EFFICIENCY | MATLAB RAVEN only | 💡 | **Hash-based lookup instead of per-query linear scan.** RAVEN loops `find(strcmp(obj(i), searchIn))` per query → O(n·m). Build a `containers.Map` `{id: position}` once (O(n)), look up in O(1); `metcomps` likewise. (Moot for raven-python — cobra's `DictList` is already hashed.) | -| G5 | ERGONOMICS (bug) | MATLAB RAVEN only | 💡 | **Disambiguate the `[1 1 1]` mask-vs-index bug.** RAVEN's `if all(objects)` conflates a logical all-true mask with the index vector `[1 1 1]` (its own comment: "This gets weird if it's all 1"). Test `islogical(objects)` explicitly instead of `all(objects)`. (Moot for raven-python — input kind is decided by dtype.) | -| G7 | NEW | raven-python helper | 💡 | **Extract a reusable `name[comp]` parser/resolver.** The composite-id parsing buried in `getIndexes`'s `metcomps` branch is the one capability cobra lacks. Expose as a standalone `parse_name_comp` / `resolve_metabolite_by_name_comp`, reused by `addRxns`/`addTransport`/`mergeModels`. This is the only piece carried into raven-python. | +| G1 | EFFICIENCY | MATLAB RAVEN only | 💡 | **Hash-based lookup instead of per-query linear scan.** RAVEN loops `find(strcmp(obj(i), searchIn))` per query → O(n·m). Build a `containers.Map` `{id: position}` once (O(n)), look up in O(1); `metcomps` likewise. (Moot for raven-toolbox — cobra's `DictList` is already hashed.) | +| G5 | ERGONOMICS (bug) | MATLAB RAVEN only | 💡 | **Disambiguate the `[1 1 1]` mask-vs-index bug.** RAVEN's `if all(objects)` conflates a logical all-true mask with the index vector `[1 1 1]` (its own comment: "This gets weird if it's all 1"). Test `islogical(objects)` explicitly instead of `all(objects)`. (Moot for raven-toolbox — input kind is decided by dtype.) | +| G7 | NEW | raven-toolbox helper | 💡 | **Extract a reusable `name[comp]` parser/resolver.** The composite-id parsing buried in `getIndexes`'s `metcomps` branch is the one capability cobra lacks. Expose as a standalone `parse_name_comp` / `resolve_metabolite_by_name_comp`, reused by `addRxns`/`addTransport`/`mergeModels`. This is the only piece carried into raven-toolbox. | -**Obsoleted by cobra (no action — these were earlier raven-python proposals now covered by `DictList`):** +**Obsoleted by cobra (no action — these were earlier raven-toolbox proposals now covered by `DictList`):** predictable return type, return objects-not-positions, configurable missing-object policy across a batch, and substring/regex matching — all already provided by `get_by_any` / `get_by_id` / `query`. @@ -325,15 +325,15 @@ batch, and substring/regex matching — all already provided by `get_by_any` / ` ## standardizeGrRules -RAVEN `core/standardizeGrRules.m` — normalize grRule syntax + flag rules not in simple +RAVEN `manipulation/standardizeGrRules.m` — normalize grRule syntax + flag rules not in simple OR-of-AND-complex (DNF) form (`findPotentialErrors`). -**Decision (raven-python): port the lint half only.** cobra auto-normalizes a GPR on assignment +**Decision (raven-toolbox): port the lint half only.** cobra auto-normalizes a GPR on assignment (`"(G1 AND G2) OR G3"` is stored as `"(G1 and G2) or G3"`), so the normalization half is redundant. The non-DNF lint has no cobra equivalent and was ported as `find_non_dnf_grrules`/`is_dnf` -([utils/gpr.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/gpr.py)). +([utils/gpr.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/gpr.py)). | # | Cat | Target | Status | Improvement | |---|---|---|---|---| -| S1 | ERGONOMICS | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Return structured lint results, don't just print.** RAVEN's `findPotentialErrors` only emits a `warning()` string; you can't act on it programmatically. raven-python returns a list of `GPRIssue(reaction_id, gpr, reason)`. **MATLAB back-port:** return the `indexes2check`/messages as a struct array (it already computes `indexes2check` — just surface it cleanly instead of only warning). | -| S2 | EFFICIENCY (robustness) | raven-python 🔨 + MATLAB RAVEN 💡 | 🔨 | **Detect non-DNF via the boolean AST, not substring search.** RAVEN scans for the `) and (`, `) and`, `and (` substrings, which is brittle (sensitive to spacing/bracketing and to gene IDs containing those characters). raven-python walks cobra's GPR AST (`is_dnf`: no OR beneath any AND), which is exact. **MATLAB back-port:** parse the rule to a tree rather than string-matching. | +| S1 | ERGONOMICS | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Return structured lint results, don't just print.** RAVEN's `findPotentialErrors` only emits a `warning()` string; you can't act on it programmatically. raven-toolbox returns a list of `GPRIssue(reaction_id, gpr, reason)`. **MATLAB back-port:** return the `indexes2check`/messages as a struct array (it already computes `indexes2check` — just surface it cleanly instead of only warning). | +| S2 | EFFICIENCY (robustness) | raven-toolbox 🔨 + MATLAB RAVEN 💡 | 🔨 | **Detect non-DNF via the boolean AST, not substring search.** RAVEN scans for the `) and (`, `) and`, `and (` substrings, which is brittle (sensitive to spacing/bracketing and to gene IDs containing those characters). raven-toolbox walks cobra's GPR AST (`is_dnf`: no OR beneath any AND), which is exact. **MATLAB back-port:** parse the rule to a tree rather than string-matching. | diff --git a/README.md b/README.md index 99082221..097f7538 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ -# raven-python +# raven-toolbox -[![CI](https://github.com/SysBioChalmers/raven-python/actions/workflows/ci.yml/badge.svg)](https://github.com/SysBioChalmers/raven-python/actions/workflows/ci.yml) +[![CI](https://github.com/SysBioChalmers/raven-toolbox/actions/workflows/ci.yml/badge.svg)](https://github.com/SysBioChalmers/raven-toolbox/actions/workflows/ci.yml) **Reconstruction, Analysis and Visualisation of Metabolic Networks — in Python.** -`raven-python` is the Python counterpart of the +`raven-toolbox` is the Python counterpart of the [RAVEN Toolbox 2](https://github.com/SysBioChalmers/RAVEN) (MATLAB). It builds on [**cobrapy**](https://github.com/opencobra/cobrapy) for everything cobrapy already does well (simulation, standard analyses, SBML I/O, model manipulation) and adds the @@ -32,12 +32,12 @@ The canonical in-memory object is always a [`cobra.Model`](https://cobrapy.readt There is no parallel RAVEN struct, no `ravenCobraWrapper`-style adapter. RAVEN-specific fields that cobra doesn't model natively (`rxnMiriams`, `metDeltaG`, `rxnConfidenceScores`, …) live in cobra's `annotation` / `notes` dictionaries. This -avoids duplicating cobra's data model and keeps raven-python interoperable with the wider +avoids duplicating cobra's data model and keeps raven-toolbox interoperable with the wider COBRA ecosystem. ## Status -raven-python has been validated against MATLAB RAVEN on **Human-GEM** (5 Hart2015 cell-line +raven-toolbox has been validated against MATLAB RAVEN on **Human-GEM** (5 Hart2015 cell-line models, Jaccard 0.975–0.980 — see [docs/humangem_validation.md](docs/studies/humangem_validation.md)). The functional scope of the original RAVEN toolbox is covered with two principled omissions: @@ -53,8 +53,6 @@ omissions: Planned or partial functionality still on the books (full detail in **[docs/todo.md](docs/reference/todo.md)**): -- [ ] **Visualization (`plotting/`)** — pathway maps (Escher), omics-overlay and - flux-distribution plots. *The main open item.* - [ ] **Metabolomics-based scoring in tINIT / ftINIT** — passing a non-empty `metabolomics` argument currently raises `NotImplementedError`. - [ ] **Published binary release ZIPs** (BLAST / DIAMOND / HMMER) — the resolver in @@ -68,12 +66,12 @@ list — they are deliberately out of scope, not pending work. ## Installation (development) ```bash -git clone https://github.com/SysBioChalmers/raven-python -cd raven-python +git clone https://github.com/SysBioChalmers/raven-toolbox +cd raven-toolbox pip install -e ".[dev]" ``` -raven-python requires Python ≥ 3.11. Genome-scale (f)tINIT MILPs currently require **Gurobi** +raven-toolbox requires Python ≥ 3.11. Genome-scale (f)tINIT MILPs currently require **Gurobi** ([details on solver portability](docs/studies/init_solver_benchmark.md)); toy and unit-test work runs on the open-source GLPK. @@ -85,7 +83,7 @@ instructions. (A hosted ReadTheDocs site is not yet published.) ## Relationship to MATLAB RAVEN -`raven-python` is an independent Python reimplementation of the +`raven-toolbox` is an independent Python reimplementation of the [RAVEN Toolbox 2](https://github.com/SysBioChalmers/RAVEN), released under the permissive **MIT** license. If you use it in scientific work, please cite the RAVEN 2 paper: diff --git a/data/manifest.example.json b/data/manifest.example.json new file mode 100644 index 00000000..8ce150e2 --- /dev/null +++ b/data/manifest.example.json @@ -0,0 +1,31 @@ +{ + "manifest_version": 1, + "generated": "2026-06-10", + "data": { + "kegg": { + "version": "kegg116", + "description": "KEGG reference model, KO/reaction tables, and prokaryote/eukaryote HMM libraries for getKEGGModelForOrganism.", + "license": "Derived from the KEGG database; redistributed with permission from KEGG.", + "source": "https://github.com/SysBioChalmers/raven-toolbox/releases/tag/v0.1.0", + "files": { + "kegg116_core.tar.gz": { "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_core.tar.gz", "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 0 }, + "kegg116_taxonomy.gz": { "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_taxonomy.gz", "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 0 }, + "kegg116_prokaryotes.hmm.gz": { "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_prokaryotes.hmm.gz", "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 0 }, + "kegg116_eukaryotes.hmm.gz": { "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_eukaryotes.hmm.gz", "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 0 } + } + } + }, + "binaries": { + "diamond": { + "version": "2.1.9", + "provides": ["diamond"], + "description": "DIAMOND protein aligner (homology-based reconstruction).", + "license": "GPL-3.0-only — ship the upstream COPYING alongside each ZIP.", + "platforms": { + "linux-x86_64": { "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/diamond-2.1.9/diamond-2.1.9-linux-x86_64.zip", "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 0 }, + "macos-arm64": { "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/diamond-2.1.9/diamond-2.1.9-macos-arm64.zip", "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 0 }, + "windows-x86_64": { "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/diamond-2.1.9/diamond-2.1.9-windows-x86_64.zip", "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 0 } + } + } + } +} diff --git a/data/manifest.json b/data/manifest.json new file mode 100644 index 00000000..36a5f040 --- /dev/null +++ b/data/manifest.json @@ -0,0 +1,35 @@ +{ + "manifest_version": 1, + "generated": "2026-06-10", + "data": { + "kegg": { + "version": "kegg116", + "description": "KEGG reference model + KO/reaction tables (core bundle), taxonomy, and prokaryote/eukaryote HMM libraries for getKEGGModelForOrganism.", + "license": "Derived from the KEGG database; redistributed with permission from KEGG.", + "source": "https://github.com/SysBioChalmers/raven-toolbox/releases/tag/v0.1.0", + "files": { + "kegg116_core.tar.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_core.tar.gz", + "sha256": "155d5806d43db2fde5783fb124f8782bbcad390a1dd80879c520d2eac9d780e7", + "bytes": 48955539 + }, + "kegg116_eukaryotes.hmm.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_eukaryotes.hmm.gz", + "sha256": "2d48bc9935575d0f9ba4178bf2df19279bff866b49c1bf83a8e15787b11d6708", + "bytes": 134002309 + }, + "kegg116_prokaryotes.hmm.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_prokaryotes.hmm.gz", + "sha256": "d80cb2a22dec9fd8336b3998e3b96ee121672f63f4041cddaf09624fe739f1af", + "bytes": 153173750 + }, + "kegg116_taxonomy.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_taxonomy.gz", + "sha256": "1edc56da94d71433e5f08c133600292c311baaf33279a959518ab08389b0e538", + "bytes": 234693 + } + } + } + }, + "binaries": {} +} diff --git a/data/manifest.schema.json b/data/manifest.schema.json new file mode 100644 index 00000000..a426492a --- /dev/null +++ b/data/manifest.schema.json @@ -0,0 +1,82 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/SysBioChalmers/raven-toolbox/manifest.schema.json", + "title": "RAVEN data/binary manifest", + "description": "Language-agnostic registry of downloadable raven-toolbox / RAVEN data artefacts and external binary bundles. Consumed by the Python resolvers (raven_toolbox.data / raven_toolbox.binaries) and by MATLAB RAVEN. Every file carries a SHA256 so consumers verify integrity after download.", + "type": "object", + "required": ["manifest_version"], + "additionalProperties": false, + "properties": { + "manifest_version": { + "type": "integer", + "const": 1, + "description": "Format version of this manifest document." + }, + "generated": { + "type": "string", + "description": "ISO-8601 date the manifest was generated (informational)." + }, + "data": { + "type": "object", + "description": "Data-artefact datasets, keyed by dataset id (e.g. 'kegg'). Maps onto raven_toolbox.data._DATA_REGISTRY.", + "additionalProperties": { "$ref": "#/$defs/dataset" } + }, + "binaries": { + "type": "object", + "description": "External command-line tool bundles, keyed by bundle id (e.g. 'blast', 'diamond', 'hmmer'). Maps onto raven_toolbox.binaries._REGISTRY.", + "additionalProperties": { "$ref": "#/$defs/bundle" } + } + }, + "$defs": { + "file": { + "type": "object", + "required": ["url", "sha256"], + "additionalProperties": false, + "properties": { + "url": { "type": "string", "format": "uri", "description": "Direct download URL (GitHub release asset, Zenodo file, etc.)." }, + "sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$", "description": "Lowercase hex SHA256 of the file." }, + "bytes": { "type": "integer", "minimum": 0, "description": "File size in bytes (informational; for progress bars / sanity checks)." } + } + }, + "dataset": { + "type": "object", + "required": ["version", "files"], + "additionalProperties": false, + "properties": { + "version": { "type": "string", "description": "Dataset version tag, e.g. 'kegg116'. Used in the cache path." }, + "description": { "type": "string" }, + "license": { "type": "string", "description": "SPDX id or free text. NOTE: KEGG-derived artefacts are subject to KEGG's terms — confirm redistribution rights before publishing." }, + "doi": { "type": "string", "description": "Zenodo (or other) DOI for this dataset version, if archived." }, + "source": { "type": "string", "format": "uri", "description": "Human-facing page for the release/record (GitHub release or Zenodo landing page)." }, + "files": { + "type": "object", + "minProperties": 1, + "description": "Artefact files keyed by filename.", + "additionalProperties": { "$ref": "#/$defs/file" } + } + } + }, + "bundle": { + "type": "object", + "required": ["version", "provides", "platforms"], + "additionalProperties": false, + "properties": { + "version": { "type": "string", "description": "Upstream tool version, e.g. '2.16.0'." }, + "provides": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1, + "description": "Executable names this bundle provides, e.g. ['blastp', 'makeblastdb']." + }, + "description": { "type": "string" }, + "license": { "type": "string", "description": "Upstream tool license (e.g. DIAMOND is GPL-3.0-only — ship its license text alongside the ZIP)." }, + "platforms": { + "type": "object", + "minProperties": 1, + "description": "One entry per platform, keyed '-' (e.g. 'linux-x86_64', 'macos-arm64', 'windows-x86_64'). Matches raven_toolbox.binaries._platform_key().", + "additionalProperties": { "$ref": "#/$defs/file" } + } + } + } + } +} diff --git a/docs/README.md b/docs/README.md index 2ccac8ea..2a6d521f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,7 +6,7 @@ The rendered documentation is built with **Sphinx** (MyST Markdown) and hosted o Build it locally: ```bash -pip install -e ".[excel,plotting]" +pip install -e ".[excel]" pip install -r docs/requirements.txt sphinx-build -b html docs docs/_build/html # open docs/_build/html/index.html diff --git a/docs/archive/ftinit_review_and_plan.md b/docs/archive/ftinit_review_and_plan.md index 63e0d664..ffe13796 100644 --- a/docs/archive/ftinit_review_and_plan.md +++ b/docs/archive/ftinit_review_and_plan.md @@ -102,7 +102,7 @@ like we did for the HMM cut-offs (K15): **RAVEN/Gurobi coupling & quirks** - `ftINITFillGapsMILP` **hard-errors on glpk-via-COBRA**; "only tested with Gurobi". - raven-python uses **optlang**, so we are solver-agnostic — but the magic numbers and + raven-toolbox uses **optlang**, so we are solver-agnostic — but the magic numbers and `Seed=26` reproducibility were Gurobi-tuned. Validation must expect *equally optimal but not identical* reaction sets (alternative optima); compare on objective value and task feasibility, not exact rxn identity. @@ -129,7 +129,7 @@ pass/feasible per task. `prepINITModel` needs **essential-reaction discovery** ( reactions, and in which direction, are essential for each task) and the task metabolite set. That work was explicitly deferred from 4a/4c to here. -## 5. What cobra/raven-python already gives us +## 5. What cobra/raven-toolbox already gives us - MILP via **optlang** (HiGHS bundled, Gurobi/CPLEX if licensed) — `run_init` already builds INIT-style MILPs this way; ftINIT extends that builder. diff --git a/docs/archive/localization_design.md b/docs/archive/localization_design.md index 12a1366b..d18ea75b 100644 --- a/docs/archive/localization_design.md +++ b/docs/archive/localization_design.md @@ -1,6 +1,6 @@ # Localization (Phase 7) — critical review + redesign -Design note for porting RAVEN's `predictLocalization.m` to raven-python. **Status: proposal, +Design note for porting RAVEN's `predictLocalization.m` to raven-toolbox. **Status: proposal, not yet implemented**; the user-facing API and the algorithmic choices are settled here before code lands, because RAVEN's all-or-nothing redistribution doesn't match how the function actually gets used. @@ -55,13 +55,13 @@ reconsidering on a modern stack. > lower) predictor score is its own implicit penalty. Pick a large penalty for > effectively mono-localised genes. > * `mergeCompartments` and `copyToComps` are ported separately as -> `raven_python.manipulation.merge_compartments` / `copy_to_compartment` (they're useful +> `raven_toolbox.manipulation.merge_compartments` / `copy_to_compartment` (they're useful > independently of `predict_localization` — for flattening for analysis or building > dual-localised pathways). > * `mapCompartments` is **not** ported — its main use case overlaps with > `compare_models`. -## 2. Proposed `predict_localization` for raven-python +## 2. Proposed `predict_localization` for raven-toolbox Decompose the function into independent concerns: @@ -197,7 +197,7 @@ we have Gurobi already wired in. The MILP gives: * an explicit optimality certificate / gap, * faster solves on the small/medium problems this is (≪ 30k binaries even for a whole-genome model with 10 compartments), -* the same well-understood `mip_gap` / `time_limit` controls the rest of raven-python uses, +* the same well-understood `mip_gap` / `time_limit` controls the rest of raven-toolbox uses, * graceful degradation: at the time limit, return the best incumbent rather than the "current SA state" with no quality guarantee. @@ -209,7 +209,7 @@ because it's smaller than ftINIT — the per-reaction big-M is at most `|C|`, no ## 3. Pluggable predictors -The algorithm consumes a `gene × compartment` score table. raven-python ships: +The algorithm consumes a `gene × compartment` score table. raven-toolbox ships: * `load_wolfpsort(path)` — RAVEN-compatible WoLF PSORT output → score table. * (later) a thin adapter for **DeepLoc** (TSV with per-class probabilities). * The format is open: a user can build the DataFrame from any source. @@ -231,7 +231,7 @@ absent from the predictor's output (RAVEN's 0.5-everywhere fallback). ## 5. Open questions to resolve before implementing -1. **How does raven-python mark "uncertain" compartmentalization?** Proposal: a +1. **How does raven-toolbox mark "uncertain" compartmentalization?** Proposal: a `notes['localization'] = 'uncertain'` flag on reactions; or a passed-in set of ids. The "auto-pick-what-to-place" mode reads this. 2. **Multi-compartment gene scoring**: simple multi-counting (the same score in every diff --git a/docs/conf.py b/docs/conf.py index 68d81c8a..1d5c9fb1 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -1,4 +1,4 @@ -"""Sphinx configuration for the raven-python documentation. +"""Sphinx configuration for the raven-toolbox documentation. Markdown sources are rendered by MyST-Parser, so the docs stay authored in the same Markdown used on GitHub. The API reference is generated from the NumPy-style @@ -20,7 +20,7 @@ def _get_version() -> str: try: - from raven_python import __version__ + from raven_toolbox import __version__ return __version__ except Exception: # pragma: no cover - docs must still build @@ -28,7 +28,7 @@ def _get_version() -> str: # -- Project information ----------------------------------------------------- -project = "raven-python" +project = "raven-toolbox" author = "Eduard Kerkhoven" copyright = f"{date.today().year}, SysBioChalmers" release = _get_version() @@ -105,10 +105,10 @@ def _get_version() -> str: # -- HTML output ------------------------------------------------------------- html_theme = "furo" -html_title = f"raven-python {release}" +html_title = f"raven-toolbox {release}" html_static_path = ["_static"] html_theme_options = { - "source_repository": "https://github.com/SysBioChalmers/raven-python/", + "source_repository": "https://github.com/SysBioChalmers/raven-toolbox/", "source_branch": "develop", "source_directory": "docs/", } diff --git a/docs/guide/analysis_and_comparison.md b/docs/guide/analysis_and_comparison.md index 8cd68cdc..38f33ac5 100644 --- a/docs/guide/analysis_and_comparison.md +++ b/docs/guide/analysis_and_comparison.md @@ -1,22 +1,22 @@ # Analysis & comparison -## Analyses — {mod}`raven_python.analysis` +## Analyses — {mod}`raven_toolbox.analysis` RAVEN analyses that are not in cobrapy's core: -- {func}`raven_python.analysis.reporter_metabolites` — Reporter Metabolites, an - around-metabolite gene-score test. raven-python uses an exact closed-form background in +- {func}`raven_toolbox.analysis.reporter_metabolites` — Reporter Metabolites, an + around-metabolite gene-score test. raven-toolbox uses an exact closed-form background in place of RAVEN's Monte-Carlo. -- {func}`raven_python.analysis.fseof` — Flux Scanning based on Enforced Objective Flux: +- {func}`raven_toolbox.analysis.fseof` — Flux Scanning based on Enforced Objective Flux: regression slope + correlation, with amplify / knockdown / knockout classes and gene aggregation. -- {func}`raven_python.analysis.random_sampling` — random-objective flux sampling (wraps - cobra's samplers); {func}`raven_python.analysis.find_good_reactions` is the companion +- {func}`raven_toolbox.analysis.random_sampling` — random-objective flux sampling (wraps + cobra's samplers); {func}`raven_toolbox.analysis.find_good_reactions` is the companion screen. -## Comparison — {mod}`raven_python.comparison` +## Comparison — {mod}`raven_toolbox.comparison` -{func}`raven_python.comparison.compare_models` compares any number of models and returns tidy +{func}`raven_toolbox.comparison.compare_models` compares any number of models and returns tidy DataFrames (reaction / metabolite / gene / subsystem presence, pairwise Jaccard, and an optional `check_tasks` pass/fail matrix). Plotting and tSNE/MDS are deliberately left out — they are one-liners in seaborn / scikit-learn on the returned frames. diff --git a/docs/guide/context_specific.md b/docs/guide/context_specific.md index f55e48be..fab1075f 100644 --- a/docs/guide/context_specific.md +++ b/docs/guide/context_specific.md @@ -1,28 +1,28 @@ # Context-specific modeling (tINIT / ftINIT) Extract a tissue- or condition-specific model from a reference GEM plus gene scores derived -from omics data. Two algorithms are provided in {mod}`raven_python.init`. +from omics data. Two algorithms are provided in {mod}`raven_toolbox.init`. ## Scoring Gene scores drive both algorithms. Build them from expression with -{func}`raven_python.init.gene_scores_from_expression` and turn them into reaction scores via -{func}`raven_python.init.score_reactions_from_genes` (a GPR walk shared with the omics +{func}`raven_toolbox.init.gene_scores_from_expression` and turn them into reaction scores via +{func}`raven_toolbox.init.score_reactions_from_genes` (a GPR walk shared with the omics adapters — see the [omics guide](omics.md)). ## tINIT -- {func}`raven_python.init.run_init` — the classic INIT MILP (rewritten in optlang). -- {func}`raven_python.init.get_init_model` — the full tINIT pipeline (dead-end removal → +- {func}`raven_toolbox.init.run_init` — the classic INIT MILP (rewritten in optlang). +- {func}`raven_toolbox.init.get_init_model` — the full tINIT pipeline (dead-end removal → `run_init`). ## ftINIT (faster, staged) -- {func}`raven_python.init.run_ftinit` — the single-step ftINIT MILP (continuous indicators +- {func}`raven_toolbox.init.run_ftinit` — the single-step ftINIT MILP (continuous indicators for positive-score reactions; binaries only on negatives — the speedup over `run_init`). -- {func}`raven_python.init.ftinit` — the full pipeline: - {func}`raven_python.init.prep_init_model` → staged `run_ftinit` → - {func}`raven_python.init.fill_tasks` → {func}`raven_python.init.remove_low_score_genes`. +- {func}`raven_toolbox.init.ftinit` — the full pipeline: + {func}`raven_toolbox.init.prep_init_model` → staged `run_ftinit` → + {func}`raven_toolbox.init.fill_tasks` → {func}`raven_toolbox.init.remove_low_score_genes`. ## Tasks and defaults diff --git a/docs/guide/index.md b/docs/guide/index.md index 46572ce0..63d6055f 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -1,12 +1,12 @@ # User guide -Task-oriented guides for each raven-python capability. Every guide links to the relevant +Task-oriented guides for each raven-toolbox capability. Every guide links to the relevant [API reference](../reference/api/index.md) entries and, where one exists, the [validation study](../studies/index.md) that backs the defaults. If you are coming from MATLAB RAVEN, read the [migration map](../reference/migration.md) first — it tells you, function by function, what -moved to raven-python, what is now a cobra one-liner, and what was intentionally dropped. +moved to raven-toolbox, what is now a cobra one-liner, and what was intentionally dropped. ```{toctree} :maxdepth: 1 diff --git a/docs/guide/io_and_manipulation.md b/docs/guide/io_and_manipulation.md index 73c4876b..d2f312d3 100644 --- a/docs/guide/io_and_manipulation.md +++ b/docs/guide/io_and_manipulation.md @@ -1,37 +1,39 @@ # I/O & model manipulation -## I/O — {mod}`raven_python.io` +## I/O — {mod}`raven_toolbox.io` -raven-python keeps everything as a {class}`cobra.Model`, so cobra's SBML I/O works +raven-toolbox keeps everything as a {class}`cobra.Model`, so cobra's SBML I/O works unchanged. On top of that it adds the RAVEN-specific formats: -- {func}`raven_python.io.read_yaml_model` / {func}`raven_python.io.write_yaml_model` — +- {func}`raven_toolbox.io.read_yaml_model` / {func}`raven_toolbox.io.write_yaml_model` — cobra-standard YAML (the `!!omap` layout), transparently handling `.yml.gz`. RAVEN-only and GECKO `ec-*` side-fields are preserved on each entry's `notes` so a read→write round-trip is - lossless. -- {func}`raven_python.io.export_model_to_sif` — Cytoscape SIF (`rc` / `rr` / `cc` graphs). -- {func}`raven_python.io.export_to_excel` — the RAVEN 5-sheet workbook (RXNS / METS / COMPS / + lossless. The full schema (top-level layout, field order, quoting rules, the GECKO + `ec-*` and `metaData` extensions) is documented in + [the YAML model format reference](../reference/yaml_format.md). +- {func}`raven_toolbox.io.export_model_to_sif` — Cytoscape SIF (`rc` / `rr` / `cc` graphs). +- {func}`raven_toolbox.io.export_to_excel` — the RAVEN 5-sheet workbook (RXNS / METS / COMPS / GENES / MODEL). Requires the `excel` extra. Excel **import** is intentionally not provided. -- {func}`raven_python.io.export_for_git` — the Standard-GEM repository layout +- {func}`raven_toolbox.io.export_for_git` — the Standard-GEM repository layout (`model//…`), for version-controlled model repos. -## Manipulation — {mod}`raven_python.manipulation` +## Manipulation — {mod}`raven_toolbox.manipulation` Structural transforms that cobra does not cover cleanly: -- **Build / copy reactions:** {func}`raven_python.manipulation.add_reactions_from_equations` +- **Build / copy reactions:** {func}`raven_toolbox.manipulation.add_reactions_from_equations` (equation-string → reactions, matching metabolites by id / name / `name[comp]`), - {func}`raven_python.manipulation.add_transport_reactions`, - {func}`raven_python.manipulation.add_reactions_from_model`. -- **Merge:** {func}`raven_python.manipulation.merge_models` (N-model merge, unify metabolites + {func}`raven_toolbox.manipulation.add_transport_reactions`, + {func}`raven_toolbox.manipulation.add_reactions_from_model`. +- **Merge:** {func}`raven_toolbox.manipulation.merge_models` (N-model merge, unify metabolites by `name[comp]`). -- **GPR / bounds:** {func}`raven_python.manipulation.change_gene_reaction_rules`, - {func}`raven_python.manipulation.change_reaction_equations`, - {func}`raven_python.manipulation.set_variance_bounds`. -- **Simplify:** {func}`raven_python.manipulation.remove_dead_end_reactions`, +- **GPR / bounds:** {func}`raven_toolbox.manipulation.change_gene_reaction_rules`, + {func}`raven_toolbox.manipulation.change_reaction_equations`, + {func}`raven_toolbox.manipulation.set_variance_bounds`. +- **Simplify:** {func}`raven_toolbox.manipulation.remove_dead_end_reactions`, `remove_duplicate_reactions`, `constrain_reversible_reactions`, `group_linear_reactions`. -- **Topology:** {func}`raven_python.manipulation.convert_to_irreversible`, - {func}`raven_python.manipulation.expand_model` (split isozyme OR-GPRs), +- **Topology:** {func}`raven_toolbox.manipulation.convert_to_irreversible`, + {func}`raven_toolbox.manipulation.expand_model` (split isozyme OR-GPRs), and the compartment helpers `merge_compartments` / `copy_to_compartment`. See the [migration map](../reference/migration.md) for which RAVEN functions these diff --git a/docs/guide/localization.md b/docs/guide/localization.md index ccf4a9f4..1773bfac 100644 --- a/docs/guide/localization.md +++ b/docs/guide/localization.md @@ -1,18 +1,18 @@ # Sub-cellular localisation -{mod}`raven_python.localization` assigns reactions to compartments by MILP — deterministic +{mod}`raven_toolbox.localization` assigns reactions to compartments by MILP — deterministic (not simulated annealing), predictor-agnostic, and partial-update friendly. 1. **Load predictor scores** into the `gene × compartment` - {class}`raven_python.localization.LocalizationScores` frame: - {func}`raven_python.localization.load_wolfpsort` (WoLF PSORT summary output) or - {func}`raven_python.localization.load_deeploc` (DeepLoc 2 per-protein CSV). raven-python + {class}`raven_toolbox.localization.LocalizationScores` frame: + {func}`raven_toolbox.localization.load_wolfpsort` (WoLF PSORT summary output) or + {func}`raven_toolbox.localization.load_deeploc` (DeepLoc 2 per-protein CSV). raven-toolbox does **not** shell out to the predictor — run it separately and feed in its output. -2. **Predict / apply:** {func}`raven_python.localization.predict_localization` is the MILP +2. **Predict / apply:** {func}`raven_toolbox.localization.predict_localization` is the MILP entry point. Pass the set of reactions to relocate (everything else is pinned); extra compartments beyond a reaction's primary one pay a `multi_compartment_penalty`. With - `apply=False` you get a {class}`raven_python.localization.LocalizationProposal` diff to - inspect before committing; {func}`raven_python.localization.apply_localization` applies a + `apply=False` you get a {class}`raven_toolbox.localization.LocalizationProposal` diff to + inspect before committing; {func}`raven_toolbox.localization.apply_localization` applies a result. The defaults and accuracy (including a predictor-noise sweep) are validated against curated diff --git a/docs/guide/omics.md b/docs/guide/omics.md index aabf0efb..082e761e 100644 --- a/docs/guide/omics.md +++ b/docs/guide/omics.md @@ -1,17 +1,17 @@ # Omics integration -{mod}`raven_python.omics` ingests **Human Protein Atlas** data and turns it into the gene +{mod}`raven_toolbox.omics` ingests **Human Protein Atlas** data and turns it into the gene scores that drive context-specific extraction. -- **Proteomics:** {func}`raven_python.omics.parse_hpa` → - {func}`raven_python.omics.hpa_gene_scores`. -- **RNA-seq:** {func}`raven_python.omics.parse_hpa_rna` → - {func}`raven_python.omics.rna_gene_scores`. +- **Proteomics:** {func}`raven_toolbox.omics.parse_hpa` → + {func}`raven_toolbox.omics.hpa_gene_scores`. +- **RNA-seq:** {func}`raven_toolbox.omics.parse_hpa_rna` → + {func}`raven_toolbox.omics.rna_gene_scores`. Both return tidy pandas DataFrames, and the scoring adapters reuse -{func}`raven_python.init.score_reactions_from_genes` (a single source of truth for the GPR +{func}`raven_toolbox.init.score_reactions_from_genes` (a single source of truth for the GPR walk), so omics-derived scores plug straight into -{func}`raven_python.init.ftinit` / {func}`raven_python.init.get_init_model` — see the +{func}`raven_toolbox.init.ftinit` / {func}`raven_toolbox.init.get_init_model` — see the [context-specific modeling guide](context_specific.md). `HPA_LEVEL_SCORES` exposes the categorical-level → score mapping used for the proteomics diff --git a/docs/guide/quickstart.md b/docs/guide/quickstart.md index dd6fc826..f6b01136 100644 --- a/docs/guide/quickstart.md +++ b/docs/guide/quickstart.md @@ -12,11 +12,11 @@ pip install -e ".[dev]" ## Load and save a model The canonical in-memory object is a {class}`cobra.Model`, so everything cobra can do is -available unchanged. raven-python adds RAVEN's YAML reader/writer (cobra-standard layout +available unchanged. raven-toolbox adds RAVEN's YAML reader/writer (cobra-standard layout plus RAVEN/GECKO side-fields preserved on `notes`): ```python -from raven_python.io import read_yaml_model, write_yaml_model +from raven_toolbox.io import read_yaml_model, write_yaml_model model = read_yaml_model("model.yml") # transparently handles .yml.gz print(model.summary()) @@ -24,8 +24,8 @@ print(model.summary()) write_yaml_model(model, "out.yml", sort_ids=True) ``` -Other I/O and structural edits live in {mod}`raven_python.io` and -{mod}`raven_python.manipulation` — see the +Other I/O and structural edits live in {mod}`raven_toolbox.io` and +{mod}`raven_toolbox.manipulation` — see the [I/O & manipulation guide](io_and_manipulation.md). ## De-novo reconstruction (homology) @@ -34,12 +34,12 @@ Build a draft for a new organism from a curated template model and a BLAST/DIAMO ortholog search: ```python -from raven_python.reconstruction.homology import get_model_from_homology +from raven_toolbox.reconstruction.homology import get_model_from_homology draft = get_model_from_homology(template_model, ortholog_hits) ``` -The KEGG route ({func}`raven_python.reconstruction.kegg.get_kegg_model_for_organism`) and +The KEGG route ({func}`raven_toolbox.reconstruction.kegg.get_kegg_model_for_organism`) and the BLAST/DIAMOND helpers are covered in the [reconstruction guide](reconstruction.md). @@ -49,8 +49,8 @@ Extract a tissue/condition-specific model from a reference GEM plus omics-derive scores: ```python -from raven_python.omics import parse_hpa_rna, rna_gene_scores -from raven_python.init import ftinit +from raven_toolbox.omics import parse_hpa_rna, rna_gene_scores +from raven_toolbox.init import ftinit rna = parse_hpa_rna("rna_tissue.tsv") scores = rna_gene_scores(reference_model, rna, tissue="liver") diff --git a/docs/guide/reconstruction.md b/docs/guide/reconstruction.md index fea01edd..ab8c6d2f 100644 --- a/docs/guide/reconstruction.md +++ b/docs/guide/reconstruction.md @@ -2,38 +2,38 @@ Two independent routes build a draft model for an organism that has no curated GEM yet. -## Homology — {mod}`raven_python.reconstruction.homology` +## Homology — {mod}`raven_toolbox.reconstruction.homology` Transfer reactions from a curated **template** model to the target organism via an ortholog search. 1. Run the search (or bring your own table): - {func}`raven_python.reconstruction.homology.run_blast` / - {func}`raven_python.reconstruction.homology.run_diamond`, then - {func}`raven_python.reconstruction.homology.make_ortholog_hits` to get the canonical + {func}`raven_toolbox.reconstruction.homology.run_blast` / + {func}`raven_toolbox.reconstruction.homology.run_diamond`, then + {func}`raven_toolbox.reconstruction.homology.make_ortholog_hits` to get the canonical `gene × gene` hits DataFrame (bidirectional / best-hits-only policies supported). 2. Draft the model: - {func}`raven_python.reconstruction.homology.get_model_from_homology` — AST-based GPR + {func}`raven_toolbox.reconstruction.homology.get_model_from_homology` — AST-based GPR rewrite, configurable complex policy, returns a {class}`cobra.Model` (plus a `HomologyResult`). The external aligners are resolved from the pinned binary registry -({mod}`raven_python.binaries`). +({mod}`raven_toolbox.binaries`). -## KEGG — {mod}`raven_python.reconstruction.kegg` +## KEGG — {mod}`raven_toolbox.reconstruction.kegg` Draft directly from KEGG orthology, either for a KEGG-listed species or from your own protein FASTA via HMM search. - **KEGG species (no FASTA):** - {func}`raven_python.reconstruction.kegg.get_kegg_model_for_organism`. -- **Your sequences (HMM search):** {func}`raven_python.reconstruction.kegg.assign_kos` → - {func}`raven_python.reconstruction.kegg.get_kegg_model_from_sequences`. The HMM cut-off + {func}`raven_toolbox.reconstruction.kegg.get_kegg_model_for_organism`. +- **Your sequences (HMM search):** {func}`raven_toolbox.reconstruction.kegg.assign_kos` → + {func}`raven_toolbox.reconstruction.kegg.get_kegg_model_from_sequences`. The HMM cut-off defaults are calibrated in the [KEGG HMM cut-off study](../studies/kegg_hmm_cutoff_calibration.md). The KEGG artefact bundle (KO tables, reference model, HMM libraries) is fetched by -{mod}`raven_python.data`; building and publishing it is a maintainer task — see +{mod}`raven_toolbox.data`; building and publishing it is a maintainer task — see [Maintaining KEGG data](../maintenance/maintaining_kegg_data.md) and the [KEGG data format](../maintenance/kegg_data_format.md). diff --git a/docs/guide/tasks_and_gapfilling.md b/docs/guide/tasks_and_gapfilling.md index ae778060..12044a12 100644 --- a/docs/guide/tasks_and_gapfilling.md +++ b/docs/guide/tasks_and_gapfilling.md @@ -1,22 +1,22 @@ # Metabolic tasks & gap-filling -## Tasks — {mod}`raven_python.tasks` +## Tasks — {mod}`raven_toolbox.tasks` A metabolic task asserts that a model can (or cannot) produce/consume a set of metabolites under given constraints. -- {class}`raven_python.tasks.Task` + {func}`raven_python.tasks.parse_task_list` — the +- {class}`raven_toolbox.tasks.Task` + {func}`raven_toolbox.tasks.parse_task_list` — the task-list file format. -- {func}`raven_python.tasks.check_tasks` + {class}`raven_python.tasks.TaskResult` — run tasks +- {func}`raven_toolbox.tasks.check_tasks` + {class}`raven_toolbox.tasks.TaskResult` — run tasks against a model. `check_tasks` reuses one model across the whole list (no per-task model copy) — at genome scale ~12× faster than a copy-per-task implementation. -- {func}`raven_python.tasks.find_task_essential_reactions` — the reactions a model must use to +- {func}`raven_toolbox.tasks.find_task_essential_reactions` — the reactions a model must use to satisfy a task list. This is the input to (f)tINIT's task layer (see the [context-specific modeling guide](context_specific.md)). -## Connectivity gap-filling — {mod}`raven_python.gapfilling` +## Connectivity gap-filling — {mod}`raven_toolbox.gapfilling` -{func}`raven_python.gapfilling.connect_blocked_reactions` adds the fewest (lowest-penalty) +{func}`raven_toolbox.gapfilling.connect_blocked_reactions` adds the fewest (lowest-penalty) template reactions so reactions that are blocked in a draft can carry flux — the connectivity flavour of RAVEN's `fillGaps`. diff --git a/docs/index.md b/docs/index.md index 45675677..83843caf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,8 +1,8 @@ -# raven-python +# raven-toolbox **Reconstruction, Analysis and Visualisation of Metabolic Networks — in Python.** -`raven-python` is the Python counterpart of the +`raven-toolbox` is the Python counterpart of the [RAVEN Toolbox 2](https://github.com/SysBioChalmers/RAVEN) (MATLAB). It builds on [**cobrapy**](https://cobrapy.readthedocs.io) for everything cobrapy already does well (simulation, standard analyses, SBML I/O, model manipulation) and adds the functionality @@ -26,7 +26,7 @@ that is unique to RAVEN: The canonical in-memory object is always a {class}`cobra.Model`. There is no parallel RAVEN struct and no `ravenCobraWrapper`-style adapter — RAVEN-specific fields that cobra doesn't model natively live in cobra's `annotation` / `notes` dictionaries. This keeps -raven-python interoperable with the wider COBRA ecosystem. +raven-toolbox interoperable with the wider COBRA ecosystem. ::: ## Where to start @@ -45,7 +45,7 @@ Install, load a model, and run your first reconstruction / ftINIT extraction. :link: reference/migration :link-type: doc -The function-by-function map from RAVEN to raven-python (and cobrapy). +The function-by-function map from RAVEN to raven-toolbox (and cobrapy). ::: :::{grid-item-card} 📚 User guide @@ -65,7 +65,7 @@ Every public function and class, generated from the docstrings. ## Status -raven-python has been validated against MATLAB RAVEN on **Human-GEM** (5 Hart2015 cell-line +raven-toolbox has been validated against MATLAB RAVEN on **Human-GEM** (5 Hart2015 cell-line models, Jaccard 0.975–0.980 — see [the Human-GEM validation study](studies/humangem_validation.md)). The functional scope of the original toolbox is covered, with two principled omissions: **MetaCyc-based reconstruction** (flagged for removal from MATLAB RAVEN too) and **dynamic diff --git a/docs/installation.md b/docs/installation.md index 6be617ac..f998eeac 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,12 +1,12 @@ # Installation -raven-python requires **Python ≥ 3.11**. +raven-toolbox requires **Python ≥ 3.11**. ## From a source checkout (development) ```bash -git clone https://github.com/SysBioChalmers/raven-python -cd raven-python +git clone https://github.com/SysBioChalmers/raven-toolbox +cd raven-toolbox pip install -e ".[dev]" ``` @@ -17,12 +17,11 @@ Some features need extra packages, exposed as optional extras: | Extra | Pulls in | Needed for | | --- | --- | --- | -| `excel` | `openpyxl` | {func}`raven_python.io.excel.export_to_excel` | -| `plotting` | `matplotlib` | visualisation (Phase 6, in progress) | +| `excel` | `openpyxl` | {func}`raven_toolbox.io.excel.export_to_excel` | | `dev` | `pytest`, `pytest-cov`, `ruff` | running the test-suite / linting | ```bash -pip install -e ".[excel,plotting,dev]" +pip install -e ".[excel,dev]" ``` ## Solvers @@ -35,7 +34,7 @@ comparison and portability notes. ## External binaries Homology and KEGG-HMM reconstruction shell out to **BLAST+ / DIAMOND / HMMER**. These are -not Python packages; raven-python resolves them from a version-pinned, SHA256-verified -release registry (see {mod}`raven_python.binaries`). The +not Python packages; raven-toolbox resolves them from a version-pinned, SHA256-verified +release registry (see {mod}`raven_toolbox.binaries`). The [binary maintenance guide](maintenance/maintaining_binaries.md) covers building and publishing those release ZIPs. diff --git a/docs/maintenance/data_manifest.md b/docs/maintenance/data_manifest.md new file mode 100644 index 00000000..047dff07 --- /dev/null +++ b/docs/maintenance/data_manifest.md @@ -0,0 +1,155 @@ +# Data & binary manifest + +Large artefacts (KEGG tables / HMMs, template models) and external-binary bundles +(BLAST / DIAMOND / HMMER) are **not** committed to the code repository. They are published +as downloadable assets and described by a single, language-agnostic **manifest** that both +raven-toolbox and MATLAB RAVEN read. Every file carries a **SHA256**, so consumers verify +integrity after download. + +- Format: [`data/manifest.schema.json`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/data/manifest.schema.json) (JSON Schema) +- Worked example: [`data/manifest.example.json`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/data/manifest.example.json) +- Live manifest: [`data/manifest.json`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/data/manifest.json) (empty until assets are published) + +The manifest is a superset of the two runtime registries: + +| Manifest section | Runtime registry | +| --- | --- | +| `data` | {data}`raven_toolbox.data._DATA_REGISTRY` | +| `binaries` | `raven_toolbox.binaries._REGISTRY` | + +```json +{ + "manifest_version": 1, + "data": { "": { "version": "...", "doi": "...", "files": { "": {"url": "...", "sha256": "...", "bytes": 0} } } }, + "binaries": { "": { "version": "...", "provides": ["..."], "platforms": { "-": {"url": "...", "sha256": "...", "bytes": 0} } } } +} +``` + +## Consuming it — Python + +Point raven-toolbox at a manifest and the resolvers populate themselves on first use, +verifying each download's checksum: + +```bash +export RAVEN_PYTHON_MANIFEST=https://github.com/SysBioChalmers/raven-toolbox/releases/download/manifest-v1/manifest.json +``` + +```python +from raven_toolbox import manifest +manifest.load_into_registries() # or load_into_registries("/path/or/url") +# now data.ensure_kegg_data() / binaries.ensure_binary("diamond") resolve from the manifest +``` + +If `RAVEN_PYTHON_MANIFEST` is set, `data.ensure_*` and `binaries.ensure_binary` load it +lazily — no explicit call needed. + +## Consuming it — MATLAB + +The same JSON is trivial to read from MATLAB (`webread` + `jsondecode`), download +(`websave`), and verify (Java's `MessageDigest`, always available in MATLAB): + +```matlab +function file = ensureDataFile(manifestUrl, dataset, name, cacheDir) + m = jsondecode(webread(manifestUrl, weboptions('ContentType','text'))); + entry = m.data.(dataset).files.(matlab.lang.makeValidName(name)); + file = fullfile(cacheDir, name); + if ~isfile(file) + websave(file, entry.url); + end + assert(strcmp(sha256(file), entry.sha256), 'SHA256 mismatch for %s', name); +end + +function hex = sha256(file) + fid = fopen(file, 'r'); raw = fread(fid, Inf, '*uint8'); fclose(fid); + md = java.security.MessageDigest.getInstance('SHA-256'); + md.update(raw); + hex = lower(reshape(dec2hex(typecast(md.digest(), 'uint8'))', 1, [])); +end +``` + +## Publishing — generating manifest entries + +After uploading a release's files, add/update an entry with the maintainer script +([`scripts/make_registry_snippet.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/scripts/make_registry_snippet.py)), +which computes each SHA256 and byte size: + +```bash +python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ + --target data --dataset kegg --version kegg116 --dir artefacts \ + --base-url https://github.com/SysBioChalmers/raven-toolbox/releases/download/kegg-kegg116 \ + --doi 10.5281/zenodo.0000000 --source https://zenodo.org/records/0000000 + +python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ + --target binary --bundle diamond --version 2.1.9 --provides diamond --dir zips \ + --base-url https://github.com/SysBioChalmers/raven-toolbox/releases/download/diamond-2.1.9 \ + --license GPL-3.0-only +``` + +## Where to host + +Release **assets are stored separately from the git tree** (GitHub keeps them in a blob +store), so attaching them to a release does **not** bloat the repository. A dedicated assets +repository is therefore **optional** — attach the assets to releases on an existing RAVEN +repo (this one, or MATLAB [RAVEN](https://github.com/SysBioChalmers/RAVEN)) and have **both +packages reuse the same release-asset URLs** via this manifest. + +Use **dedicated tags** for the assets — e.g. `kegg-kegg116`, `diamond-2.1.9` — rather than +attaching them to code-milestone releases like `v0.1.0a1`. KEGG data updates roughly yearly +while the code changes often; dedicated tags keep the two cadences decoupled while still +living in one repository. The manifest's per-dataset `version` does the rest (it namespaces +the download cache). + +Both GitHub Releases and Zenodo are just URLs in the manifest, so consumers don't care — +mix them per file: + +- **GitHub Releases** — simplest, free, language-agnostic, up to **~2 GB per file**. The + default home for the manifest and most assets. +- **Zenodo** — adds a citable **DOI**, long-term archival, and handles files **larger than + 2 GB** (up to 50 GB/record). Use it for individual large HMM libraries or anything you want + citable; point just that file's `url` at the Zenodo record. + +### Auto-publishing to Zenodo from GitHub (only if you need DOIs / >2 GB files) + +:::{important} +The **native GitHub↔Zenodo integration** (flip a switch, publish a Release → DOI) archives +the **repository source zipball** at the tag — it does **not** capture files attached to the +Release. So it only works for assets *committed into the repo*, which defeats the purpose for +multi-GB binaries. Use it for a *software* DOI, not for the data assets. +::: + +If you do want Zenodo DOIs (or need to host files >2 GB), keep it GitHub-driven with a small +**GitHub Action** that, on release, uploads the assets to Zenodo via its REST API (e.g. +[`zenodraft`](https://github.com/zenodraft/zenodraft)). You cut a normal GitHub Release with +the files attached; the Action mirrors them to Zenodo and mints a new version DOI. Drop this +into whichever repo hosts the asset releases as `.github/workflows/zenodo.yml`: + +```yaml +name: Mirror release assets to Zenodo +on: + release: + types: [published] +jobs: + zenodo: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: "20" } + - name: Download this release's assets + run: gh release download "${{ github.event.release.tag_name }}" --dir assets + env: { GH_TOKEN: "${{ github.token }}" } + - name: Deposit a new version on Zenodo + run: npx zenodraft@latest version create --publish ${{ vars.ZENODO_CONCEPT_DOI }} assets/* + env: { ZENODO_ACCESS_TOKEN: "${{ secrets.ZENODO_TOKEN }}" } +``` + +Then record the resulting DOI in the manifest via the `--doi` flag above. Net result: you only +ever interact with GitHub Releases; Zenodo archiving + DOIs happen automatically. + +## Per-asset recommendations + +| Asset | Home | Notes | +| --- | --- | --- | +| **Software binaries** (BLAST / DIAMOND / HMMER) | **bioconda** preferred; or release ZIPs via the resolver | DIAMOND is **GPL-3.0** — ship its license text in the ZIP; keep it as a separate asset, never bundled into the MIT wheel. | +| **KEGG HMMs / tables** | GitHub release (dedicated `kegg-*` tag); Zenodo for libraries >2 GB | Derived from the KEGG dump and **redistributed with permission from KEGG**. Note the provenance in the release notes / manifest `license`. | +| **Template models** (Human-GEM, yeast-GEM) | **Don't re-host** | Fetch from their canonical repos by pinned release tag — respects their licenses and avoids stale copies. | diff --git a/docs/maintenance/index.md b/docs/maintenance/index.md index a382f421..542180ab 100644 --- a/docs/maintenance/index.md +++ b/docs/maintenance/index.md @@ -8,6 +8,9 @@ rebuild and release them. artefact releases. - **[Maintaining binaries](maintaining_binaries.md)** — building and publishing the external-binary (BLAST / DIAMOND / HMMER) ZIP releases. +- **[Data & binary manifest](data_manifest.md)** — the shared manifest that lists every + published artefact / binary (consumed by raven-toolbox and MATLAB RAVEN), where to host + assets (GitHub Releases vs Zenodo), and the GitHub→Zenodo auto-publish setup. ```{toctree} :hidden: @@ -15,4 +18,5 @@ rebuild and release them. kegg_data_format maintaining_kegg_data maintaining_binaries +data_manifest ``` diff --git a/docs/maintenance/kegg_data_format.md b/docs/maintenance/kegg_data_format.md index efb6d130..7ccc6881 100644 --- a/docs/maintenance/kegg_data_format.md +++ b/docs/maintenance/kegg_data_format.md @@ -1,6 +1,6 @@ # KEGG relational-table storage format -This note records *why* raven-python stores its KEGG-derived relational tables as +This note records *why* raven-toolbox stores its KEGG-derived relational tables as **gzipped TSV**, and what other options we deliberately deferred. It applies to the maintainer-built KEGG artefacts described in PLAN.md §2.3b — the `ko_reaction`, `organism_gene_ko`, KO-name, and reaction-flag tables. @@ -11,45 +11,47 @@ tables (the YAML I/O transparently gzips on a `.gz` suffix). On the real KEGG du this is ~1.1 MB (vs ~30 MB as SBML) for the full 12k-reaction gene-free model. End users do not build any of this: the published artefacts are fetched and cached -under `~/.cache/raven-python/data/kegg-/` by `ensure_data` (see -`raven_python.data`), mirroring how binaries are provisioned. +under `~/.cache/raven-toolbox/data/kegg-/` by `ensure_data` (see +`raven_toolbox.data`), mirroring how binaries are provisioned. The core tables and the +reference model are distributed together as a single `_core.tar.gz` +(`ensure_kegg_data` extracts it on first use); the per-file format below is unchanged. +The HMM libraries and the `taxonomy` file are separate, individually-fetched artefacts. ## Decision (current) -- **Small tables** (`ko_reaction`, `ko_names`, `rxn_flags`): **gzipped TSV - (`.tsv.gz`)**. Each is well under 1 MB, so compression choice is irrelevant; - gzip keeps them MATLAB-native and dependency-free. -- **The large `organism_gene_ko` table**: **xz-compressed TSV - (`organism_gene_ko.tsv.xz`), with rows sorted by `(organism, gene)`**. +- **All tables** (`ko_reaction`, `ko_names`, `rxn_flags`, and the large + `organism_gene_ko`): **gzipped TSV (`.tsv.gz`)**. Published assets are + version-prefixed, e.g. `kegg116_organism_gene_ko.tsv.gz`. +- The large `organism_gene_ko` table keeps its rows **sorted by `(organism, gene)`**. -Why the large table differs. It carries KEGG's ~9M gene↔KO associations and -dominates the artefact set (≈78 MB as unsorted gzipped TSV). Two cheap, -stdlib-only changes cut that to ≈27 MB (2.9×): +Why everything is gzip — even the big table. `organism_gene_ko` carries KEGG's +~9M gene↔KO associations and dominates the artefact set. Sorting by +`(organism, gene)` before writing makes gene IDs from one organism adjacent +(shared locus-tag/numeric prefixes), which both helps the compressor and matches +the by-organism query pattern in `get_kegg_model_for_organism`; the sort is an +external merge sort bounded to `chunk_rows` in memory (see +`stream_organism_gene_ko`), so it stays scalable. On the real dump this lands +around ~74 MB. -1. **Sort by `(organism, gene)`** before writing. Gene IDs from one organism - share long common prefixes (locus tags, numeric runs); sorting makes them - adjacent so the compressor can fold them. This alone takes 78 → 48 MB and - happens to match the by-organism query pattern in - `get_kegg_model_for_organism`. The sort is an external merge sort bounded to - `chunk_rows` in memory (see `stream_organism_gene_ko`), so it stays scalable. -2. **xz instead of gzip** (Python stdlib `lzma`). Its larger dictionary captures - cross-row redundancy gzip's 32 KB window misses: sorted + xz reaches ≈27 MB. +We previously xz-compressed this one file (≈27 MB, ~2.9× smaller). We switched it +to **gzip** so the *same* artefact is readable by MATLAB's built-in `gunzip` with +no external tool — the artefacts are shared with MATLAB RAVEN, and `.xz` would +force an external `xz`/`unxz` dependency. The size cost (~74 vs ~27 MB on a +once-per-release download) buys a dependency-free, cross-tool, cross-platform +read; xz's larger dictionary is not worth a MATLAB toolchain requirement. -- **pandas reads/writes both with zero extra dependencies** — compression is - inferred from the `.gz`/`.xz` suffix; `lzma` and `gzip` are both stdlib, so - this works natively on Windows, macOS, and Linux with no external binary. -- **MATLAB caveat:** `readtable` reads gzipped TSV after a `gunzip`, but MATLAB - has no built-in xz decompressor. The small tables stay MATLAB-native; the - large table needs an external `unxz` (or Java/`7-Zip`) before `readtable` on - the MATLAB side. The xz file is raven-python's (Python) primary artefact; this - trades a little MATLAB convenience on the one big file for a ~3× size cut. +- **pandas reads/writes gzip with zero extra dependencies** — compression is + inferred from the `.gz` suffix; `gzip` is stdlib, so this works natively on + Windows, macOS, and Linux with no external binary. +- **MATLAB:** `readtable` reads every table after a built-in `gunzip`, with no + external binary on any file. ## Options considered | Format | Python cost | MATLAB cost | Notes | | --- | --- | --- | --- | | **Gzipped TSV** ✅ | none (stdlib/pandas) | none (`readtable`) | Universal, text, types re-specified on read. Chosen. | -| Parquet | `pyarrow` or `fastparquet` (~40–60 MB wheel) as a `raven-python[kegg]` extra | needs ≥ R2019a (`parquetread`, native) | Smaller, faster, typed, columnar. Win mainly at scale / repeated random access. | +| Parquet | `pyarrow` or `fastparquet` (~40–60 MB wheel) as a `raven-toolbox[kegg]` extra | needs ≥ R2019a (`parquetread`, native) | Smaller, faster, typed, columnar. Win mainly at scale / repeated random access. | | SQLite | none (stdlib `sqlite3`) | **needs Database Toolbox** | Rejected: the MATLAB-side toolbox requirement breaks the "same files, both languages, no extra deps" goal. | ## When to revisit @@ -57,7 +59,7 @@ stdlib-only changes cut that to ≈27 MB (2.9×): Reconsider Parquet (or SQLite) if any of these become true: - The `organism_gene_ko` table grows large enough that load *time* (not just - size — the sort+xz change above already addresses on-disk size) becomes a real + size — the sort above already keeps on-disk size in check) becomes a real bottleneck. The remaining inefficiency is that building one species' model still loads all ~9M rows; sorted order makes a `searchsorted`/row-group by-organism read the natural next step before reaching for Parquet. @@ -68,5 +70,5 @@ Reconsider Parquet (or SQLite) if any of these become true: If revisited, prefer **Parquet** over SQLite (no MATLAB toolbox dependency; MATLAB reads Parquet natively from R2019a). It could be offered as an optional -`raven-python[kegg]` extra (pyarrow) alongside the TSV default, rather than replacing +`raven-toolbox[kegg]` extra (pyarrow) alongside the TSV default, rather than replacing it — keeping the dependency-free path intact for users who don't opt in. diff --git a/docs/maintenance/maintaining_binaries.md b/docs/maintenance/maintaining_binaries.md index 85689c08..0b8f044e 100644 --- a/docs/maintenance/maintaining_binaries.md +++ b/docs/maintenance/maintaining_binaries.md @@ -1,7 +1,7 @@ # Maintaining bundled binaries (BLAST+, DIAMOND, …) -Audience: **raven-python maintainers / the GitHub repo owner.** This explains how -raven-python ships external command-line tools, how to update their versions, and how +Audience: **raven-toolbox maintainers / the GitHub repo owner.** This explains how +raven-toolbox ships external command-line tools, how to update their versions, and how to build **minimal-footprint** ZIPs to attach to a GitHub release. > End users never read this. They get a binary automatically via `ensure_binary`, @@ -12,12 +12,12 @@ to build **minimal-footprint** ZIPs to attach to a GitHub release. ## 1. How binary provisioning works -raven-python does **not** vendor binaries in the git repo or on PyPI. Instead: +raven-toolbox does **not** vendor binaries in the git repo or on PyPI. Instead: 1. For each tool we publish **version-pinned ZIPs as GitHub release assets**. -2. A **registry** (`src/raven_python/binaries_registry.json`) maps each *bundle* to its +2. A **registry** (`src/raven_toolbox/binaries_registry.json`) maps each *bundle* to its version, the executables it provides, and per-platform `{asset, sha256}`. -3. At run time `raven_python.binaries.ensure_binary("blastp")` resolves a tool in this +3. At run time `raven_toolbox.binaries.ensure_binary("blastp")` resolves a tool in this order — and only reaches the download as a last resort: ``` @@ -35,7 +35,7 @@ bundle provides both `blastp` and `makeblastdb`), so they are fetched once. --- -## 2. What raven-python actually needs — ship only these +## 2. What raven-toolbox actually needs — ship only these Distribute the **minimum** set of executables. Everything else (other suite tools, docs, examples, changelogs) must be excluded. @@ -113,9 +113,9 @@ Example: bump DIAMOND to a new version for Linux x86-64. Repeat per `(os, arch)` ```bash sha256sum diamond-2.1.11-linux-x86_64.zip # shasum -a 256 on macOS ``` -8. **Attach the ZIP to a raven-python GitHub release** (a release tagged for the binary +8. **Attach the ZIP to a raven-toolbox GitHub release** (a release tagged for the binary set, e.g. `binaries-2024.06`, keeps them independent of code releases). -9. **Update the registry** `src/raven_python/binaries_registry.json` — bump `version` +9. **Update the registry** `src/raven_toolbox/binaries_registry.json` — bump `version` and set the per-platform `asset` + `sha256`: ```json { @@ -125,7 +125,7 @@ Example: bump DIAMOND to a new version for Linux x86-64. Repeat per `(os, arch)` "platforms": { "linux-x86_64": { "asset": "diamond-2.1.11-linux-x86_64.zip", - "url": "https://github.com/SysBioChalmers/raven-python/releases/download/binaries-2024.06/diamond-2.1.11-linux-x86_64.zip", + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/binaries-2024.06/diamond-2.1.11-linux-x86_64.zip", "sha256": "" } } @@ -177,7 +177,7 @@ version, upstream checksum, and the SHA256 you published. ### Native OS support per tool -raven-python invokes each tool through `subprocess.run([resolved_path, …])` — that +raven-toolbox invokes each tool through `subprocess.run([resolved_path, …])` — that call is itself cross-platform, so the real constraint is whether a given tool has a binary that runs natively on each OS. It varies: @@ -197,8 +197,8 @@ Implications: HMM build (3b.3) and HMM query (3b.5) do not**: HMMER and CD-HIT have no Windows binaries, and bioconda has no Windows packages for any of them. Bundling can't fix this — there is no binary to bundle. -- **Windows users should run raven-python inside WSL2** (or a Linux container), where - every tool is native Linux. raven-python does **not** replicate RAVEN's +- **Windows users should run raven-toolbox inside WSL2** (or a Linux container), where + every tool is native Linux. raven-toolbox does **not** replicate RAVEN's `getWSLpath`/`wsl …` path translation: it calls the resolved binary directly, so mixing native-Windows Python with WSL binaries is unsupported — keep the whole stack inside WSL2. @@ -211,12 +211,12 @@ Implications: After building the per-platform ZIPs (named `---.zip`) and uploading them to the release, generate the `_REGISTRY` entry — checksums and -URLs — with [`scripts/make_registry_snippet.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/scripts/README.md): +URLs — with [`scripts/make_registry_snippet.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/scripts/README.md): ```bash python scripts/make_registry_snippet.py binary --bundle blast --version 2.16.0 \ --provides blastp makeblastdb --dir zips \ - --base-url https://github.com/ORG/raven-python/releases/download/blast-2.16.0 + --base-url https://github.com/ORG/raven-toolbox/releases/download/blast-2.16.0 ``` It prints the ready-to-paste `_REGISTRY["blast"]` block; its SHA256 helper is the diff --git a/docs/maintenance/maintaining_kegg_data.md b/docs/maintenance/maintaining_kegg_data.md index 356b574c..8eaa0cdb 100644 --- a/docs/maintenance/maintaining_kegg_data.md +++ b/docs/maintenance/maintaining_kegg_data.md @@ -1,6 +1,6 @@ # Maintaining the KEGG data artefacts -This guide is for the **package maintainer** who rebuilds raven-python's KEGG +This guide is for the **package maintainer** who rebuilds raven-toolbox's KEGG artefacts once per KEGG release. End users never do this — they download the published, version-pinned artefacts. The build has two implemented steps so far: **3b.1 download** (`reconstruction/kegg/download.py`) and **3b.2 parse** @@ -45,7 +45,7 @@ different file with `netrc_path=...` (see below); the format is identical. With `~/.netrc` in place, no credentials need to be passed in code: ```python -from raven_python.reconstruction.kegg import download_kegg_dump +from raven_toolbox.reconstruction.kegg import download_kegg_dump # Reads ~/.netrc, fetches the KEGG archives, extracts and arranges them. download_kegg_dump("keggdb") @@ -74,13 +74,15 @@ KEGG release). ## Step 3b.2 — parse into the published artefacts ```python -from raven_python.reconstruction.kegg import parse_kegg_dump +from raven_toolbox.reconstruction.kegg import parse_kegg_dump -parse_kegg_dump("keggdb", "artefacts") +parse_kegg_dump("keggdb", "artefacts", version="kegg116") ``` This writes the gene-free reference model (`reference_model.yml.gz`, gzipped -RAVEN/cobra YAML) and the relational tables as gzipped TSV. See +RAVEN/cobra YAML) and the relational tables as gzipped TSV. With `version=` set, +every output filename is version-prefixed (e.g. `kegg116_organism_gene_ko.tsv.gz`), +matching the published release assets. See [kegg_data_format.md](kegg_data_format.md) for what those tables contain and the format rationale. @@ -96,9 +98,9 @@ searches. This needs **HMMER** (`hmmbuild`, `hmmpress`), **MAFFT**, and > matrix in [maintaining_binaries.md](maintaining_binaries.md#native-os-support-per-tool). ```python -from raven_python.reconstruction.kegg import build_hmm_library, read_kegg_table +from raven_toolbox.reconstruction.kegg import build_hmm_library, read_kegg_table -organism_gene_ko = read_kegg_table("artefacts/organism_gene_ko.tsv.xz") +organism_gene_ko = read_kegg_table("artefacts/kegg116_organism_gene_ko.tsv.gz") for domain in ("prokaryotes", "eukaryotes"): build_hmm_library( organism_gene_ko, @@ -111,38 +113,53 @@ for domain in ("prokaryotes", "eukaryotes"): For each KO in the domain it gathers the member sequences, dereplicates with CD-HIT (~90 % identity), aligns with MAFFT, trains a profile with `hmmbuild`, and -finally concatenates and `hmmpress`-es them into a single `library.hmm` for fast -`hmmscan` querying. This is the slowest step (hours, once per KEGG release); it -skips KOs whose `.hmm` already exists, so it is resumable. The resulting -libraries are published as version-pinned artefacts alongside the reference model -and tables. +finally concatenates them into a single `library.hmm`. This is the slowest step +(hours, once per KEGG release); it skips KOs whose `.hmm` already exists, so it is +resumable. The concatenated library is published **gzipped** as +`_.hmm.gz` (e.g. `kegg116_prokaryotes.hmm.gz`); end users +decompress it and search it directly with `hmmsearch` (see `ensure_kegg_hmm_library`), +which keeps the download ~10× smaller than a binary index, stays portable across +HMMER versions, and lets the same artefact serve MATLAB RAVEN. ## Building and publishing in one go -[`scripts/build_kegg_artefacts.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/scripts/README.md) runs 3b.2 (+ 3b.3 with -`--hmms`) and lays the output out as publishable assets (`.hmm` named for -`ensure_kegg_hmm_library`): +[`scripts/build_kegg_artefacts.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/scripts/README.md) runs 3b.2 (+ 3b.3 with +`--hmms`) and lays the output out as publishable, version-prefixed assets: the core +model files (reference model + KO/reaction tables) bundled into `_core.tar.gz` +(which `ensure_kegg_data` extracts), and `_.hmm.gz` per domain (named +for `ensure_kegg_hmm_library`). It also publishes +`_taxonomy.gz` — the domain split plus the source for +[`phyl_dist`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/kegg/taxonomy.py), +which regenerates RAVEN's `keggPhylDist` (used by GECKO) with no `.mat` file: ```bash -python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts --hmms --threads 8 +python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts \ + --version kegg116 --hmms --threads 8 ``` -Upload the contents of `artefacts/` to a release, then emit the registry entry for -`raven_python.data._DATA_REGISTRY` with [`scripts/make_registry_snippet.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/scripts/README.md): +Upload the contents of `artefacts/` to the release, then record the artefacts in +both the shared `data/manifest.json` and `raven_toolbox.data._DATA_REGISTRY` with +[`scripts/make_registry_snippet.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/scripts/README.md) +(it computes each file's SHA256 + size): ```bash +# shared source of truth (read by raven-toolbox and MATLAB RAVEN): +python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ + --target data --dataset kegg --version kegg116 --dir artefacts \ + --base-url https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0 +# in-code registry, so end users auto-fetch with no env var (paste into _DATA_REGISTRY): python scripts/make_registry_snippet.py data --dataset kegg --version kegg116 \ - --dir artefacts --base-url https://github.com/ORG/raven-python/releases/download/kegg-data-kegg116 + --dir artefacts --base-url https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0 ``` -Paste the printed block into `_DATA_REGISTRY`; from then on `ensure_data` fetches -and verifies the artefacts for end users automatically. +From then on `ensure_data` fetches and verifies the artefacts for end users +automatically. ## End-user paths (3b.4 / 3b.5) End users do **not** run the steps above — the published artefacts are fetched and -cached automatically by `ensure_data` (`raven_python.data`) under -`~/.cache/raven-python/data/kegg-/` on first use, so the entry points below +cached automatically by `ensure_data` (`raven_toolbox.data`) under +`~/.cache/raven-toolbox/data/kegg-/` on first use, so the entry points below can be called with no local paths at all (pass an explicit `artefact_dir=`/ `library=` to use your own build instead). Two runtime entry points build a draft model from the artefacts: @@ -152,6 +169,7 @@ model from the artefacts: cross-platform. `organism_id="prokaryotes"`/`"eukaryotes"` builds a whole-domain model (pass `taxonomy=`). - **3b.5 — organism not in KEGG** (`get_kegg_model_from_sequences`): `hmmscan`-es a - proteome FASTA against the pressed `library.hmm`, so it needs **HMMER** - (`hmmscan`) — Linux/macOS or WSL2 (see the OS matrix). Tune assignment with + proteome FASTA against the domain library (decompressed and `hmmpress`-ed from + the published `.hmm.gz` on first use), so it needs **HMMER** (`hmmpress`, + `hmmscan`) — Linux/macOS or WSL2 (see the OS matrix). Tune assignment with `cutoff`, `min_score_ratio_ko`, `min_score_ratio_g`. diff --git a/docs/reference/api/analysis.md b/docs/reference/api/analysis.md index 187f874d..2dc1cbae 100644 --- a/docs/reference/api/analysis.md +++ b/docs/reference/api/analysis.md @@ -1,10 +1,10 @@ -# `raven_python.analysis` +# `raven_toolbox.analysis` Analyses not in cobrapy's core: Reporter Metabolites (an around-metabolite gene-score test), FSEOF (Flux Scanning based on Enforced Objective Flux), and random-objective flux sampling. ```{eval-rst} -.. automodule:: raven_python.analysis +.. automodule:: raven_toolbox.analysis :members: :imported-members: ``` diff --git a/docs/reference/api/comparison.md b/docs/reference/api/comparison.md index 36c06160..d4915100 100644 --- a/docs/reference/api/comparison.md +++ b/docs/reference/api/comparison.md @@ -1,11 +1,11 @@ -# `raven_python.comparison` +# `raven_toolbox.comparison` Structural and functional comparison across multiple models — tidy DataFrames of reaction / metabolite / gene / subsystem presence, pairwise Jaccard, and an optional `check_tasks` pass/fail matrix. ```{eval-rst} -.. automodule:: raven_python.comparison +.. automodule:: raven_toolbox.comparison :members: :imported-members: ``` diff --git a/docs/reference/api/gapfilling.md b/docs/reference/api/gapfilling.md index a853b1e8..cd994630 100644 --- a/docs/reference/api/gapfilling.md +++ b/docs/reference/api/gapfilling.md @@ -1,11 +1,11 @@ -# `raven_python.gapfilling` +# `raven_toolbox.gapfilling` Connectivity gap-filling against template models: add the fewest (lowest-penalty) template reactions so reactions blocked in a draft can carry flux. (For fill-until-feasible gap-fill, use {func}`cobra.flux_analysis.gapfill`.) ```{eval-rst} -.. automodule:: raven_python.gapfilling +.. automodule:: raven_toolbox.gapfilling :members: :imported-members: ``` diff --git a/docs/reference/api/index.md b/docs/reference/api/index.md index b3945b09..18019b41 100644 --- a/docs/reference/api/index.md +++ b/docs/reference/api/index.md @@ -4,7 +4,7 @@ Generated from the package's NumPy-style docstrings. There is one page per subpa documents that subpackage's public API (its `__all__`) — the full signature and parameter docs for every public function and class. -The canonical object throughout is {class}`cobra.Model`; raven-python adds the +The canonical object throughout is {class}`cobra.Model`; raven-toolbox adds the RAVEN-specific operations on top of it rather than introducing a parallel model type. ```{toctree} @@ -23,8 +23,3 @@ analysis comparison resolvers ``` - -:::{note} -The `plotting` subpackage is a stub — pathway-map and omics-overlay visualisation -(Phase 6) is not yet implemented. -::: diff --git a/docs/reference/api/init.md b/docs/reference/api/init.md index cd70f8b1..40f2bd87 100644 --- a/docs/reference/api/init.md +++ b/docs/reference/api/init.md @@ -1,10 +1,10 @@ -# `raven_python.init` +# `raven_toolbox.init` Context-specific model extraction (tINIT / ftINIT): the classic INIT MILP, the staged ftINIT MILP, gene→reaction scoring, task-aware gap-filling, and the linear-merge reduction. ```{eval-rst} -.. automodule:: raven_python.init +.. automodule:: raven_toolbox.init :members: :imported-members: ``` diff --git a/docs/reference/api/io.md b/docs/reference/api/io.md index cbe7f7a3..806ab38a 100644 --- a/docs/reference/api/io.md +++ b/docs/reference/api/io.md @@ -1,10 +1,10 @@ -# `raven_python.io` +# `raven_toolbox.io` RAVEN-specific I/O: YAML (cobra + Metabolic Atlas / Human-GEM extensions), SIF, Excel export, and the Standard-GEM `model//…` git layout. ```{eval-rst} -.. automodule:: raven_python.io +.. automodule:: raven_toolbox.io :members: :imported-members: ``` diff --git a/docs/reference/api/localization.md b/docs/reference/api/localization.md index 1eaddcbd..c5b8fd22 100644 --- a/docs/reference/api/localization.md +++ b/docs/reference/api/localization.md @@ -1,12 +1,12 @@ -# `raven_python.localization` +# `raven_toolbox.localization` Sub-cellular localisation — predictor-agnostic and partial-update friendly. -{func}`raven_python.localization.predict_localization` is the MILP entry point; the loaders +{func}`raven_toolbox.localization.predict_localization` is the MILP entry point; the loaders parse predictor outputs (WoLF PSORT, DeepLoc 2) into the `gene × compartment` score frame the algorithm consumes. ```{eval-rst} -.. automodule:: raven_python.localization +.. automodule:: raven_toolbox.localization :members: :imported-members: ``` diff --git a/docs/reference/api/manipulation.md b/docs/reference/api/manipulation.md index f7ba63f8..c63181cb 100644 --- a/docs/reference/api/manipulation.md +++ b/docs/reference/api/manipulation.md @@ -1,11 +1,11 @@ -# `raven_python.manipulation` +# `raven_toolbox.manipulation` Generic {class}`cobra.Model` structural transforms that cobrapy does not cover cleanly: reaction building from equations, batch GPR / bound changes, irreversibility splitting, isozyme expansion, compartment merge / copy, and model merging by name. ```{eval-rst} -.. automodule:: raven_python.manipulation +.. automodule:: raven_toolbox.manipulation :members: :imported-members: ``` @@ -13,6 +13,6 @@ isozyme expansion, compartment merge / copy, and model merging by name. ## Compartment helpers ```{eval-rst} -.. automodule:: raven_python.manipulation.compartments +.. automodule:: raven_toolbox.manipulation.compartments :members: ``` diff --git a/docs/reference/api/omics.md b/docs/reference/api/omics.md index 8ea784fe..d4edbb12 100644 --- a/docs/reference/api/omics.md +++ b/docs/reference/api/omics.md @@ -1,10 +1,10 @@ -# `raven_python.omics` +# `raven_toolbox.omics` Omics integration — Human Protein Atlas (HPA) proteomics + RNA-seq parsing and gene-scoring adapters. The entry point for tissue-specific (f)tINIT runs. ```{eval-rst} -.. automodule:: raven_python.omics +.. automodule:: raven_toolbox.omics :members: :imported-members: ``` diff --git a/docs/reference/api/reconstruction.md b/docs/reference/api/reconstruction.md index 17ac8254..2072d095 100644 --- a/docs/reference/api/reconstruction.md +++ b/docs/reference/api/reconstruction.md @@ -1,4 +1,4 @@ -# `raven_python.reconstruction` +# `raven_toolbox.reconstruction` De novo reconstruction from KEGG and protein homology (BLAST / DIAMOND). @@ -7,7 +7,7 @@ De novo reconstruction from KEGG and protein homology (BLAST / DIAMOND). Homology-based reconstruction from template models (`getModelFromHomology`, BLAST / DIAMOND). ```{eval-rst} -.. automodule:: raven_python.reconstruction.homology +.. automodule:: raven_toolbox.reconstruction.homology :members: :imported-members: ``` @@ -18,7 +18,7 @@ KEGG-based draft reconstruction (`getKEGGModelForOrganism` and friends): downloa parsing → HMM libraries (maintainer build steps), then the runtime model for a KEGG species. ```{eval-rst} -.. automodule:: raven_python.reconstruction.kegg +.. automodule:: raven_toolbox.reconstruction.kegg :members: :imported-members: ``` diff --git a/docs/reference/api/resolvers.md b/docs/reference/api/resolvers.md index 3266fcd9..4552f851 100644 --- a/docs/reference/api/resolvers.md +++ b/docs/reference/api/resolvers.md @@ -1,22 +1,32 @@ # Resource resolvers -Version-pinned, SHA256-verified registries that fetch the external resources raven-python +Version-pinned, SHA256-verified registries that fetch the external resources raven-toolbox depends on but does not vendor. -## `raven_python.binaries` +## `raven_toolbox.binaries` External-tool resolver for BLAST+ / DIAMOND / HMMER (release-ZIP registry). ```{eval-rst} -.. automodule:: raven_python.binaries +.. automodule:: raven_toolbox.binaries :members: ``` -## `raven_python.data` +## `raven_toolbox.data` Data-bundle resolver (KEGG artefacts and template-model data). ```{eval-rst} -.. automodule:: raven_python.data +.. automodule:: raven_toolbox.data + :members: +``` + +## `raven_toolbox.manifest` + +Loads a shared [data/binary manifest](../../maintenance/data_manifest.md) into the two +registries above (and is consulted lazily via `$RAVEN_PYTHON_MANIFEST`). + +```{eval-rst} +.. automodule:: raven_toolbox.manifest :members: ``` diff --git a/docs/reference/api/tasks.md b/docs/reference/api/tasks.md index 168fa9ea..9ea32541 100644 --- a/docs/reference/api/tasks.md +++ b/docs/reference/api/tasks.md @@ -1,11 +1,11 @@ -# `raven_python.tasks` +# `raven_toolbox.tasks` Metabolic task definition, parsing, and checking — the task-list file format, running tasks against a model, and finding the reactions a model must use to satisfy a task list (the input for (f)tINIT's task layer). ```{eval-rst} -.. automodule:: raven_python.tasks +.. automodule:: raven_toolbox.tasks :members: :imported-members: ``` diff --git a/docs/reference/api/utils.md b/docs/reference/api/utils.md index fb688163..ea659c5e 100644 --- a/docs/reference/api/utils.md +++ b/docs/reference/api/utils.md @@ -1,9 +1,9 @@ -# `raven_python.utils` +# `raven_toolbox.utils` Shared helpers — GPR linting, elemental balance, model curation checks, id sorting. ```{eval-rst} -.. automodule:: raven_python.utils +.. automodule:: raven_toolbox.utils :members: :imported-members: ``` @@ -11,6 +11,6 @@ Shared helpers — GPR linting, elemental balance, model curation checks, id sor ## Name / compartment parsing ```{eval-rst} -.. automodule:: raven_python.utils.parse +.. automodule:: raven_toolbox.utils.parse :members: ``` diff --git a/docs/reference/index.md b/docs/reference/index.md index 01b96a6b..1e53dc0c 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -1,12 +1,15 @@ # Reference -Conceptual and API reference for raven-python. +Conceptual and API reference for raven-toolbox. -- **[RAVEN ↔ raven-python migration map](migration.md)** — the function-by-function map - from MATLAB RAVEN to raven-python (and cobrapy where appropriate). Start here if you're +- **[RAVEN ↔ raven-toolbox migration map](migration.md)** — the function-by-function map + from MATLAB RAVEN to raven-toolbox (and cobrapy where appropriate). Start here if you're porting RAVEN code. +- **[YAML model format](yaml_format.md)** — the shared YAML schema produced and consumed + by cobrapy, raven-toolbox, and RAVEN MATLAB, with a fully-annotated example and the + field-order / quoting rules. - **[MATLAB RAVEN back-port proposals](matlab_raven_backports.md)** — improvements - raven-python makes that are candidates to back-port into the MATLAB toolbox. + raven-toolbox makes that are candidates to back-port into the MATLAB toolbox. - **[Improvements over RAVEN](improvements.md)** — the full catalogue of correctness / ergonomics improvements (the `IMPROVEMENTS.md` master list). - **[API reference](api/index.md)** — every public function and class, generated from the @@ -16,6 +19,7 @@ Conceptual and API reference for raven-python. :hidden: migration +yaml_format matlab_raven_backports improvements api/index diff --git a/docs/reference/matlab_raven_backports.md b/docs/reference/matlab_raven_backports.md index 0c96f005..644e3ad8 100644 --- a/docs/reference/matlab_raven_backports.md +++ b/docs/reference/matlab_raven_backports.md @@ -15,9 +15,9 @@ fix is identical in spirit; the patch shape just differs. --- -## `core/getModelFromHomology.m` +## `reconstruction/homology/getModelFromHomology.m` -Implemented in raventoolbox as [`reconstruction.homology.get_model_from_homology`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/reconstruction/homology/homology.py). +Implemented in raventoolbox as [`reconstruction.homology.get_model_from_homology`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/homology/homology.py). * **H1** 🧹 — Split the overloaded `strictness` 1/2/3 into two orthogonal options: `bidirectional` (reciprocal hits) and `bestHitsOnly`. Reciprocal @@ -34,9 +34,9 @@ Implemented in raventoolbox as [`reconstruction.homology.get_model_from_homology preference can flip the chosen template when the database grows. Keep E-value as an option. -## `core/getKEGGModelForOrganism.m` and the KEGG pipeline +## `reconstruction/kegg/getKEGGModelForOrganism.m` and the KEGG pipeline -Implemented across [`reconstruction.kegg`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/reconstruction/kegg/). +Implemented across [`reconstruction.kegg`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/kegg/). * **K1** 🐛 — In the KEGG flat-file parser, read each reaction's equation from its **own `EQUATION` field**, not by matching line *i* of `reaction.lst` @@ -70,11 +70,11 @@ Implemented across [`reconstruction.kegg`](https://github.com/SysBioChalmers/rav switches when DP-cost > 0.65 × (RAM − 2.5 GB) / 4.2e-9). Same protein set, no sequences dropped. * **K14** ⚡ — Sort `organism_gene_ko` by `(organism, gene)` and store it - xz-compressed (`organism_gene_ko.tsv.xz`). Cuts the dominant artefact from - ~78 MB to ~27 MB (2.9×): gene IDs within an organism share long prefixes - so adjacency makes them far more compressible, and xz's larger window - catches cross-row redundancy gzip's 32 KB window misses. MATLAB needs an - external `unxz` to read the file; the small tables remain plain gzipped TSV. + gzipped (`organism_gene_ko.tsv.gz`). Sorting makes gene IDs within an organism + adjacent (shared locus-tag/numeric prefixes), so they compress far better + (~78 → 48 MB) and the order matches the by-organism query. Stored as gzip (not + xz) so MATLAB reads it with built-in `gunzip`, since the same artefact is shared + with raven-toolbox; xz would be ~3× smaller but needs an external `unxz`. * **K15** 🐛 — Recalibrate the HMM-query KO-assignment defaults: change `cutoff` from `1e-50` to `1e-30` and `min_score_ratio_g` from `0.8` to `0.9`. `min_score_ratio_ko` can stay at `0.3` but is empirically inert @@ -87,9 +87,9 @@ Implemented across [`reconstruction.kegg`](https://github.com/SysBioChalmers/rav rose from 0.84 → 0.94 (reaction recall 0.87 → 0.97) with no precision loss. Full numbers in [docs/kegg_hmm_cutoff_calibration.md](../studies/kegg_hmm_cutoff_calibration.md). -## `core/FSEOF.m` +## `analysis/FSEOF.m` -Implemented in raventoolbox as [`analysis.fseof`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/analysis/fseof.py). +Implemented in raventoolbox as [`analysis.fseof`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/analysis/fseof.py). * **FS1** 🐛 — Replace the strict step-by-step monotonicity gate (a target is discarded if any single step's flux fails to exceed the previous) with a @@ -121,9 +121,9 @@ Implemented in raventoolbox as [`analysis.fseof`](https://github.com/SysBioChalm matches the selection criterion. The amplify/knockdown/knockout label (after FS2) should use the slope of `|flux|`, not endpoint comparison. -## `core/randomSampling.m` +## `analysis/randomSampling.m` -Implemented in raventoolbox as [`analysis.random_sampling`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/analysis/sampling.py). +Implemented in raventoolbox as [`analysis.random_sampling`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/analysis/sampling.py). * **SAMP1** ⚡ — Compute `goodRxns` (loop-free, flux-carrying objective candidates) via a **single FVA pass**, not the current per-reaction `parfor` @@ -140,9 +140,9 @@ Implemented in raventoolbox as [`analysis.random_sampling`](https://github.com/S "unbounded objective". Scope `replaceBoundsWithInf` to sampling only, after the goodRxns set is found. -## `core/reporterMetabolites.m` +## `analysis/reporterMetabolites.m` -Implemented in raventoolbox as [`analysis.reporter_metabolites`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/analysis/reporter.py). +Implemented in raventoolbox as [`analysis.reporter_metabolites`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/analysis/reporter.py). * **RM1** ⚡🐛 — Replace the per-neighbour-count Monte Carlo background correction (currently 100 000 random sets drawn *for each distinct @@ -152,9 +152,9 @@ Implemented in raventoolbox as [`analysis.reporter_metabolites`](https://github. metabolite Z is just `(metZ − √n·μ) / σ`. This removes both the slow sampling step and its run-to-run randomness (results become deterministic). -## `core/runINIT.m` +## `INIT/runINIT.m` -Implemented in raventoolbox as [`init.run_init`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/init.py). +Implemented in raventoolbox as [`init.run_init`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/init.py). * **I4** 🐛 — Drop the hard-coded big-M (1000) in the MILP and use **each reaction's own upper bound** instead: `v ≤ ub · x`. Expose `eps` and @@ -162,9 +162,9 @@ Implemented in raventoolbox as [`init.run_init`](https://github.com/SysBioChalme fits ±1000-bounded models with O(1) scores — flagged in RAVEN as scale-dependent and tunable, but no API knobs are exposed. -## `core/ftINIT.m` and the ftINIT pipeline +## `INIT/ftINIT.m` and the ftINIT pipeline -Implemented across [`init.ftinit`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/ftinit.py), [`init.taskfill`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/taskfill.py), [`init.merge`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/merge.py), [`init.prep`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/prep.py). +Implemented across [`init.ftinit`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/ftinit.py), [`init.taskfill`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/taskfill.py), [`init.merge`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/merge.py), [`init.prep`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/prep.py). * **FT3** 🐛 — Same big-M issue as `runINIT` — use each reaction's own bound as big-M instead of the fixed 100/1000. Expose `force_on` and `force_on_ess` @@ -184,24 +184,24 @@ Implemented across [`init.ftinit`](https://github.com/SysBioChalmers/raven-pytho to test. Make the all-negative tie-break **deterministic** (sort + take the least-negative) instead of random. -## `core/addRxns.m` +## `manipulation/addRxns.m` -Implemented in raventoolbox as [`manipulation.add_reactions_from_equations`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/add.py). +Implemented in raventoolbox as [`manipulation.add_reactions_from_equations`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/add.py). * **A1** 🧹 — Accept a string keyword (`metsBy = 'id'` / `'name'`) instead of the opaque `eqnType = 1 / 2 / 3` integer. Call-sites become self-documenting. -## `core/checkModelStruct.m` (or its curation subset) +## `queries/checkModelStruct.m` (or its curation subset) -Implemented in raventoolbox as [`utils.check_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/validate.py). +Implemented in raventoolbox as [`utils.check_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/validate.py). * **V2** 🧹 — Return a struct array of issues (one per finding, with `category` / `target` / `message` fields) instead of printing warnings or throwing on the first problem. Programmatically filterable. -## `core/getElementalBalance.m` +## `queries/getElementalBalance.m` -Implemented in raventoolbox as [`utils.get_elemental_balance`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/balance.py). +Implemented in raventoolbox as [`utils.get_elemental_balance`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/balance.py). * **B2** 🐛 — A reaction with no metabolites (an empty `S(:,j)`) currently falls through to `balanceStatus = 1` ("balanced") because both pre-loops @@ -217,9 +217,9 @@ Implemented in raventoolbox as [`utils.get_elemental_balance`](https://github.co Empty reactions then report `-1` ("missing information") rather than vacuously balanced. -## `core/findPotentialErrors.m` (GPR linting) +## `manipulation/findPotentialErrors.m` (GPR linting) -Implemented in raventoolbox as [`utils.find_non_dnf_grrules`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/gpr.py). +Implemented in raventoolbox as [`utils.find_non_dnf_grrules`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/gpr.py). * **S1** 🧹 — Return the `indexes2check` array (which the function already computes) plus per-reaction reason strings as a struct array, instead of @@ -229,7 +229,7 @@ Implemented in raventoolbox as [`utils.find_non_dnf_grrules`](https://github.com to spacing and bracketing and to gene IDs containing those characters; the tree walk (no OR beneath any AND) is exact. -## `core/getIndexes.m` +## `queries/getIndexes.m` * **G1** ⚡ — Build a `containers.Map` `{id: position}` once per call and use hashed lookup (O(1)) instead of looping `find(strcmp(obj(i), searchIn))` diff --git a/docs/reference/migration.md b/docs/reference/migration.md index 185b4b88..63621397 100644 --- a/docs/reference/migration.md +++ b/docs/reference/migration.md @@ -1,15 +1,15 @@ -# RAVEN → raven-python migration reference +# RAVEN → raven-toolbox migration reference -A function-by-function map from the MATLAB RAVEN Toolbox to raven-python (and cobrapy where +A function-by-function map from the MATLAB RAVEN Toolbox to raven-toolbox (and cobrapy where appropriate). For each RAVEN function we record one of four outcomes: -* ✅ **ported** — there's a direct raven-python replacement; see the link. +* ✅ **ported** — there's a direct raven-toolbox replacement; see the link. * 🗒️ **cheatsheet** — cobrapy already covers it; a one-liner or short idiom does the job (recorded under each row). * ⛔ **not ported** — explicit decision not to bring it across, with rationale. -* 🆕 **new in raven-python** — functionality raven-python adds that has no RAVEN counterpart. +* 🆕 **new in raven-toolbox** — functionality raven-toolbox adds that has no RAVEN counterpart. -For raven-python's deliberate improvements over RAVEN (and which of them are candidates to +For raven-toolbox's deliberate improvements over RAVEN (and which of them are candidates to upstream into MATLAB RAVEN), see [IMPROVEMENTS.md](improvements.md). ## Design principle @@ -17,88 +17,88 @@ upstream into MATLAB RAVEN), see [IMPROVEMENTS.md](improvements.md). The in-memory object is always a [`cobra.Model`](https://cobrapy.readthedocs.io). There is no parallel RAVEN struct, no `ravenCobraWrapper`-style adapter. RAVEN fields that cobra doesn't model natively (`rxnMiriams`, `metDeltaG`, `rxnConfidenceScores`, …) live in -cobra's `annotation` / `notes` dictionaries. raven-python's YAML I/O follows the cobra YAML +cobra's `annotation` / `notes` dictionaries. raven-toolbox's YAML I/O follows the cobra YAML standard plus the geckopy enzyme-constrained extension, so ecModels round-trip. --- ## Foundation: utilities, manipulation, I/O -| RAVEN | raven-python | Notes | +| RAVEN | raven-toolbox | Notes | |---|---|---| -| `addRxns` (equations) | ✅ [`manipulation.add_reactions_from_equations`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/add.py) | The keystone: equation-string → reactions, matching mets by id, name, or `name[comp]`; strict / auto policies for new mets and genes. | -| `addTransport` | ✅ [`manipulation.add_transport_reactions`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/transport.py) | Transport across compartments, matching mets by name, sequential `tr_NNNN` ids. cobra has no transport primitive. | -| `addRxnsGenesMets` | ✅ [`manipulation.add_reactions_from_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/transfer.py) | Copy reactions from a source model, matching mets by `name[comp]` (vs cobra's strict-by-id merge). | -| `mergeModels` | ✅ [`manipulation.merge_models`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/merge.py) | N-model merge, unify mets by `name[comp]` (or id), keep all reactions (id collisions renamed). cobra's `merge` is pairwise / strict-by-id. | -| `changeRxns`, `changeGrRules` | ✅ [`manipulation.change_reaction_equations`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/change.py), `change_gene_reaction_rules` | Stoichiometry change in place; batch GPR set/append (`(old) or (new)`). | -| `setParam('var', …)` | ✅ [`manipulation.set_variance_bounds`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/parameters.py) | ±% band around measured values. **Other modes (lb/ub/eq/obj/unc)**: 🗒️ cobra one-liners (`reaction.bounds`, `model.objective`, `Configuration().bounds`). | +| `addRxns` (equations) | ✅ [`manipulation.add_reactions_from_equations`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/add.py) | The keystone: equation-string → reactions, matching mets by id, name, or `name[comp]`; strict / auto policies for new mets and genes. | +| `addTransport` | ✅ [`manipulation.add_transport_reactions`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/transport.py) | Transport across compartments, matching mets by name, sequential `tr_NNNN` ids. cobra has no transport primitive. | +| `addRxnsGenesMets` | ✅ [`manipulation.add_reactions_from_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/transfer.py) | Copy reactions from a source model, matching mets by `name[comp]` (vs cobra's strict-by-id merge). | +| `mergeModels` | ✅ [`manipulation.merge_models`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/merge.py) | N-model merge, unify mets by `name[comp]` (or id), keep all reactions (id collisions renamed). cobra's `merge` is pairwise / strict-by-id. | +| `changeRxns`, `changeGrRules` | ✅ [`manipulation.change_reaction_equations`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/change.py), `change_gene_reaction_rules` | Stoichiometry change in place; batch GPR set/append (`(old) or (new)`). | +| `setParam('var', …)` | ✅ [`manipulation.set_variance_bounds`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/parameters.py) | ±% band around measured values. **Other modes (lb/ub/eq/obj/unc)**: 🗒️ cobra one-liners (`reaction.bounds`, `model.objective`, `Configuration().bounds`). | | `setExchangeBounds` | ⛔ not ported | cobra's `model.medium = {ex_id: uptake}` covers it. | -| `simplifyModel` (gap modes) | ✅ [`manipulation.remove_dead_end_reactions`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/simplify.py), `remove_duplicate_reactions`, `constrain_reversible_reactions`, `group_linear_reactions` | Cobra-covered modes (no-flux→`find_blocked_reactions`, zero-interval, unconstrained) are 🗒️ cheatsheeted. `group_linear` is lossy (drops genes), per RAVEN. | -| `removeMets`, `removeGenes` | ✅ [`manipulation.remove_metabolites`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/remove.py), `remove_genes` | Delegate to cobra; add `by_name` cross-compartment deletion (mets) and a `blocked_reactions` policy (genes). `removeReactions` itself ⛔ — cobra's `remove_reactions` covers it. | -| `convertToIrrev` | ✅ [`manipulation.convert_to_irreversible`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/irreversible.py) | Splits reversible non-exchange reactions into forward + `_REV`. Adopted from geckopy. | -| `expandModel` | ✅ [`manipulation.expand_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/expand.py) | Splits OR-GPR (isozyme) reactions into one per AND-clause. Adopted from geckopy. | -| `mergeCompartments` | ✅ [`manipulation.merge_compartments`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/compartments.py) | Collapse a multi-compartment model into one; deduplicate identical reactions; optionally drop one-met collapses (`drop_single_metabolite_reactions`). | -| `copyToComps` | ✅ [`manipulation.copy_to_compartment`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/compartments.py) | Duplicate reactions into a target compartment (idempotent; `delete_original=True` makes it a move). | +| `simplifyModel` (gap modes) | ✅ [`manipulation.remove_dead_end_reactions`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/simplify.py), `remove_duplicate_reactions`, `constrain_reversible_reactions`, `group_linear_reactions` | Cobra-covered modes (no-flux→`find_blocked_reactions`, zero-interval, unconstrained) are 🗒️ cheatsheeted. `group_linear` is lossy (drops genes), per RAVEN. | +| `removeMets`, `removeGenes` | ✅ [`manipulation.remove_metabolites`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/remove.py), `remove_genes` | Delegate to cobra; add `by_name` cross-compartment deletion (mets) and a `blocked_reactions` policy (genes). `removeReactions` itself ⛔ — cobra's `remove_reactions` covers it. | +| `convertToIrrev` | ✅ [`manipulation.convert_to_irreversible`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/irreversible.py) | Splits reversible non-exchange reactions into forward + `_REV`. Adopted from geckopy. | +| `expandModel` | ✅ [`manipulation.expand_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/expand.py) | Splits OR-GPR (isozyme) reactions into one per AND-clause. Adopted from geckopy. | +| `mergeCompartments` | ✅ [`manipulation.merge_compartments`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/compartments.py) | Collapse a multi-compartment model into one; deduplicate identical reactions; optionally drop one-met collapses (`drop_single_metabolite_reactions`). | +| `copyToComps` | ✅ [`manipulation.copy_to_compartment`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/compartments.py) | Duplicate reactions into a target compartment (idempotent; `delete_original=True` makes it a move). | | `mapCompartments` | ⛔ not ported | Overlaps with `comparison.compare_models` on the reaction-id intersection. | -| `getElementalBalance` | ✅ [`utils.get_elemental_balance`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/balance.py) | Graded `balanced` / `unbalanced` / `unknown` — `unknown` catches a missing formula that cobra's `check_mass_balance` silently miscounts. | -| `checkModelStruct` (curation subset) | ✅ [`utils.check_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/validate.py) | Structured curation report. RAVEN's struct/type checks are moot in cobra. | -| `is_dnf` / GPR check (from `standardizeGrRules`) | ✅ [`utils.is_dnf`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/gpr.py), `find_non_dnf_grrules` | Lint-only half; cobra auto-normalises GPRs on assignment, so the rewriting half isn't ported. | -| `getIndexes` (`metcomps` sliver) | ✅ [`utils.parse_name_comp`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/parse.py) | The only `getIndexes` bit cobra doesn't already cover. | +| `getElementalBalance` | ✅ [`utils.get_elemental_balance`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/balance.py) | Graded `balanced` / `unbalanced` / `unknown` — `unknown` catches a missing formula that cobra's `check_mass_balance` silently miscounts. | +| `checkModelStruct` (curation subset) | ✅ [`utils.check_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/validate.py) | Structured curation report. RAVEN's struct/type checks are moot in cobra. | +| `is_dnf` / GPR check (from `standardizeGrRules`) | ✅ [`utils.is_dnf`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/gpr.py), `find_non_dnf_grrules` | Lint-only half; cobra auto-normalises GPRs on assignment, so the rewriting half isn't ported. | +| `getIndexes` (`metcomps` sliver) | ✅ [`utils.parse_name_comp`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/parse.py) | The only `getIndexes` bit cobra doesn't already cover. | | `editMiriam`, `extractMiriam` | ⛔ not ported | cobra's `.annotation` is already a `{namespace: id(s)}` dict — read/write it directly. | | `getRxnsInComp`, `getMetsInComp` | ⛔ not ported | One-liners over cobra's `reaction.compartments` / `metabolite.compartment`. | | `constructEquations` | ⛔ not ported | `reaction.build_reaction_string(use_metabolite_names=...)` already does both id and name equations. | -| `sortIdentifiers` | ✅ [`utils.sort_identifiers`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/utils/sort.py) | Model-wide alphabetical sort; also via `sort_ids=` on `write_yaml_model`. | +| `sortIdentifiers` | ✅ [`utils.sort_identifiers`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/utils/sort.py) | Model-wide alphabetical sort; also via `sort_ids=` on `write_yaml_model`. | ### I/O -| RAVEN | raven-python | Notes | +| RAVEN | raven-toolbox | Notes | |---|---|---| -| `readYAMLmodel`, `writeYAMLmodel` | ✅ [`io.read_yaml_model`, `write_yaml_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/io/yaml.py) | Aligned to cobra's `!!omap` writer (RAVEN `fa281a1`). Adds the RAVEN-only top-level per-entry keys (inchis/deltaG/metFrom/notes, confidence_score/references/rxnFrom/deltaG, protein) into `.notes`, plus `version`/`metaData`/GECKO `ec-*`. cobra-readable output verified. | -| `exportModelToSIF` | ✅ [`io.export_model_to_sif`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/io/sif.py) | Cytoscape SIF (`rc`/`rr`/`cc` graphs). | -| `exportToExcelFormat` (export only) | ✅ [`io.export_to_excel`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/io/excel.py) | RAVEN 5-sheet xlsx (RXNS / METS / COMPS / GENES / MODEL). Excel **import** is intentionally excluded. | -| `exportForGit` | ✅ [`io.export_for_git`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/io/git.py) | Standard-GEM repo layout (`model//…`). | +| `readYAMLmodel`, `writeYAMLmodel` | ✅ [`io.read_yaml_model`, `write_yaml_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/io/yaml.py) | Aligned to cobra's `!!omap` writer (RAVEN `fa281a1`). Adds the RAVEN-only top-level per-entry keys (inchis/deltaG/metFrom/notes, confidence_score/references/rxnFrom/deltaG, protein) into `.notes`, plus `version`/`metaData`/GECKO `ec-*`. cobra-readable output verified. | +| `exportModelToSIF` | ✅ [`io.export_model_to_sif`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/io/sif.py) | Cytoscape SIF (`rc`/`rr`/`cc` graphs). | +| `exportToExcelFormat` (export only) | ✅ [`io.export_to_excel`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/io/excel.py) | RAVEN 5-sheet xlsx (RXNS / METS / COMPS / GENES / MODEL). Excel **import** is intentionally excluded. | +| `exportForGit` | ✅ [`io.export_for_git`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/io/git.py) | Standard-GEM repo layout (`model//…`). | | `importYAML/SBML/Mat/Excel` | 🗒️ cobra's standard readers | `cobra.io.read_sbml_model` / `load_json_model` / etc.; Excel import not ported. | ## Reconstruction -| RAVEN | raven-python | Notes | +| RAVEN | raven-toolbox | Notes | |---|---|---| -| `getModelFromHomology` + `getBlast`/`getDiamond` | ✅ [`reconstruction.homology.get_model_from_homology`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/reconstruction/homology/homology.py), `run_blast`, `run_diamond`, `blast_from_table` | Core homology reconstruction with structured improvements (bidirectional / best-hits-only, AST GPR rewrite, complex policy, bitscore best-hits, DataFrame ortholog map). | -| KEGG download → species model (5 steps) | ✅ [`reconstruction.kegg.*`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/reconstruction/kegg/) | All five steps: `download.fetch_keggdb`, `parse.read_kegg_table` + reference model, `hmm.build_libraries`, `organism.build_kegg_model_for_organism` (no-FASTA), `query.assign_kos` + `run_hmmscan`. | -| `getPhylDist` | ⛔ not ported | Fixed prok90/euk90 libraries make the distance matrix moot. | +| `getModelFromHomology` + `getBlast`/`getDiamond` | ✅ [`reconstruction.homology.get_model_from_homology`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/homology/homology.py), `run_blast`, `run_diamond`, `blast_from_table` | Core homology reconstruction with structured improvements (bidirectional / best-hits-only, AST GPR rewrite, complex policy, bitscore best-hits, DataFrame ortholog map). | +| KEGG download → species model (5 steps) | ✅ [`reconstruction.kegg.*`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/kegg/) | All five steps: `download.fetch_keggdb`, `parse.read_kegg_table` + reference model, `hmm.build_libraries`, `organism.build_kegg_model_for_organism` (no-FASTA), `query.assign_kos` + `run_hmmscan`. | +| `getPhylDist` | ✅ [`reconstruction.kegg.phyl_dist`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/kegg/taxonomy.py) (+ `parse_taxonomy`) | Lineage parsing **and** the distance matrix (RAVEN's `keggPhylDist`), regenerated from the published `taxonomy` artefact so GECKO's organism-distance kcat selection needs no `.mat`. The per-organism HMM-subsampling use is moot here (fixed prok90/euk90 libraries). | | `getMetaCycModelForOrganism` | ⛔ not ported (and **flagged for removal from MATLAB RAVEN**) | BLAST-to-single-representatives is low-precision at every cutoff. See [IMPROVEMENTS.md](improvements.md) under `R-MetaCyc`. | ## Tasks, gap-filling, INIT, ftINIT -| RAVEN | raven-python | Notes | +| RAVEN | raven-toolbox | Notes | |---|---|---| -| `parseTaskList`, `checkTasks` | ✅ [`tasks.parse_task_list`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/tasks/tasklist.py), `check_tasks` ([tasks/check.py](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/tasks/check.py)) | `check_tasks` reuses one model across the task list (no per-task model copy) — at genome scale ~12× faster than the copy-per-task implementation it replaced. | -| `fillGaps` (connectivity mode) | ✅ [`gapfilling.connect_blocked_reactions`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/gapfilling/fill.py) | MILP (min penalty-weighted template reactions s.t. blocked reactions carry flux). Targeted mode → 🗒️ `cobra.gapfill`. | -| `runINIT`, `scoreComplexModel`, `getINITModel` | ✅ [`init.run_init`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/init.py), [`init.score_reactions_from_genes`, `gene_scores_from_expression`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/score.py), [`init.get_init_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/build.py) | INIT MILP rewritten in optlang (no sparse-matrix construction); RNA-seq scoring is `5·ln(level/ref)`-clamped. | -| `ftINIT`, `prepINITModel`, `ftINITInternalAlg`, `getINITSteps` | ✅ [`init.ftinit`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/ftinit.py), [`init.prep_init_model`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/prep.py), [`init.run_ftinit`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/ftinit.py), [`init.get_init_steps`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/steps.py) | Staged MILP + linear merge + scaling (`rescaleModelForINIT`). Validated against RAVEN on Human-GEM (Jaccard 0.975–0.980; see [humangem_validation.md](../studies/humangem_validation.md)). Metabolomics-based scoring is the one piece not yet implemented (raises `NotImplementedError`). | -| `ftINITFillGaps`, `ftINITFillGapsMILP`, `ftINITFillGapsForAllTasks` | ✅ [`init.fill_tasks`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/taskfill.py) | Task-aware gap-filling within ftINIT; in-place `_feasible` check + bounded fill MILP (`mip_gap`, `time_limit`). | -| `mergeLinear` | ✅ [`init.merge_linear`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/merge.py) | Linear merge of unit-stoichiometry chains; bookkeeping (`group_ids`, `reversed_rxns`) to map back to the reference model. | -| `removeLowScoreGenes` | ✅ [`init.remove_low_score_genes`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/genes.py) | Final gene-prune step of ftINIT. | +| `parseTaskList`, `checkTasks` | ✅ [`tasks.parse_task_list`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/tasks/tasklist.py), `check_tasks` ([tasks/check.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/tasks/check.py)) | `check_tasks` reuses one model across the task list (no per-task model copy) — at genome scale ~12× faster than the copy-per-task implementation it replaced. | +| `fillGaps` (connectivity mode) | ✅ [`gapfilling.connect_blocked_reactions`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/gapfilling/fill.py) | MILP (min penalty-weighted template reactions s.t. blocked reactions carry flux). Targeted mode → 🗒️ `cobra.gapfill`. | +| `runINIT`, `scoreComplexModel`, `getINITModel` | ✅ [`init.run_init`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/init.py), [`init.score_reactions_from_genes`, `gene_scores_from_expression`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/score.py), [`init.get_init_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/build.py) | INIT MILP rewritten in optlang (no sparse-matrix construction); RNA-seq scoring is `5·ln(level/ref)`-clamped. | +| `ftINIT`, `prepINITModel`, `ftINITInternalAlg`, `getINITSteps` | ✅ [`init.ftinit`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/ftinit.py), [`init.prep_init_model`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/prep.py), [`init.run_ftinit`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/ftinit.py), [`init.get_init_steps`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/steps.py) | Staged MILP + linear merge + scaling (`rescaleModelForINIT`). Validated against RAVEN on Human-GEM (Jaccard 0.975–0.980; see [humangem_validation.md](../studies/humangem_validation.md)). Metabolomics-based scoring is the one piece not yet implemented (raises `NotImplementedError`). | +| `ftINITFillGaps`, `ftINITFillGapsMILP`, `ftINITFillGapsForAllTasks` | ✅ [`init.fill_tasks`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/taskfill.py) | Task-aware gap-filling within ftINIT; in-place `_feasible` check + bounded fill MILP (`mip_gap`, `time_limit`). | +| `mergeLinear` | ✅ [`init.merge_linear`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/merge.py) | Linear merge of unit-stoichiometry chains; bookkeeping (`group_ids`, `reversed_rxns`) to map back to the reference model. | +| `removeLowScoreGenes` | ✅ [`init.remove_low_score_genes`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/genes.py) | Final gene-prune step of ftINIT. | | `fitTasks` | ⛔ not ported | Niche; tasks are normally consumed via `checkTasks` / ftINIT's task layer. | ## Omics, analysis, comparison -| RAVEN | raven-python | Notes | +| RAVEN | raven-toolbox | Notes | |---|---|---| -| `parseHPA`, `parseHPArna`, `scoreModel` | ✅ [`omics.parse_hpa`, `parse_hpa_rna`, `hpa_gene_scores`, `rna_gene_scores`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/omics/hpa.py) | Pandas-tidy DataFrames; scoring adapters reuse `score_reactions_from_genes` (single source of truth for the GPR walk). | -| `reporterMetabolites` | ✅ [`analysis.reporter_metabolites`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/analysis/reporter.py) | Exact closed-form background replaces RAVEN's Monte-Carlo (RM1 in IMPROVEMENTS). | -| `FSEOF` | ✅ [`analysis.fseof`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/analysis/fseof.py) | Regression slope + correlation, amplify/knockdown/knockout classes, gene aggregation (FS1–FS4 in IMPROVEMENTS). | -| `randomSampling` | ✅ [`analysis.sample`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/analysis/sampling.py) | Wraps cobra's flux sampling. | -| `compareMultipleModels` | ✅ [`comparison.compare_models`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/comparison/compare.py) | Tidy DataFrames (reactions / mets / genes / subsystems presence + pairwise Jaccard + optional `check_tasks` pass/fail). Plotting and tSNE/MDS are 🗒️ one-liners in seaborn/scikit-learn; intentionally not in the function. | +| `parseHPA`, `parseHPArna`, `scoreModel` | ✅ [`omics.parse_hpa`, `parse_hpa_rna`, `hpa_gene_scores`, `rna_gene_scores`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/omics/hpa.py) | Pandas-tidy DataFrames; scoring adapters reuse `score_reactions_from_genes` (single source of truth for the GPR walk). | +| `reporterMetabolites` | ✅ [`analysis.reporter_metabolites`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/analysis/reporter.py) | Exact closed-form background replaces RAVEN's Monte-Carlo (RM1 in IMPROVEMENTS). | +| `FSEOF` | ✅ [`analysis.fseof`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/analysis/fseof.py) | Regression slope + correlation, amplify/knockdown/knockout classes, gene aggregation (FS1–FS4 in IMPROVEMENTS). | +| `randomSampling` | ✅ [`analysis.sample`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/analysis/sampling.py) | Wraps cobra's flux sampling. | +| `compareMultipleModels` | ✅ [`comparison.compare_models`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/comparison/compare.py) | Tidy DataFrames (reactions / mets / genes / subsystems presence + pairwise Jaccard + optional `check_tasks` pass/fail). Plotting and tSNE/MDS are 🗒️ one-liners in seaborn/scikit-learn; intentionally not in the function. | | `runDynamicFBA` | ⛔ not ported | Established Python implementations exist: [`dfba`](https://pypi.org/project/dfba/) (Pinheiro et al.; CVODES-backed), [`reframed`](https://pypi.org/project/reframed/) (Machado lab), [`mewpy`](https://pypi.org/project/mewpy/) (Cunha lab). Cobrapy itself has none, but re-porting would duplicate maintained prior art. | ## Localisation (Phase 7) -| RAVEN | raven-python | Notes | +| RAVEN | raven-toolbox | Notes | |---|---|---| -| `predictLocalization` | ✅ [`localization.predict_localization`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/predict.py) | Deterministic MILP (not simulated annealing). Caller-passed `reactions_to_relocate` set (everything else pinned). Multi-compartment by default: primary "free", extras pay `multi_compartment_penalty`. Tolerates incomplete models (no silent reaction removal). `apply=False` returns a `LocalizationProposal` diff. Real-data validation against curated yeast-GEM in [yeast_localization_benchmark.md](../studies/yeast_localization_benchmark.md). | -| `getWoLFScores`, `parseScores('wolf')` | ✅ [`localization.load_wolfpsort`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/scores.py) | Parses WoLF PSORT summary output (RAVEN-compatible); row-normalised. Does not shell out to the WoLF PSORT binary — run that separately and feed in the output. | -| `parseScores('deeploc')` | ✅ [`localization.load_deeploc`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/scores.py) | DeepLoc 2 per-protein CSV (Protein_ID / Localizations / Signals + one column per compartment). | +| `predictLocalization` | ✅ [`localization.predict_localization`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/predict.py) | Deterministic MILP (not simulated annealing). Caller-passed `reactions_to_relocate` set (everything else pinned). Multi-compartment by default: primary "free", extras pay `multi_compartment_penalty`. Tolerates incomplete models (no silent reaction removal). `apply=False` returns a `LocalizationProposal` diff. Real-data validation against curated yeast-GEM in [yeast_localization_benchmark.md](../studies/yeast_localization_benchmark.md). | +| `getWoLFScores`, `parseScores('wolf')` | ✅ [`localization.load_wolfpsort`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/scores.py) | Parses WoLF PSORT summary output (RAVEN-compatible); row-normalised. Does not shell out to the WoLF PSORT binary — run that separately and feed in the output. | +| `parseScores('deeploc')` | ✅ [`localization.load_deeploc`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/scores.py) | DeepLoc 2 per-protein CSV (Protein_ID / Localizations / Signals + one column per compartment). | ## Things deliberately not ported @@ -106,15 +106,15 @@ standard plus the geckopy enzyme-constrained extension, so ecModels round-trip. * **`checkModelStruct` struct/type checks** — moot in cobra. * **`runDynamicFBA`** — see Omics/analysis row. * **`getMetaCycModelForOrganism`** — see Reconstruction row; flagged for upstream removal. -* **`getPhylDist`** — fixed prok90/euk90 libraries make it moot. +* **`getPhylDist` per-organism HMM subsampling** — fixed prok90/euk90 libraries make it moot (the distance matrix itself **is** ported, as `reconstruction.kegg.phyl_dist`, for GECKO). * **`mapCompartments`** — overlaps with `compare_models`. * **`editMiriam`, `extractMiriam`, `getRxnsInComp`, `getMetsInComp`, `constructEquations`, `getIndexes`** (most), **`setExchangeBounds`**, **most `setParam` modes**, **`getBlastFromExcel` Excel branch** — cobra one-liners; recorded above. -## "New in raven-python" entry points +## "New in raven-toolbox" entry points These are not direct RAVEN ports: -* [`comparison.compare_models`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/comparison/compare.py) — already in RAVEN, but the raven-python version returns tidy DataFrames suitable for downstream analysis (RAVEN's version embeds heatmap/tSNE plotting). -* [`reconstruction.homology.run_blast` / `run_diamond` / `blast_from_table`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/reconstruction/homology/blast.py) — generic subprocess wrappers; a DataFrame is the canonical "hits" object (no `blastStructure` struct). -* [`binaries.resolve_binary`, `ensure_binary`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/binaries.py) — version-pinned release-ZIP registry for external tools (BLAST/DIAMOND/HMMER), SHA256-verified. -* [`localization.LocalizationProposal`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/predict.py) — the diff-preview mode (`apply=False`). +* [`comparison.compare_models`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/comparison/compare.py) — already in RAVEN, but the raven-toolbox version returns tidy DataFrames suitable for downstream analysis (RAVEN's version embeds heatmap/tSNE plotting). +* [`reconstruction.homology.run_blast` / `run_diamond` / `blast_from_table`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/reconstruction/homology/blast.py) — generic subprocess wrappers; a DataFrame is the canonical "hits" object (no `blastStructure` struct). +* [`binaries.resolve_binary`, `ensure_binary`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/binaries.py) — version-pinned release-ZIP registry for external tools (BLAST/DIAMOND/HMMER), SHA256-verified. +* [`localization.LocalizationProposal`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/predict.py) — the diff-preview mode (`apply=False`). diff --git a/docs/reference/todo.md b/docs/reference/todo.md index 8c24129a..2aa2120d 100644 --- a/docs/reference/todo.md +++ b/docs/reference/todo.md @@ -2,24 +2,13 @@ What's still on the books. See [raven_migration.md](migration.md) for the function-by- function port status (most of it is done); see [IMPROVEMENTS.md](improvements.md) for the -catalogue of raven-python improvements that should also be back-ported into MATLAB RAVEN. +catalogue of raven-toolbox improvements that should also be back-ported into MATLAB RAVEN. ## Major -### Visualization (`plotting/`) - -Not started. RAVEN has limited plotting (`drawMap` etc., MATLAB-bound). For raven-python the most -useful targets are: - -* Pathway maps / Escher integration for context-specific models. -* Omics overlay (gene-score / expression heatmaps on the reaction set). -* Flux distribution overlays. - -cobrapy + Escher already covers a lot here — likely a thin integration layer rather than a port. - ### Metabolomics-based scoring in ftINIT -The metabolomics-detected metabolite production-reward block in [`init.ftinit`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/init/ftinit.py) +The metabolomics-detected metabolite production-reward block in [`init.ftinit`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/init/ftinit.py) currently raises `NotImplementedError` if a non-empty `metabolomics` argument is passed. The linear merge eliminates degree-2 detected metabolites, so it needs producer-group-mapping + negative-producer force-flux constraints — the most intricate MILP piece, for the least-used @@ -28,7 +17,7 @@ input. Worth doing only when a real user request lands. ## Infrastructure * **Binary ZIP releases** for BLAST/DIAMOND (Phase 3a). The runtime resolver in - [`binaries.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/binaries.py) is ready; the registry is empty until ZIPs are + [`binaries.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/binaries.py) is ready; the registry is empty until ZIPs are published as GitHub release assets. * **KEGG data artefact releases.** See [maintaining_kegg_data.md](../maintenance/maintaining_kegg_data.md). @@ -39,7 +28,7 @@ input. Worth doing only when a real user request lands. * [IMPROVEMENTS.md](improvements.md) — items marked 💡 *proposed* are candidates to implement (and back-port). -## Upstream blockers (not raven-python work, but worth tracking) +## Upstream blockers (not raven-toolbox work, but worth tracking) * `optlang.hybrid_interface.Configuration.clone()` bug — blocks HiGHS at any scale (CI catches it in `tests/test_init_solvers.py`). diff --git a/docs/reference/yaml_format.md b/docs/reference/yaml_format.md new file mode 100644 index 00000000..028f4dc9 --- /dev/null +++ b/docs/reference/yaml_format.md @@ -0,0 +1,332 @@ +# RAVEN / cobrapy YAML model format + +This document describes the YAML format produced and consumed by + +- **cobrapy** ([`cobra.io.{load,save}_yaml_model`](https://github.com/opencobra/cobrapy)) +- **raven-toolbox** (`raven_toolbox.io.yaml.{read,write}_yaml_model`, see [API](api/index.md)) +- **RAVEN MATLAB** (`readYAMLmodel.m` / `writeYAMLmodel.m` in the [RAVEN repo](https://github.com/SysBioChalmers/RAVEN/tree/feat/yeast-gem-shared/io)) + +The same file can be round-tripped through any of the three. cobrapy is the canonical core; raven-toolbox and RAVEN MATLAB add namespaced extensions (RAVEN curation fields, MIRIAM cross-refs already covered by cobrapy's `annotation`, and the GECKO `ec-*` sections) without disturbing the cobra-known shape. + +--- + +## At a glance + +```yaml +!!omap +- metaData: !!omap + - id: yeastGEM_develop + - name: The Consensus Genome-Scale Metabolic Model of Yeast + - version: 9.0.0 + - date: 2026-05-27 + - taxonomy: taxonomy/559292 +- metabolites: + - !!omap + - id: s_0001 + - name: ATP + - compartment: c + - charge: -4 + - formula: C10H16N5O13P3 + - annotation: !!omap + - kegg.compound: C00002 + - smiles: "[O-]P(=O)([O-])OP(=O)([O-])O..." + - inchis: InChI=1S/C10H16N5O13P3/... + - deltaG: -2768.1 +- reactions: + - !!omap + - id: r_0001 + - name: hexokinase + - metabolites: !!omap + - s_0001: -1.0 + - s_0568: -1.0 + - s_0394: 1.0 + - s_0423: 1.0 + - lower_bound: 0.0 + - upper_bound: 1000.0 + - gene_reaction_rule: YGL253W or YCL040W or YFR053C + - subsystem: Glycolysis / Gluconeogenesis + - notes: "MetaNetX ID curated (PR #220)" + - annotation: !!omap + - kegg.reaction: R00299 + - sbo: SBO:0000176 + - eccodes: 2.7.1.1 + - deltaG: -17.39 + - confidence_score: 2.0 +- genes: + - !!omap + - id: YGL253W + - name: HXK2 + - annotation: !!omap + - uniprot: P04807 +- compartments: !!omap + - c: cytoplasm + - e: extracellular +``` + +Three structural rules are non-obvious and worth pointing out before the field-by-field detail: + +1. The whole document is one **ordered mapping** — `!!omap` — at the root. Every nested map that should preserve key order is also `!!omap` (metaData, each metabolite / reaction / gene entry, `annotation`, `metabolites`, `compartments`, and the ec sections). +2. Each metabolite, reaction, and gene is **one `- !!omap` element** of a list. Inside that mapping, every field is written as `- key: value`. This is cobrapy's native shape and is what RAVEN MATLAB's reader keys off. +3. Strings are **unquoted by default**; quotes appear only when YAML would otherwise misparse the value (leading `-`, `[`, `?` or `:`; embedded `: ` or ` #`; values that look like `true` / `false` / `null`). + +--- + +## Top-level layout + +``` +!!omap +- metaData: !!omap # optional; RAVEN extension +- metabolites: # required +- reactions: # required +- genes: # required (may be `genes: []`) +- compartments: !!omap # required +- gecko_light: # optional; GECKO extension +- ec-rxns: # optional; GECKO extension +- ec-enzymes: # optional; GECKO extension +``` + +| Key | Required | Source | Notes | +|-----|----------|--------|-------| +| `metaData` | optional | RAVEN | Provenance block. Holds `id`, `name`, `version`, `date`, `taxonomy`, optionally `givenName` / `familyName` / `email` / `organization` / `note` / `sourceUrl`, plus `defaultLB` / `defaultUB`. Cobrapy ignores this block (no semantic loss for the core model). | +| `metabolites` | yes | cobra core | Ordered list of `- !!omap` entries. | +| `reactions` | yes | cobra core | Ordered list of `- !!omap` entries. | +| `genes` | yes | cobra core | Ordered list; may be `genes: []` for a model with no genes. | +| `compartments` | yes | cobra core | `!!omap` of `: `. | +| `gecko_light` | optional | GECKO | Scalar boolean. Cobrapy / raven-toolbox emit this at the top level; the older spelling `geckoLight` inside `metaData` is still accepted on read. | +| `ec-rxns` | optional | GECKO | Per-reaction kcat / source / enzymes coupling table. | +| `ec-enzymes` | optional | GECKO | Per-enzyme MW / sequence / concentration table. | + +Cobrapy writes `id` / `name` / `version` at the root level instead of inside `metaData`. The RAVEN readers accept both placements; the RAVEN writers normalize to the `metaData` form. + +--- + +## Metabolite entry + +Field order (cobra-core first, then RAVEN extensions): + +```yaml +- !!omap + - id: s_0001 # required + - name: ATP # cobra + - compartment: c # cobra + - charge: -4 # cobra (number) + - formula: C10H16N5O13P3 # cobra + - notes: "free-text" # cobra + - annotation: !!omap # cobra (MIRIAM + smiles) + - kegg.compound: C00002 + - chebi: + - CHEBI:15422 + - CHEBI:30616 + - sbo: SBO:0000247 + - smiles: "OC1=NC..." # quoted when it contains [ ] : etc. + - inchis: "InChI=1S/..." # RAVEN extension + - deltaG: -2768.1 # RAVEN extension + - metFrom: KEGG # RAVEN extension +``` + +Cobrapy emits exactly the first seven keys (the cobra-core block). raven-toolbox and RAVEN MATLAB additionally emit `inchis`, `deltaG`, and `metFrom` when those fields are populated. On read, cobrapy puts the RAVEN extensions on the metabolite as attribute fall-through; raven-toolbox captures them into `metabolite.notes` (keyed by their YAML name); RAVEN MATLAB stores them on `model.inchis` / `model.metDeltaG` / `model.metFrom`. + +Annotation entries with multiple values are emitted as a YAML list (`chebi:` then several `-` items). Single-value entries are emitted inline (`kegg.compound: C00002`). SMILES strings live inside the annotation block under the `smiles` key — not as a top-level metabolite field, which is the historical RAVEN MATLAB shape and is still accepted on read for backward compatibility. + +--- + +## Reaction entry + +```yaml +- !!omap + - id: r_0001 # required + - name: hexokinase # cobra + - metabolites: !!omap # cobra (sorted by met id) + - s_0001: -1.0 + - s_0394: 1.0 + - lower_bound: 0.0 # cobra (number) + - upper_bound: 1000.0 # cobra (number) + - gene_reaction_rule: YGL253W or YCL040W # cobra + - objective_coefficient: 1 # cobra; omitted when 0 + - subsystem: Glycolysis / Gluconeogenesis # cobra + - notes: "MetaNetX ID curated (PR #220)" # cobra + - annotation: !!omap # cobra + - kegg.reaction: R00299 + - sbo: SBO:0000176 + - eccodes: # RAVEN extension + - 2.7.1.1 + - 2.7.1.2 + - references: "PMID:12345" # RAVEN extension + - rxnFrom: KEGG # RAVEN extension + - deltaG: -17.39 # RAVEN extension + - confidence_score: 2.0 # RAVEN extension +``` + +Some fields are conditional: + +- `objective_coefficient` is only written when non-zero (cobrapy convention). +- The `metabolites` block uses `!!omap []` (flow-style empty omap) when the reaction has no metabolites — this keeps the file a valid YAML 1.2 document. +- `eccodes` is written inline (`eccodes: 2.7.1.1`) when there is exactly one code, and as a list when there are several. Same for `references`. + +**Notes key naming.** Cobrapy and the current raven-toolbox / RAVEN MATLAB writers use **`notes`**. Pre-`feat/yeast-gem-shared` yeast-GEM files used `rxnNotes`; both readers accept that as a legacy alias. + +**Bounds typing.** Bounds are emitted as floats with an explicit decimal point (`1000.0`, `-1000.0`), matching Python's float repr and cobrapy's output. + +--- + +## Gene entry + +```yaml +- !!omap + - id: YGL253W # required + - name: HXK2 # cobra; omitted when empty + - annotation: !!omap # cobra + - uniprot: P04807 + - ncbigene: 856421 + - protein: P04807 # RAVEN extension +``` + +Empty names (`name: ''`) are not emitted (matches RAVEN MATLAB's historical behavior). + +--- + +## Compartments + +```yaml +- compartments: !!omap + - c: cytoplasm + - e: extracellular + - m: mitochondrion +``` + +Just an `!!omap` of `: ` pairs. Compartments don't carry their own MIRIAMs in the current format. + +--- + +## metaData (RAVEN extension) + +```yaml +- metaData: !!omap + - id: yeastGEM_develop + - name: The Consensus Genome-Scale Metabolic Model of Yeast + - version: 9.0.0 + - date: 2026-05-27 + - defaultLB: -1000.0 + - defaultUB: 1000.0 + - givenName: Eduard + - familyName: Kerkhoven + - email: eduardk@chalmers.se + - organization: Chalmers University of Technology + - taxonomy: taxonomy/559292 + - note: "Saccharomyces cerevisiae - strain S288C" + - sourceUrl: https://github.com/SysBioChalmers/yeast-GEM +``` + +Pure provenance. Cobrapy ignores the block; raven-toolbox keeps the verbatim dictionary on `model.notes['metaData']` and additionally lifts `id` / `name` / `version` to `model.id` / `model.name` / `model.notes['version']` so cobra-shape accessors find them. RAVEN MATLAB populates `model.id` / `model.name` / `model.version` / `model.annotation.*` from the same fields. + +`date` is preserved across round-trips when present on the model; otherwise the writer fills in `YYYY-MM-DD` of the current date. + +--- + +## GECKO sections + +For enzyme-constrained models, three additional top-level keys carry the EC layer: + +```yaml +- gecko_light: false # true for the "light" formulation +- ec-rxns: + - !!omap + - id: r_0001 + - kcat: 25.3 + - source: brenda + - notes: "" + - eccodes: 2.7.1.1 + - enzymes: !!omap + - P04807: 1.0 +- ec-enzymes: + - !!omap + - genes: YGL253W + - enzymes: P04807 + - mw: 53942 + - sequence: "MVHLGPK..." + - concs: .nan +``` + +These map onto `model.ec` in RAVEN MATLAB and `raven_toolbox.io.ec_data.EcData` (attached as `model.ec`) in raven-toolbox. Cobrapy ignores the sections. + +The older spelling `geckoLight` inside `metaData` is also accepted on read. + +--- + +## Annotations + +The `annotation` block uses MIRIAM-style namespace keys. Cobrapy treats the block as a free-form dictionary; raven-toolbox preserves it verbatim through `cobra.Metabolite.annotation` / `Reaction.annotation` / `Gene.annotation`; RAVEN MATLAB maps it to `model.metMiriams` / `rxnMiriams` / `geneMiriams`. + +- A single value is written inline: `kegg.compound: C00002`. +- Multiple values are written as a YAML list: + + ```yaml + - chebi: + - CHEBI:15422 + - CHEBI:30616 + ``` + +- The `smiles` key inside a metabolite's `annotation` carries the SMILES string (cobrapy convention). RAVEN MATLAB historically emitted `smiles` as a metabolite top-level field; both readers still accept that, but writes are normalized to the annotation block. +- The `sbo` key carries the Systems Biology Ontology term assigned by `assignSBOterms` / `add_sbo_terms`. + +--- + +## Numbers, strings, quoting + +**Numbers.** Whole-number floats are written with an explicit `.0` (`1000.0`, `-1000.0`, `0.0`). Other floats use up to 15 significant digits (`-17.39`, `-2768.1`). `NaN` is encoded as `.nan`; `+Inf` / `-Inf` as `.inf` / `-.inf` (YAML 1.2 conventions). + +**Strings.** Default style is bare (no quotes). The writer falls back to double-quoted style when the value: + +- starts with `-`, `?`, `:`, or any flow indicator (`[`, `]`, `{`, `}`, `,`, `&`, `*`, `!`, `|`, `>`, `%`, `@`, `` ` ``, `#`); +- contains `: ` (would otherwise be parsed as a key/value), ` #` (comment), or one of `[`, `]`, `{`, `}`; +- has leading or trailing whitespace; +- spells a YAML reserved word case-insensitively (`true`, `false`, `null`, `yes`, `no`, `on`, `off`, `~`). + +In a double-quoted string, only `\` and `"` are escaped. Other characters (including Unicode and newlines if the underlying model permitted them) are passed through. + +--- + +## Tooling interoperability matrix + +| File written by ↓ \ Reader → | cobrapy | raven-toolbox | RAVEN MATLAB | +|---|---|---|---| +| cobrapy (`save_yaml_model`) | full | full + extras land in `notes` via attribute fall-through | works for root-level `id` / `name` / `version` (added in this release) | +| raven-toolbox (`write_yaml_model`) | core (no `metaData`-derived `id`); RAVEN extras live as unknown top-level keys but don't break parsing | full | full | +| RAVEN MATLAB (`writeYAMLmodel`) | core (no `metaData`-derived `id`); RAVEN extras land via attribute fall-through | full | full | + +"Full" = every field read back into its canonical position on the model object; "core" = cobrapy-known fields, RAVEN extensions ignored or kept on the object as attribute fall-through (`reaction.eccodes` etc., not re-emitted on save). A round-trip through cobrapy is therefore **lossy for RAVEN extensions** — only the core fields survive `cobrapy.load → cobrapy.save`. Round-trips through raven-toolbox or RAVEN MATLAB are lossless. + +--- + +## What round-tripping looks like + +Loading `yeast-GEM.yml` (2748 metabolites, 4102 reactions, 1143 genes) and re-writing it through any of the three tools preserves every documented piece of content: + +| Count | After round-trip | +|---|---| +| metabolites | 2748 / 2748 | +| reactions | 4102 / 4102 | +| genes | 1143 / 1143 | +| reactions with eccodes | 2411 | +| reactions with deltaG | 3984 | +| metabolites with deltaG | 2696 | +| metabolites with SMILES | 1788 | +| reactions with notes (rxnNotes) | 1443 | + +(Cobrapy round-trips give 2748 / 4102 / 1143 for the core but drop the RAVEN extensions in the rightmost column — that's the documented loss.) + +--- + +## What changed in `feat/yeast-gem-shared` + +- raven-toolbox writer no longer drops `!!omap` tags (was producing files RAVEN MATLAB's reader couldn't load). +- raven-toolbox now preserves `eccodes` and accepts the legacy `rxnNotes` reaction key on read. +- RAVEN MATLAB writer reorders metabolite / reaction fields to match cobrapy. +- RAVEN MATLAB writer renames the reaction `rxnNotes` key to `notes` and emits SMILES inside the annotation block (still accepts both shapes on read). +- RAVEN MATLAB writer's `preserveQuotes` default is now `false`; values that need quoting (SMILES with `[O-]`, leading flow indicators, booleans, `: `-containing strings) are quoted defensively per value. +- RAVEN MATLAB writer emits whole-number bounds as `1000.0` (matches cobrapy / Python float repr) instead of `1000`. +- RAVEN MATLAB reader accepts cobrapy's root-level `id` / `name` / `version` / `gecko_light`, the `!!omap`-tagged `metaData` header, and `notes` (canonical) in addition to `rxnNotes` (legacy). +- Empty `reaction.metabolites` blocks are emitted as `!!omap []` (valid YAML 1.2) rather than an empty `!!omap` with no value. +- Document-start marker `---` dropped to match cobrapy's bare `!!omap` root. + +These changes are byte-stable for cobrapy and raven-toolbox users; existing yeast-GEM YAML files continue to load. The first time a yeast-GEM curation pass rewrites the file with the new MATLAB writer, the diff will look large (because of the reordering and quote-style changes) but the model content is unchanged. diff --git a/docs/requirements.txt b/docs/requirements.txt index 98219c2a..49f81444 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,5 +1,5 @@ # Documentation build toolchain (used by ReadTheDocs and `make -C docs html`). -# The raven-python package itself is installed separately (see .readthedocs.yaml). +# The raven-toolbox package itself is installed separately (see .readthedocs.yaml). sphinx>=7.2 furo>=2024.1.29 myst-parser>=2.0 diff --git a/docs/studies/humangem_validation.md b/docs/studies/humangem_validation.md index bbaae6be..3f2c79d5 100644 --- a/docs/studies/humangem_validation.md +++ b/docs/studies/humangem_validation.md @@ -1,8 +1,8 @@ -# Human-GEM cell-type model validation: raven-python vs RAVEN +# Human-GEM cell-type model validation: raven-toolbox vs RAVEN -Validation of raven-python's tINIT/ftINIT against MATLAB RAVEN on a real genome-scale +Validation of raven-toolbox's tINIT/ftINIT against MATLAB RAVEN on a real genome-scale reconstruction (Human-GEM) using the Hart2015 RNA-seq dataset (5 cell lines: DLD1, -GBM, HCT116, HELA, RPE1). The goal is functional equivalence — do raven-python and RAVEN +GBM, HCT116, HELA, RPE1). The goal is functional equivalence — do raven-toolbox and RAVEN extract the *same* context-specific reaction sets from the same inputs? ## Method @@ -10,21 +10,21 @@ extract the *same* context-specific reaction sets from the same inputs? * **Template & inputs.** RAVEN built the ftINIT reference model from Human-GEM (`prepHumanModelForftINIT`: remove drug/exchange/artificial reactions, set spontaneous/custom lists) and exported it as `raven_refModel.xml` (10198 reactions). - raven-python builds on that *same* exported model, so the candidate reaction universe is + raven-toolbox builds on that *same* exported model, so the candidate reaction universe is identical and set comparison is exact. * **Scoring.** Gene scores from `log2(TPM+1)`-style expression via `gene_scores_from_expression`, mapped to reactions through the GPR (`score_reactions_from_genes`), matching RAVEN's `getExprForRxnScore`. * **ftINIT.** Series `1+1` (2 staged MILP steps). RAVEN run via `ftINIT.m` with Gurobi; - raven-python via `raven_python.init.ftinit` with Gurobi (`mip_gap=0.001`, `time_limit=600`). -* **tINIT.** raven-python `get_init_model` (classic single-MILP INIT) on HCT116, compared to + raven-toolbox via `raven_toolbox.init.ftinit` with Gurobi (`mip_gap=0.001`, `time_limit=600`). +* **tINIT.** raven-toolbox `get_init_model` (classic single-MILP INIT) on HCT116, compared to the ftINIT result for the same cell line. -* **Tasks.** Two raven-python ftINIT variants: *no-task* (expression only) and +* **Tasks.** Two raven-toolbox ftINIT variants: *no-task* (expression only) and *task-constrained* (essential metabolic tasks, `metabolicTasks_Essential.txt`, force task-essential reactions to be kept). RAVEN's reference is task-constrained. * **Solver.** Gurobi 13.0.1 for both tools. -## Engineering findings (raven-python tractability) +## Engineering findings (raven-toolbox tractability) Getting ftINIT to run at genome scale surfaced three issues, all now fixed and matching RAVEN's design: @@ -50,7 +50,7 @@ comparable to RAVEN. ### Reaction counts -| cell line | RAVEN ftINIT | raven-python ftINIT (no-task) | raven-python ftINIT (task) | +| cell line | RAVEN ftINIT | raven-toolbox ftINIT (no-task) | raven-toolbox ftINIT (task) | |-----------|-------------:|--------------------------:|-----------------------:| | DLD1 | 7782 | 7744 | 7774 | | GBM | 7668 | 7667 | 7680 | @@ -59,12 +59,12 @@ comparable to RAVEN. | RPE1 | 7569 | 7564 | 7570 | Counts agree within ~0.5 % everywhere; the task-constrained run is closest (e.g. RPE1 -7570 vs 7569, HCT116 7776 vs 7780). raven-python tINIT (HCT116) gives 6024 reactions — a +7570 vs 7569, HCT116 7776 vs 7780). raven-toolbox tINIT (HCT116) gives 6024 reactions — a smaller model, as expected from the different (classic INIT) objective. -### Agreement — raven-python (no-task) ftINIT vs RAVEN ftINIT +### Agreement — raven-toolbox (no-task) ftINIT vs RAVEN ftINIT -| cell line | shared | only raven-python | only RAVEN | Jaccard | +| cell line | shared | only raven-toolbox | only RAVEN | Jaccard | |-----------|-------:|--------------:|-----------:|--------:| | DLD1 | 7667 | 77 | 115 | 0.976 | | GBM | 7562 | 105 | 106 | 0.973 | @@ -77,9 +77,9 @@ though this run is *expression-only* while RAVEN's reference is task-constrained "only RAVEN" surplus (≈99–125) is expected to include task-essential reactions that the task-constrained run (below) recovers. -### Agreement — raven-python (task-constrained) ftINIT vs RAVEN ftINIT +### Agreement — raven-toolbox (task-constrained) ftINIT vs RAVEN ftINIT -| cell line | shared | only raven-python | only RAVEN | Jaccard | +| cell line | shared | only raven-toolbox | only RAVEN | Jaccard | |-----------|-------:|--------------:|-----------:|--------:| | DLD1 | 7699 | 75 | 83 | 0.980 | | GBM | 7588 | 92 | 80 | 0.978 | @@ -88,12 +88,12 @@ task-constrained run (below) recovers. | RPE1 | 7493 | 77 | 76 | 0.980 | Adding the essential metabolic tasks (same task list RAVEN uses) raises agreement to -**Jaccard 0.978–0.980** and makes the disagreement symmetric (only-raven-python ≈ only-RAVEN +**Jaccard 0.978–0.980** and makes the disagreement symmetric (only-raven-toolbox ≈ only-RAVEN ≈ 80), confirming the prediction: the task constraints recover RAVEN's task-essential reactions. The residual ≈80 reactions each way out of ~7700 is at the level expected from MIP-gap tolerance (both accept near-optimal incumbents) and alternate optima. -### raven-python tINIT vs ftINIT (HCT116) +### raven-toolbox tINIT vs ftINIT (HCT116) tINIT 6024 rxns vs ftINIT 7752; shared 5957, Jaccard 0.762. tINIT is nearly a subset (only 67 reactions unique to it) — the two methods agree on a common core, with ftINIT @@ -102,7 +102,7 @@ This matches the expected tINIT/ftINIT relationship rather than indicating a def ## Conclusions -From identical inputs on a genome-scale human reconstruction, raven-python reproduces RAVEN's +From identical inputs on a genome-scale human reconstruction, raven-toolbox reproduces RAVEN's ftINIT reaction selection to **97.5 % (no-task) and 98 % (task-constrained) set identity** across five cell lines — strong evidence of functional equivalence between the two independent implementations. Agreement is symmetric and the residual (~80 reactions each diff --git a/docs/studies/index.md b/docs/studies/index.md index 1db5f8ef..e78ec547 100644 --- a/docs/studies/index.md +++ b/docs/studies/index.md @@ -1,9 +1,9 @@ # Studies & validation -Empirical validation runs and parameter calibrations that back raven-python's defaults and +Empirical validation runs and parameter calibrations that back raven-toolbox's defaults and its equivalence claims against MATLAB RAVEN. -- **[Human-GEM validation](humangem_validation.md)** — raven-python ftINIT vs MATLAB RAVEN +- **[Human-GEM validation](humangem_validation.md)** — raven-toolbox ftINIT vs MATLAB RAVEN on 5 Hart2015 cell lines (Jaccard 0.975–0.980). - **[(f)tINIT parameter calibration](init_param_calibration.md)** — clean-data calibration plus input-robustness study (`mip_gap` / `big_m` / `force_on` / `eps` / `prod_weight` / diff --git a/docs/studies/init_param_calibration.md b/docs/studies/init_param_calibration.md index 899641af..0192e405 100644 --- a/docs/studies/init_param_calibration.md +++ b/docs/studies/init_param_calibration.md @@ -1,6 +1,6 @@ # (f)tINIT parameter calibration & input-robustness -Empirical study of raven-python's (f)tINIT parameters on a genome-scale model (Human-GEM, +Empirical study of raven-toolbox's (f)tINIT parameters on a genome-scale model (Human-GEM, Hart2015 / HCT116). Two questions: 1. **Calibration** — on clean data, which parameter values give the best speed/quality @@ -322,7 +322,7 @@ choice, is what bridges the gap. ## 3. Cross-solver portability See [init_solver_benchmark.md](init_solver_benchmark.md) for the genome-scale -solver comparison (Gurobi/HiGHS/GLPK) and [tests/test_init_solvers.py](https://github.com/SysBioChalmers/raven-python/blob/develop/tests/test_init_solvers.py) +solver comparison (Gurobi/HiGHS/GLPK) and [tests/test_init_solvers.py](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/tests/test_init_solvers.py) for CI parameterised over installed MILP backends. Headline: at genome scale only Gurobi is viable today; HiGHS fails on an upstream optlang `hybrid_interface.clone()` bug; GLPK ignores `configuration.timeout` on MIP and ran 1 h+ without converging. Toy-scale diff --git a/docs/studies/init_solver_benchmark.md b/docs/studies/init_solver_benchmark.md index 1cd97ac7..e986ac53 100644 --- a/docs/studies/init_solver_benchmark.md +++ b/docs/studies/init_solver_benchmark.md @@ -21,13 +21,13 @@ companion to the CI-scale `tests/test_init_solvers.py`. All our tractability tuning (big-M=100, `rescaleModelForINIT`, `mip_gap`, `time_limit`) was done on Gurobi and it pays off. * **HiGHS** (`hybrid_interface`) **does not work with cobra at all in this stack** — not - raven-python's bug. Cobra sets `model.solver = "hybrid"` which calls + raven-toolbox's bug. Cobra sets `model.solver = "hybrid"` which calls `optlang.interface.Model.clone()`, which re-applies a stored `lp_method="primal"` parameter that the `hybrid_interface.Configuration` rejects (it accepts only `auto/simplex/interior point`). This breaks `model.copy()` and any flow that swaps the solver — i.e. the whole pipeline. The same failure mode shows up at toy scale in `tests/test_init_solvers.py` (5/5 fail), so CI catches it now. Upstream optlang/cobra - patch needed; nothing to fix in raven-python. + patch needed; nothing to fix in raven-toolbox. * **GLPK** loads the model but its MIP solver does **not honor `configuration.timeout`** for this problem — we set the 900 s wall limit, GLPK still ran 1 h+ at 100 % CPU without producing a solution and had to be killed. GLPK has no @@ -49,7 +49,7 @@ companion to the CI-scale `tests/test_init_solvers.py`. parameter values that the generic clone path emits, or the clone path should drop unknown LP-method values gracefully. 2. GLPK's MIP solve should honor `configuration.timeout`. If upstream won't, - raven-python could implement a watchdog (separate thread sending `SIGINT` after the + raven-toolbox could implement a watchdog (separate thread sending `SIGINT` after the wall limit) specifically when the solver is GLPK. ## Reproducing diff --git a/docs/studies/kegg_hmm_cutoff_calibration.md b/docs/studies/kegg_hmm_cutoff_calibration.md index c2fd1066..873eea46 100644 --- a/docs/studies/kegg_hmm_cutoff_calibration.md +++ b/docs/studies/kegg_hmm_cutoff_calibration.md @@ -29,7 +29,7 @@ in three steps: recall = |pred ∩ truth| / |truth|, F1. Reaction-level: `rxn_rec` = fraction of the organism's true reactions recovered (KO→reaction via `ko_reaction`); `rxn_novel` = predicted reactions **not** in the annotation set. -- Reproduce with [`scripts/analyze_hmm_cutoffs.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/scripts/analyze_hmm_cutoffs.py). +- Reproduce with [`scripts/analyze_hmm_cutoffs.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/scripts/analyze_hmm_cutoffs.py). ### Important caveat @@ -199,5 +199,5 @@ Full reconstruction of *S. cerevisiae* two ways, at the old defaults: Reaction recall 96.3 % (1305/1355 shared, Jaccard 0.86); gene recall 96.6 % (807/835 shared, Jaccard 0.87). The annotation path also exercises the new -`organism_gene_ko.tsv.xz` artefact (K14). `hmmscan` throughput ≈ 0.1 s/query +`organism_gene_ko.tsv.gz` artefact (K14). `hmmscan` throughput ≈ 0.1 s/query against either library on 12 threads (yeast: 6021 queries in 633 s). diff --git a/docs/studies/yeast_localization_benchmark.md b/docs/studies/yeast_localization_benchmark.md index 09aa49ec..fa883647 100644 --- a/docs/studies/yeast_localization_benchmark.md +++ b/docs/studies/yeast_localization_benchmark.md @@ -1,10 +1,10 @@ # yeast-GEM localisation benchmark -Real-data validation of [`localization.predict_localization`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/localization/predict.py) +Real-data validation of [`localization.predict_localization`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/localization/predict.py) on the curated yeast-GEM. The benchmark is end-to-end — model, scoring, MILP — and sweeps predictor noise so the failure modes are visible, not just the headline accuracy. -* Driver: [`scripts/benchmark_localization_yeast.py`](https://github.com/SysBioChalmers/raven-python/blob/develop/scripts/benchmark_localization_yeast.py) +* Driver: [`scripts/benchmark_localization_yeast.py`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/scripts/benchmark_localization_yeast.py) * Yeast-GEM source: `pcSecYeastSpecies/Model/yeastGEM.xml` (3991 reactions, 1147 genes, 14 compartments). * Run command: @@ -26,7 +26,7 @@ sweeps predictor noise so the failure modes are visible, not just the headline a compartments collapse to 12 placement targets in the truth set (extracellular and the lipid particle / vacuolar membrane variants stay distinct). 2. **Flattening**: the model is squashed into one compartment with - [`manipulation.merge_compartments`](https://github.com/SysBioChalmers/raven-python/blob/develop/src/raven_python/manipulation/compartments.py) + [`manipulation.merge_compartments`](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/src/raven_toolbox/manipulation/compartments.py) so the predictor cannot lean on metabolite-topology evidence. Without this step every GPR'd reaction's "predicted" compartment is just its current one — vacuous. 3. **Reference scores**: each gene gets `1.0` in every compartment that hosts one of diff --git a/pyproject.toml b/pyproject.toml index adb2ab7e..6235f39b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,8 +3,8 @@ requires = ["hatchling"] build-backend = "hatchling.build" [project] -name = "raven-python" -version = "0.1.0a1" +name = "raven-toolbox" +version = "0.2.0" description = "Reconstruction, Analysis and Visualization of Metabolic Networks in Python, a port of the RAVEN Toolbox built on cobrapy" readme = "README.md" requires-python = ">=3.11" @@ -49,23 +49,21 @@ dependencies = [ dev = [ "pytest>=7", "pytest-cov", - "ruff>=0.4", + "ruff==0.15.15", + "mypy>=1.8", ] excel = [ "openpyxl>=3.1", ] -plotting = [ - "matplotlib>=3.5", -] [project.urls] -Homepage = "https://github.com/SysBioChalmers/raven-python" -Source = "https://github.com/SysBioChalmers/raven-python" -Issues = "https://github.com/SysBioChalmers/raven-python/issues" +Homepage = "https://github.com/SysBioChalmers/raven-toolbox" +Source = "https://github.com/SysBioChalmers/raven-toolbox" +Issues = "https://github.com/SysBioChalmers/raven-toolbox/issues" "RAVEN MATLAB" = "https://github.com/SysBioChalmers/RAVEN" [tool.hatch.build.targets.wheel] -packages = ["src/raven_python"] +packages = ["src/raven_toolbox"] [tool.pytest.ini_options] testpaths = ["tests"] @@ -82,3 +80,9 @@ ignore = [ [tool.ruff.lint.per-file-ignores] "tests/*" = ["E402"] + +[tool.mypy] +python_version = "3.11" +files = ["src/raven_toolbox"] +# cobra / optlang / scipy / ruamel ship no type stubs; don't error on their imports. +ignore_missing_imports = true diff --git a/scripts/README.md b/scripts/README.md index 67a3b362..e1d1d898 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,7 +1,7 @@ # Maintainer scripts Release-time tooling. Not part of the installed package — run them from a checkout -with raven-python installed (`pip install -e .`). End users never need these. +with raven-toolbox installed (`pip install -e .`). End users never need these. ## `build_kegg_artefacts.py` @@ -18,18 +18,18 @@ python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts --hmms -- ## `make_registry_snippet.py` After uploading the files to a release, compute their SHA256 and print the entry -to merge into the runtime registry — `raven_python.data._DATA_REGISTRY` (data) or -`raven_python.binaries._REGISTRY` (binary ZIP bundles). The checksum helper is shared +to merge into the runtime registry — `raven_toolbox.data._DATA_REGISTRY` (data) or +`raven_toolbox.binaries._REGISTRY` (binary ZIP bundles). The checksum helper is shared with the resolvers, so published checksums always match what `ensure_data` / `ensure_binary` verify. ```bash # Data artefacts: python scripts/make_registry_snippet.py data --dataset kegg --version kegg116 \ - --dir artefacts --base-url https://github.com/ORG/raven-python/releases/download/kegg-data-kegg116 + --dir artefacts --base-url https://github.com/ORG/raven-toolbox/releases/download/kegg-data-kegg116 # Binary bundle (ZIPs named ---.zip): python scripts/make_registry_snippet.py binary --bundle blast --version 2.16.0 \ --provides blastp makeblastdb --dir zips \ - --base-url https://github.com/ORG/raven-python/releases/download/blast-2.16.0 + --base-url https://github.com/ORG/raven-toolbox/releases/download/blast-2.16.0 ``` diff --git a/scripts/analyze_hmm_cutoffs.py b/scripts/analyze_hmm_cutoffs.py index 654fc021..73488689 100644 --- a/scripts/analyze_hmm_cutoffs.py +++ b/scripts/analyze_hmm_cutoffs.py @@ -29,8 +29,8 @@ import numpy as np import pandas as pd -from raven_python.reconstruction.kegg.parse import read_kegg_table -from raven_python.reconstruction.kegg.query import ( +from raven_toolbox.reconstruction.kegg.parse import read_kegg_table +from raven_toolbox.reconstruction.kegg.query import ( assign_kos, parse_hmmscan_tblout, run_hmmscan, @@ -50,7 +50,7 @@ def load_ko2rxn(artefacts: Path) -> dict[str, set[str]]: def ground_truth(artefacts: Path, org: str, ko2rxn) -> tuple[set, set]: - ogk = read_kegg_table(artefacts / "organism_gene_ko.tsv.xz") + ogk = read_kegg_table(artefacts / "organism_gene_ko.tsv.gz") rows = ogk[ogk["organism"].str.lower() == org] pairs = set(zip(rows["gene"], rows["ko"], strict=True)) rxns = {r for _, ko in pairs for r in ko2rxn.get(ko, ())} diff --git a/scripts/analyze_init_params.py b/scripts/analyze_init_params.py index 1059006f..523af535 100644 --- a/scripts/analyze_init_params.py +++ b/scripts/analyze_init_params.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 """Parameter calibration for (f)tINIT — intrinsic speed/quality sweeps (Phase 4d.7). -Genome-scale benchmark that sweeps the MILP/conditioning parameters of raven_python's -:func:`raven_python.init.run_ftinit`, :func:`raven_python.init.ftinit`, :func:`run_init`, and +Genome-scale benchmark that sweeps the MILP/conditioning parameters of raven_toolbox's +:func:`raven_toolbox.init.run_ftinit`, :func:`raven_toolbox.init.ftinit`, :func:`run_init`, and :func:`prep_init_model` and records, for each value, the *intrinsic* trade-off: wall-clock solve time, the MILP objective, and how far the result drifts from the tightest-setting ("reference") run — both in objective (relative gap) and in the **kept-reaction set** @@ -46,17 +46,17 @@ import cobra -from raven_python.init import ( +from raven_toolbox.init import ( ftinit, gene_scores_from_expression, get_init_model, prep_init_model, score_reactions_from_genes, ) -from raven_python.init.ftinit import run_ftinit -from raven_python.init.merge import group_rxn_scores -from raven_python.init.prep import rescale_for_init -from raven_python.init.steps import get_init_steps +from raven_toolbox.init.ftinit import run_ftinit +from raven_toolbox.init.merge import group_rxn_scores +from raven_toolbox.init.prep import rescale_for_init +from raven_toolbox.init.steps import get_init_steps # Sweep grids (first value of each tolerance sweep is the tight "reference"). MIP_GAPS = (0.0002, 0.001, 0.003, 0.01, 0.03, 0.1) diff --git a/scripts/analyze_init_robustness.py b/scripts/analyze_init_robustness.py index 1cac79ae..8c47cde6 100644 --- a/scripts/analyze_init_robustness.py +++ b/scripts/analyze_init_robustness.py @@ -51,13 +51,13 @@ import cobra import numpy as np -from raven_python.init import ( +from raven_toolbox.init import ( ftinit, gene_scores_from_expression, get_init_model, score_reactions_from_genes, ) -from raven_python.tasks import check_tasks, parse_task_list +from raven_toolbox.tasks import check_tasks, parse_task_list # Degradation grid (severity per kind). A mild and a severe point per kind. GRADIENT = { diff --git a/scripts/analyze_init_solvers.py b/scripts/analyze_init_solvers.py index e1e7b6c9..26781ab2 100644 --- a/scripts/analyze_init_solvers.py +++ b/scripts/analyze_init_solvers.py @@ -30,7 +30,7 @@ import cobra -from raven_python.init import ftinit, gene_scores_from_expression, score_reactions_from_genes +from raven_toolbox.init import ftinit, gene_scores_from_expression, score_reactions_from_genes _INTERFACES = {"gurobi": "gurobi_interface", "hybrid": "hybrid_interface", "glpk": "glpk_interface"} diff --git a/scripts/benchmark_localization_yeast.py b/scripts/benchmark_localization_yeast.py index 31c20760..c09eda69 100644 --- a/scripts/benchmark_localization_yeast.py +++ b/scripts/benchmark_localization_yeast.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Benchmark :func:`raven_python.localization.predict_localization` on yeast-GEM. +"""Benchmark :func:`raven_toolbox.localization.predict_localization` on yeast-GEM. Treats yeast-GEM's curated compartmentalisation as ground truth, flattens the model with :func:`merge_compartments` to a single compartment (so the algorithm cannot lean on @@ -32,8 +32,8 @@ import numpy as np import pandas as pd -from raven_python.localization import LocalizationScores, predict_localization -from raven_python.manipulation.compartments import merge_compartments +from raven_toolbox.localization import LocalizationScores, predict_localization +from raven_toolbox.manipulation.compartments import merge_compartments # --------------------------------------------------------------------------- inputs diff --git a/scripts/build_kegg_artefacts.py b/scripts/build_kegg_artefacts.py index 13fd00e7..1f594bfe 100644 --- a/scripts/build_kegg_artefacts.py +++ b/scripts/build_kegg_artefacts.py @@ -5,50 +5,53 @@ ``download_kegg_dump`` / ``fetch_keggdb``): * 3b.2 — ``parse_kegg_dump`` → ``reference_model.yml.gz`` + the gzipped-TSV tables; -* 3b.3 — ``build_hmm_library`` per domain → a pressed ``.hmm`` (+ hmmpress - sidecars), named so :func:`raven_python.data.ensure_kegg_hmm_library` can fetch them. +* 3b.3 — ``build_hmm_library`` per domain → a gzipped concatenated flatfile + ``_.hmm.gz`` (the client decompresses it and searches with + ``hmmsearch``), named so :func:`raven_toolbox.data.ensure_kegg_hmm_library` can fetch it. -Everything lands in ``--out`` ready to upload as release assets; feed that +Pass ``--version`` (e.g. ``kegg116``) to version-prefix every output filename, matching +the published release assets. Everything lands in ``--out`` ready to upload; feed that directory to ``scripts/make_registry_snippet.py data`` to emit the registry entry. Examples -------- Tables + reference model only (fast, no binaries):: - python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts + python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts --version kegg116 Full build incl. both HMM libraries (slow; needs HMMER/MAFFT/CD-HIT):: python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts \\ - --hmms --threads 8 + --version kegg116 --hmms --threads 8 """ from __future__ import annotations import argparse +import gzip import shutil +import tarfile from pathlib import Path -from raven_python.reconstruction.kegg import ( +from raven_toolbox.reconstruction.kegg import ( build_hmm_library, parse_kegg_dump, read_kegg_table, ) -# hmmpress sidecar extensions, alongside the .hmm. -_HMM_SIDECARS = (".h3f", ".h3i", ".h3m", ".h3p") +def _publish_library(work: dict, out_dir: Path, domain: str, prefix: str = "") -> Path: + """Gzip a built ``library.hmm`` to ``out_dir/.hmm.gz``. -def _publish_library(work: dict, out_dir: Path, domain: str) -> Path: - """Copy a built ``library.hmm`` (+ sidecars) to ``out_dir/.hmm``.""" + Only the concatenated flatfile is published (gzip, portable across HMMER + versions); the client decompresses it and searches with ``hmmsearch``, so the + same artefact also serves MATLAB RAVEN. + """ library = work["library"] if library is None: raise SystemExit(f"No HMMs built for {domain!r}; nothing to publish.") - target = out_dir / f"{domain}.hmm" - shutil.copyfile(library, target) - for suffix in _HMM_SIDECARS: - sidecar = library.with_name(library.name + suffix) - if sidecar.exists(): - shutil.copyfile(sidecar, target.with_name(target.name + suffix)) + target = out_dir / f"{prefix}{domain}.hmm.gz" + with open(library, "rb") as src, gzip.open(target, "wb") as out: + shutil.copyfileobj(src, out) return target @@ -56,6 +59,10 @@ def main(argv: list[str] | None = None) -> None: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument("--keggdb", required=True, type=Path, help="arranged KEGG dump directory") parser.add_argument("--out", required=True, type=Path, help="artefact output directory") + parser.add_argument( + "--version", default=None, + help="KEGG version (e.g. kegg116); version-prefixes every output filename", + ) parser.add_argument("--hmms", action="store_true", help="also build the HMM libraries") parser.add_argument( "--domains", nargs="+", default=["prokaryotes", "eukaryotes"], help="HMM domains to build" @@ -70,11 +77,21 @@ def main(argv: list[str] | None = None) -> None: args = parser.parse_args(argv) args.out.mkdir(parents=True, exist_ok=True) + prefix = f"{args.version}_" if args.version else "" print(">>> Parsing KEGG dump (3b.2)...") - paths = parse_kegg_dump(args.keggdb, args.out) + paths = parse_kegg_dump(args.keggdb, args.out, version=args.version) for name, path in paths.items(): print(f" {name}: {path}") + # Publish the taxonomy file too: domain split, plus the source for phyl_dist + # (RAVEN's keggPhylDist, used by GECKO). It is a raw dump file, so gzip it as-is. + tax_src = args.keggdb / "taxonomy" + if tax_src.is_file(): + tax_out = args.out / f"{prefix}taxonomy.gz" + with open(tax_src, "rb") as src, gzip.open(tax_out, "wb") as out: + shutil.copyfileobj(src, out) + print(f" taxonomy: {tax_out}") + if args.hmms: ogk = read_kegg_table(paths["organism_gene_ko"]) genes_pep = args.keggdb / "genes.pep" @@ -86,12 +103,27 @@ def main(argv: list[str] | None = None) -> None: domain=domain, seq_identity=args.seq_identity, parttree_residues=args.parttree_residues, threads=args.threads, ) - published = _publish_library(work, args.out, domain) + published = _publish_library(work, args.out, domain, prefix) print(f" {domain}: {published} ({len(work['hmms'])} profiles)") + # Bundle the core model artefacts (reference model + tables) into one archive that + # ensure_kegg_data fetches and extracts. HMMs and taxonomy stay separate assets. + # (After the HMM step, which reads organism_gene_ko.) + core_members = [ + paths[n] + for n in ("reference_model", "ko_reaction", "ko_names", "organism_gene_ko", "rxn_flags") + ] + bundle = args.out / f"{prefix}core.tar.gz" + with tarfile.open(bundle, "w:gz") as tar: + for member in core_members: + tar.add(member, arcname=member.name) + for member in core_members: + member.unlink() + print(f" core bundle: {bundle} ({len(core_members)} files)") + print(f"\n>>> Done. Upload the contents of {args.out} as release assets, then run:") print(" python scripts/make_registry_snippet.py data --dataset kegg " - f"--version --dir {args.out} --base-url ") + f"--version {args.version or ''} --dir {args.out} --base-url ") if __name__ == "__main__": diff --git a/scripts/make_registry_snippet.py b/scripts/make_registry_snippet.py index 3efa49ef..461e0500 100644 --- a/scripts/make_registry_snippet.py +++ b/scripts/make_registry_snippet.py @@ -2,7 +2,7 @@ """Emit ready-to-paste registry entries for published artefacts / binary ZIPs. Computes the SHA256 of each file and prints the Python/JSON entry to merge into -``raven_python.data._DATA_REGISTRY`` (data artefacts) or ``raven_python.binaries._REGISTRY`` +``raven_toolbox.data._DATA_REGISTRY`` (data artefacts) or ``raven_toolbox.binaries._REGISTRY`` (binary bundles). Run once per release, after uploading the files to the release. Examples @@ -11,15 +11,28 @@ python scripts/make_registry_snippet.py data \\ --dataset kegg --version kegg116 --dir artefacts \\ - --base-url https://github.com/ORG/raven_python/releases/download/kegg-data-kegg116 + --base-url https://github.com/ORG/raven_toolbox/releases/download/kegg-data-kegg116 Binary bundle (one ZIP per platform, named ``---.zip``):: python scripts/make_registry_snippet.py binary \\ --bundle blast --version 2.16.0 --provides blastp makeblastdb --dir zips \\ - --base-url https://github.com/ORG/raven_python/releases/download/blast-2.16.0 + --base-url https://github.com/ORG/raven_toolbox/releases/download/blast-2.16.0 -The SHA256 helper is shared with the runtime resolvers (``raven_python.binaries``), so +Add/update an entry in the shared ``manifest.json`` (the single source of truth read by +both raven-toolbox and MATLAB RAVEN — see data/manifest.schema.json):: + + python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \\ + --target data --dataset kegg --version kegg116 --dir artefacts \\ + --base-url https://github.com/ORG/raven-data/releases/download/kegg-kegg116 \\ + --doi 10.5281/zenodo.0000000 + + python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \\ + --target binary --bundle diamond --version 2.1.9 --provides diamond --dir zips \\ + --base-url https://github.com/ORG/raven-data/releases/download/diamond-2.1.9 \\ + --license GPL-3.0-only + +The SHA256 helper is shared with the runtime resolvers (``raven_toolbox.binaries``), so published checksums always match what ``ensure_data`` / ``ensure_binary`` verify. """ from __future__ import annotations @@ -29,7 +42,7 @@ import sys from pathlib import Path -from raven_python.binaries import _sha256 +from raven_toolbox.binaries import _sha256 def _files_in(directory: Path) -> list[Path]: @@ -68,34 +81,117 @@ def render(key: str, entry: dict) -> str: return json.dumps({key: entry}, indent=4) +# --- manifest.json (shared source of truth) -------------------------------- + + +def _file_meta(path: Path, base: str) -> dict: + """Manifest file record: url + sha256 + byte size.""" + return {"url": f"{base}/{path.name}", "sha256": _sha256(path), "bytes": path.stat().st_size} + + +def manifest_data_entry(version: str, base_url: str, directory: Path, **meta: str) -> dict: + """Build a manifest ``data`` dataset entry (registry fields + optional metadata).""" + base = base_url.rstrip("/") + files = {p.name: _file_meta(p, base) for p in _files_in(directory)} + if not files: + raise SystemExit(f"No files found in {directory}") + entry = {"version": version} + entry.update({k: v for k, v in meta.items() if v}) # description/license/doi/source + entry["files"] = files + return entry + + +def manifest_binary_entry( + bundle: str, version: str, provides: list[str], base_url: str, directory: Path, **meta: str +) -> dict: + """Build a manifest ``binaries`` bundle entry from per-platform ZIPs.""" + base = base_url.rstrip("/") + prefix = f"{bundle}-{version}-" + platforms = { + zp.name[len(prefix) : -len(".zip")]: _file_meta(zp, base) + for zp in sorted(directory.glob(f"{prefix}*.zip")) + } + if not platforms: + raise SystemExit(f"No {prefix}*.zip files found in {directory}") + entry = {"version": version, "provides": provides} + entry.update({k: v for k, v in meta.items() if v}) # description/license + entry["platforms"] = platforms + return entry + + +def update_manifest(manifest_path: Path, section: str, key: str, entry: dict) -> None: + """Insert ``entry`` under ``manifest[section][key]`` and write the manifest back.""" + if manifest_path.exists(): + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + else: + manifest = {"manifest_version": 1} + manifest.setdefault("manifest_version", 1) + manifest.setdefault(section, {})[key] = entry + manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + + def main(argv: list[str] | None = None) -> None: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) sub = parser.add_subparsers(dest="kind", required=True) - d = sub.add_parser("data", help="data-artefact registry entry (raven_python.data)") + d = sub.add_parser("data", help="data-artefact registry entry (raven_toolbox.data)") d.add_argument("--dataset", required=True, help="dataset key, e.g. 'kegg'") d.add_argument("--version", required=True) d.add_argument("--dir", required=True, type=Path, help="directory of uploaded artefacts") d.add_argument("--base-url", required=True, help="release download URL prefix") - b = sub.add_parser("binary", help="binary-bundle registry entry (raven_python.binaries)") + b = sub.add_parser("binary", help="binary-bundle registry entry (raven_toolbox.binaries)") b.add_argument("--bundle", required=True, help="bundle key, e.g. 'blast'") b.add_argument("--version", required=True) b.add_argument("--provides", nargs="+", required=True, help="executables the bundle provides") b.add_argument("--dir", required=True, type=Path, help="directory of uploaded ZIPs") b.add_argument("--base-url", required=True, help="release download URL prefix") + m = sub.add_parser("manifest", help="add/update an entry in the shared manifest.json") + m.add_argument("--manifest", required=True, type=Path, help="manifest.json to create/update") + m.add_argument("--target", required=True, choices=["data", "binary"]) + m.add_argument("--version", required=True) + m.add_argument("--dir", required=True, type=Path, help="directory of uploaded files") + m.add_argument("--base-url", required=True, help="release download URL prefix") + m.add_argument("--dataset", help="data: dataset key, e.g. 'kegg'") + m.add_argument("--bundle", help="binary: bundle key, e.g. 'diamond'") + m.add_argument("--provides", nargs="+", help="binary: executables the bundle provides") + m.add_argument("--description") + m.add_argument("--license") + m.add_argument("--doi", help="data: Zenodo (or other) DOI for this version") + m.add_argument("--source", help="data: human-facing release/record page") + args = parser.parse_args(argv) if args.kind == "data": key, entry = args.dataset, data_entry(args.dataset, args.version, args.base_url, args.dir) - target = "raven_python/data.py _DATA_REGISTRY" - else: + target = "raven_toolbox/data.py _DATA_REGISTRY" + print(f"# Merge into {target}:", file=sys.stderr) + print(render(key, entry)) + elif args.kind == "binary": key = args.bundle entry = binary_entry(args.bundle, args.version, args.provides, args.base_url, args.dir) - target = "raven_python/binaries.py _REGISTRY" - - print(f"# Merge into {target}:", file=sys.stderr) - print(render(key, entry)) + target = "raven_toolbox/binaries.py _REGISTRY" + print(f"# Merge into {target}:", file=sys.stderr) + print(render(key, entry)) + else: # manifest + if args.target == "data": + if not args.dataset: + parser.error("--dataset is required for --target data") + entry = manifest_data_entry( + args.version, args.base_url, args.dir, + description=args.description, license=args.license, doi=args.doi, source=args.source, + ) + update_manifest(args.manifest, "data", args.dataset, entry) + print(f"Updated {args.manifest}: data/{args.dataset} ({len(entry['files'])} files)", file=sys.stderr) + else: + if not (args.bundle and args.provides): + parser.error("--bundle and --provides are required for --target binary") + entry = manifest_binary_entry( + args.bundle, args.version, args.provides, args.base_url, args.dir, + description=args.description, license=args.license, + ) + update_manifest(args.manifest, "binaries", args.bundle, entry) + print(f"Updated {args.manifest}: binaries/{args.bundle} ({len(entry['platforms'])} platforms)", file=sys.stderr) if __name__ == "__main__": diff --git a/src/raven_python/__init__.py b/src/raven_python/__init__.py deleted file mode 100644 index 591c4c18..00000000 --- a/src/raven_python/__init__.py +++ /dev/null @@ -1,10 +0,0 @@ -"""raven_python — Python counterpart of the RAVEN Toolbox, built on cobrapy. - -raven_python reuses cobrapy for simulation, standard analyses, SBML I/O, and model -manipulation, and provides the RAVEN-specific functionality on top: de novo -reconstruction (KEGG / homology), context-specific modeling (tINIT / ftINIT), -metabolic task validation, connectivity gap-filling, omics integration (HPA), -sub-cellular localisation, N-model comparison, and the RAVEN-style I/O formats. -""" - -__version__ = "0.0.1" diff --git a/src/raven_python/comparison/__init__.py b/src/raven_python/comparison/__init__.py deleted file mode 100644 index e4b4c199..00000000 --- a/src/raven_python/comparison/__init__.py +++ /dev/null @@ -1,7 +0,0 @@ -"""Structural and functional comparison across multiple models. - -See :func:`raven_python.comparison.compare.compare_models`. -""" -from raven_python.comparison.compare import ModelComparison, compare_models - -__all__ = ["ModelComparison", "compare_models"] diff --git a/src/raven_python/data.py b/src/raven_python/data.py deleted file mode 100644 index b1264be7..00000000 --- a/src/raven_python/data.py +++ /dev/null @@ -1,135 +0,0 @@ -"""Fetch and cache published data artefacts (KEGG reference model, tables, HMMs). - -The mirror of :mod:`raven_python.binaries` for *data*: a version-pinned registry of -downloadable artefacts, fetched on first use, SHA256-verified, and cached under -platformdirs so end users never rebuild them from a KEGG dump (that is the -maintainer's job — see docs/maintaining_kegg_data.md). - -Resolution for any artefact file: - - explicit local dir → cached copy → download from the registry (verify, - cache) → FileNotFoundError with guidance - -The registry is **empty until the artefacts are published** (same as -``binaries._REGISTRY``); until then ``ensure_data_file`` raises an actionable -error. Cache layout:: - - $XDG_CACHE_HOME/raven_python/data/-/ - (or ~/.cache/raven_python/data/... if XDG_CACHE_HOME is unset) -""" -from __future__ import annotations - -import os -import shutil -from pathlib import Path -from urllib.request import urlopen - -from raven_python.binaries import _sha256 - -# dataset -> {"version": str, "files": {filename: {"url": str, "sha256": str}}} -# Populated when raven_python publishes the KEGG artefacts as release assets. -_DATA_REGISTRY: dict = {} - -# The core KEGG artefacts needed to build a model (no HMM libraries). -CORE_KEGG_FILES = ( - "reference_model.yml.gz", - "ko_reaction.tsv.gz", - "ko_names.tsv.gz", - "organism_gene_ko.tsv.xz", - "rxn_flags.tsv.gz", -) - - -def _data_cache_dir() -> Path: - base = os.environ.get("XDG_CACHE_HOME") or (Path.home() / ".cache") - return Path(base) / "raven_python" / "data" - - -def _bundle(dataset: str, registry: dict) -> dict: - bundle = registry.get(dataset) - if bundle is None: - raise FileNotFoundError( - f"No data artefacts registered for {dataset!r}. Either pass a local " - f"directory of artefacts, or build them per docs/maintaining_kegg_data.md." - ) - return bundle - - -def ensure_data_file( - dataset: str, - filename: str, - *, - version: str | None = None, - registry: dict | None = None, -) -> Path: - """Download (if needed) and return the cached path to one artefact file. - - Looks the file up in the registry for ``dataset`` (at ``version`` or the - registry's default), downloads it to the version-pinned cache directory, - verifies its SHA256, and returns the path. Re-uses an already-cached copy. - """ - registry = _DATA_REGISTRY if registry is None else registry - bundle = _bundle(dataset, registry) - ver = version or bundle["version"] - entry = bundle.get("files", {}).get(filename) - if entry is None: - raise FileNotFoundError( - f"{filename!r} is not registered for {dataset!r} {ver}. " - f"Available: {sorted(bundle.get('files', {}))}." - ) - - dest_dir = _data_cache_dir() / f"{dataset}-{ver}" - dest = dest_dir / filename - if dest.exists(): - return dest - - dest_dir.mkdir(parents=True, exist_ok=True) - tmp = dest.with_name(dest.name + ".part") - with urlopen(entry["url"]) as resp, open(tmp, "wb") as out: # noqa: S310 (trusted registry URLs) - shutil.copyfileobj(resp, out) - digest = _sha256(tmp) - if digest != entry["sha256"]: - tmp.unlink(missing_ok=True) - raise ValueError( - f"SHA256 mismatch for {dataset}/{filename} ({ver}): " - f"expected {entry['sha256']}, got {digest}." - ) - tmp.replace(dest) - return dest - - -def ensure_kegg_data( - *, - version: str | None = None, - files: tuple[str, ...] = CORE_KEGG_FILES, - registry: dict | None = None, -) -> Path: - """Ensure the core KEGG artefacts are cached; return their directory. - - Fetches each of ``files`` (default :data:`CORE_KEGG_FILES`) for the ``kegg`` - dataset and returns the cache directory holding them — ready to pass as the - ``artefact_dir`` of :func:`get_kegg_model_for_organism_from_artefacts`. - """ - registry = _DATA_REGISTRY if registry is None else registry - ver = version or _bundle("kegg", registry)["version"] - for filename in files: - ensure_data_file("kegg", filename, version=ver, registry=registry) - return _data_cache_dir() / f"kegg-{ver}" - - -def ensure_kegg_hmm_library( - domain: str, *, version: str | None = None, registry: dict | None = None -) -> Path: - """Ensure a domain HMM library (and its hmmpress index) is cached; return its path. - - ``domain`` is ``"prokaryotes"`` or ``"eukaryotes"``. Fetches ``.hmm`` - plus the ``hmmpress`` sidecar files (``.h3f/.h3i/.h3m/.h3p``) and returns the - path to the ``.hmm`` (the argument for :func:`run_hmmscan`). - """ - registry = _DATA_REGISTRY if registry is None else registry - ver = version or _bundle("kegg", registry)["version"] - base = f"{domain}.hmm" - library = ensure_data_file("kegg", base, version=ver, registry=registry) - for suffix in (".h3f", ".h3i", ".h3m", ".h3p"): - ensure_data_file("kegg", base + suffix, version=ver, registry=registry) - return library diff --git a/src/raven_python/plotting/__init__.py b/src/raven_python/plotting/__init__.py deleted file mode 100644 index 4fb38392..00000000 --- a/src/raven_python/plotting/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Pathway-map and omics-overlay visualisation (stub — not yet implemented).""" diff --git a/src/raven_python/reconstruction/kegg/taxonomy.py b/src/raven_python/reconstruction/kegg/taxonomy.py deleted file mode 100644 index 463fccec..00000000 --- a/src/raven_python/reconstruction/kegg/taxonomy.py +++ /dev/null @@ -1,71 +0,0 @@ -"""Parse the KEGG ``taxonomy`` file into per-organism category lineages. - -Ports the file-reading half of RAVEN ``getPhylDist`` (the distance-matrix half is -step 3b.5). The ``taxonomy`` file is an indented tree: ``#``-prefixed lines name a -category, the number of leading ``#`` giving its depth; organism lines are -tab-separated ``T-numberorg_idname...``. Each organism inherits the -stack of categories above it, the first of which is its domain (``Prokaryotes`` / -``Eukaryotes``). - -Used by 3b.3 to split genes into the prok/euk HMM libraries, and (later) by 3b.5 -for phylogenetic distances. -""" -from __future__ import annotations - -import warnings -from pathlib import Path - - -def parse_taxonomy(path: str | Path) -> dict[str, list[str]]: - """Return ``{organism_id: [category, ...]}`` from outermost to innermost.""" - org_categories: dict[str, list[str]] = {} - stack: list[str] = [] - skipped_level_warned = False - with open(path, encoding="utf-8") as handle: - for line_no, raw in enumerate(handle, start=1): - line = raw.rstrip("\n") - if not line.strip(): - continue - if line.startswith("#"): - depth = len(line) - len(line.lstrip("#")) - name = line[depth:].strip() - if depth - 1 > len(stack): - # Depth-skip (e.g. ## then ####): the original `stack[:depth-1]` - # silently produced a too-short lineage. Pad with explicit - # blanks so downstream slices stay aligned; warn once. - if not skipped_level_warned: - warnings.warn( - f"{path}: taxonomy depth skips a level near line {line_no} " - f"({'#' * depth} {name!r} appeared with stack {stack!r}); " - "padding the missing levels with '' (later occurrences silenced).", - stacklevel=2, - ) - skipped_level_warned = True - stack = stack + [""] * (depth - 1 - len(stack)) - else: - stack = stack[: depth - 1] - stack.append(name) - else: - fields = line.split("\t") if "\t" in line else line.split() - if len(fields) < 2: - continue - org_categories[fields[1].strip()] = list(stack) - return org_categories - - -def organism_domains(path: str | Path) -> dict[str, str]: - """Return ``{organism_id: domain}`` (the outermost category).""" - return {org: cats[0] for org, cats in parse_taxonomy(path).items() if cats} - - -def organisms_in_domain(path: str | Path, domain: str) -> set[str]: - """Organism ids whose outermost category matches ``domain`` (case-insensitive). - - Accepts a prefix, so ``"prok"`` matches ``"Prokaryotes"``. - """ - needle = domain.lower() - return { - org - for org, dom in organism_domains(path).items() - if dom.lower().startswith(needle) or needle.startswith(dom.lower()) - } diff --git a/src/raven_toolbox/__init__.py b/src/raven_toolbox/__init__.py new file mode 100644 index 00000000..b1bc2253 --- /dev/null +++ b/src/raven_toolbox/__init__.py @@ -0,0 +1,15 @@ +"""raven_toolbox — Python counterpart of the RAVEN Toolbox, built on cobrapy. + +raven_toolbox reuses cobrapy for simulation, standard analyses, SBML I/O, and model +manipulation, and provides the RAVEN-specific functionality on top: de novo +reconstruction (KEGG / homology), context-specific modeling (tINIT / ftINIT), +metabolic task validation, connectivity gap-filling, omics integration (HPA), +sub-cellular localisation, N-model comparison, and the RAVEN-style I/O formats. +""" + +from importlib.metadata import PackageNotFoundError, version + +try: + __version__ = version("raven-toolbox") +except PackageNotFoundError: # running from a source tree without an install + __version__ = "0.0.0+unknown" diff --git a/src/raven_python/analysis/__init__.py b/src/raven_toolbox/analysis/__init__.py similarity index 73% rename from src/raven_python/analysis/__init__.py rename to src/raven_toolbox/analysis/__init__.py index d85afef4..fbd7e273 100644 --- a/src/raven_python/analysis/__init__.py +++ b/src/raven_toolbox/analysis/__init__.py @@ -4,9 +4,9 @@ * :func:`fseof` — Flux Scanning based on Enforced Objective Flux. * :func:`random_sampling` — random-objective flux sampling. """ -from raven_python.analysis.fseof import FSEOFResult, fseof -from raven_python.analysis.reporter import ReporterResult, reporter_metabolites -from raven_python.analysis.sampling import ( +from raven_toolbox.analysis.fseof import FSEOFResult, fseof +from raven_toolbox.analysis.reporter import ReporterResult, reporter_metabolites +from raven_toolbox.analysis.sampling import ( RandomSamplingResult, find_good_reactions, random_sampling, diff --git a/src/raven_python/analysis/fseof.py b/src/raven_toolbox/analysis/fseof.py similarity index 100% rename from src/raven_python/analysis/fseof.py rename to src/raven_toolbox/analysis/fseof.py diff --git a/src/raven_python/analysis/reporter.py b/src/raven_toolbox/analysis/reporter.py similarity index 97% rename from src/raven_python/analysis/reporter.py rename to src/raven_toolbox/analysis/reporter.py index 5d96d47b..5150968e 100644 --- a/src/raven_python/analysis/reporter.py +++ b/src/raven_toolbox/analysis/reporter.py @@ -60,6 +60,8 @@ def _reporter_one(model: cobra.Model, gene_z: dict[str, float], test: str) -> Re raw = zs.sum() / math.sqrt(n) # Exact background correction for sampling-with-replacement (see module doc). corrected = (raw - math.sqrt(n) * mu) / sigma if sigma > 0 else 0.0 + if not math.isfinite(corrected): # never on clamped z-scores; guards the p-value + corrected = 0.0 rows.append( { "metabolite": met.id, diff --git a/src/raven_python/analysis/sampling.py b/src/raven_toolbox/analysis/sampling.py similarity index 91% rename from src/raven_python/analysis/sampling.py rename to src/raven_toolbox/analysis/sampling.py index 429b164a..5d110545 100644 --- a/src/raven_python/analysis/sampling.py +++ b/src/raven_toolbox/analysis/sampling.py @@ -36,7 +36,9 @@ import cobra import numpy as np import pandas as pd +from cobra.exceptions import OptimizationError from cobra.flux_analysis import flux_variability_analysis, pfba +from optlang.symbolics import add logger = logging.getLogger(__name__) @@ -185,14 +187,23 @@ def random_sampling( weights = rng.random(n_objectives) * signs terms = [w * good_rxn_objs[j].flux_expression for j, w in zip(chosen, weights, strict=True)] - model.objective = model.problem.Objective(sum(terms), direction="max") + # add() (not sum()) builds the symbolic objective in one pass; sum() + # re-canonicalises the optlang expression on every term (O(n^2)). + model.objective = model.problem.Objective(add(terms), direction="max") sol = model.optimize() if sol.status == "optimal" and abs(sol.objective_value) > 1e-8: - samples[i, :] = (pfba(model) if min_flux else sol).fluxes.reindex(reaction_ids).to_numpy() + fluxes = (pfba(model) if min_flux else sol).fluxes.reindex(reaction_ids) + if fluxes.isna().any(): # solver returned an unexpected reaction set + missing = fluxes.index[fluxes.isna()].tolist() + raise OptimizationError( + "solver returned fluxes missing reaction(s) " + f"{missing[:5]}; cannot assemble a NaN-free sample matrix." + ) + samples[i, :] = fluxes.to_numpy() break if attempt == max_attempts: if not suppress_errors: - raise RuntimeError( + raise OptimizationError( "Could not find a non-zero, loop-free solution after " f"{max_attempts} attempts for sample {i}. Review the model's " "constraints, or set suppress_errors=True." diff --git a/src/raven_toolbox/annotation/__init__.py b/src/raven_toolbox/annotation/__init__.py new file mode 100644 index 00000000..4d5d426b --- /dev/null +++ b/src/raven_toolbox/annotation/__init__.py @@ -0,0 +1,24 @@ +"""Annotation helpers — SBO term assignment, ΔG CSV persistence. + +These are the pieces of yeast-GEM's ``missingFields`` module that are +organism-agnostic enough to live upstream. Default parameter values +match the RAVEN/yeast convention so the functions are immediately +useful on the standard layout; consumers with different naming pass +overrides. +""" +from raven_toolbox.annotation.delta_g import load_delta_g_csv, save_delta_g_csv +from raven_toolbox.annotation.sbo import ( + DEFAULT_BIOMASS_MET_NAMES, + DEFAULT_BIOMASS_RXN_NAME, + DEFAULT_NGAM_RXN_NAME, + add_sbo_terms, +) + +__all__ = [ + "DEFAULT_BIOMASS_MET_NAMES", + "DEFAULT_BIOMASS_RXN_NAME", + "DEFAULT_NGAM_RXN_NAME", + "add_sbo_terms", + "load_delta_g_csv", + "save_delta_g_csv", +] diff --git a/src/raven_toolbox/annotation/delta_g.py b/src/raven_toolbox/annotation/delta_g.py new file mode 100644 index 00000000..e0a90e0e --- /dev/null +++ b/src/raven_toolbox/annotation/delta_g.py @@ -0,0 +1,115 @@ +"""Persist scalar annotations (Gibbs free energy, etc.) as a side-car CSV. + +The committed SBML standard has no slot for thermodynamic ΔG values, so +projects like yeast-GEM keep them in a paired CSV instead. This module +provides the generic load / save round-trip — the file format is a two +column table ``, `` — and exposes the per-call schema so +existing projects can configure their conventions. + +The functions operate on cobra ``.notes`` (string round-tripped through +SBML), keeping any other notes untouched. +""" +from __future__ import annotations + +import math +from collections.abc import Iterable +from pathlib import Path + +import cobra +import pandas as pd + + +def load_delta_g_csv( + entities: Iterable, + path: str | Path, + *, + id_column: str = "Var1", + value_column: str = "Var2", + note_key: str = "deltaG", + verbose: bool = False, +) -> int: + """Stamp ``note_key`` on each entity from a CSV of ``id → value``. + + Parameters + ---------- + entities + Cobra entities (``model.metabolites`` or ``model.reactions``). + path + Path to the CSV. + id_column, value_column + Column labels in the CSV. Defaults match the yeast-GEM + convention (``Var1`` / ``Var2`` from MATLAB's ``array2table``). + note_key + Key under which the value is stored on ``entity.notes``. + Default ``"deltaG"``. + verbose + Print a summary of unmatched entity ids. + + Returns + ------- + The number of entities that were stamped (i.e. matched the CSV). + """ + df = pd.read_csv(path) + if id_column not in df.columns or value_column not in df.columns: + raise ValueError( + f"{path} columns {list(df.columns)} do not contain " + f"both {id_column!r} and {value_column!r}" + ) + lookup = dict(zip(df[id_column], df[value_column], strict=True)) + + stamped = 0 + missing: list[str] = [] + for entity in entities: + value = lookup.get(entity.id) + if value is None or (isinstance(value, float) and math.isnan(value)): + missing.append(entity.id) + continue + entity.notes[note_key] = str(value) + stamped += 1 + + if verbose and missing: + print( + f"{len(missing)} entity ids were not in {path} " + f"(e.g. {missing[:3]}); left untouched." + ) + return stamped + + +def save_delta_g_csv( + entities: Iterable, + path: str | Path, + *, + id_column: str = "Var1", + value_column: str = "Var2", + note_key: str = "deltaG", +) -> int: + """Dump ``entity.notes[note_key]`` for each entity to a CSV. + + Entities without ``note_key`` set get ``NaN`` written, preserving + one-row-per-entity ordering (mirrors MATLAB's ``array2table([ids, + values])`` behaviour). + + Returns + ------- + The number of rows written (= number of entities). + """ + rows: list[tuple[str, float]] = [] + for entity in entities: + raw = entity.notes.get(note_key) + if raw is None: + value: float = math.nan + else: + try: + value = float(raw) + except ValueError: + value = math.nan + rows.append((entity.id, value)) + pd.DataFrame(rows, columns=[id_column, value_column]).to_csv(path, index=False) + return len(rows) + + +# Re-export the cobra Model type for type-checker friendliness; helps +# IDEs surface the right hints to callers that hand us model.metabolites +# / model.reactions directly. +__all__ = ["load_delta_g_csv", "save_delta_g_csv"] +_ = cobra # silence "imported but unused" — used for typing context above diff --git a/src/raven_toolbox/annotation/sbo.py b/src/raven_toolbox/annotation/sbo.py new file mode 100644 index 00000000..d0306cb6 --- /dev/null +++ b/src/raven_toolbox/annotation/sbo.py @@ -0,0 +1,180 @@ +"""SBO term assignment. + +Port of the generic core of yeast-GEM's ``code/missingFields/addSBOterms.m``. +Defaults reproduce the yeast-GEM behaviour; overrides parameterise the +pseudo-metabolite / pseudo-reaction name conventions, the transport +detector, and a yeast-specific bug-compat flag. +""" +from __future__ import annotations + +from collections.abc import Callable, Iterable + +import cobra + +DEFAULT_BIOMASS_MET_NAMES: frozenset[str] = frozenset( + {"biomass", "DNA", "RNA", "protein", "carbohydrate", + "lipid", "cofactor", "ion"} +) +DEFAULT_BIOMASS_RXN_NAME: str = "biomass pseudoreaction" +DEFAULT_NGAM_RXN_NAME: str = "non-growth associated maintenance reaction" + + +def add_sbo_terms( + model: cobra.Model, + *, + biomass_met_names: Iterable[str] = DEFAULT_BIOMASS_MET_NAMES, + biomass_met_suffixes: Iterable[str] = (" backbone", " chain"), + biomass_rxn_name: str = DEFAULT_BIOMASS_RXN_NAME, + ngam_rxn_name: str = DEFAULT_NGAM_RXN_NAME, + pseudoreaction_name_substrings: Iterable[str] = ("pseudoreaction", "SLIME rxn"), + transport_detector: Callable[[cobra.Model], set[str]] | None = None, + only_last_reaction_for_pseudo: bool = False, +) -> cobra.Model: + """Assign SBO terms to metabolites and reactions in-place. + + Metabolite SBO assignment + - SBO:0000649 (Biomass) for metabolites whose ``name`` is in + ``biomass_met_names`` or ends with any of + ``biomass_met_suffixes`` (default: ``" backbone"`` / ``" chain"``). + - SBO:0000247 (Simple chemical) otherwise. + + Reaction SBO assignment + - SBO:0000176 (Metabolic reaction) default. + - Single-reactant reactions (exchange / sink / demand): + * extracellular → SBO:0000627 (exchange) + * coefficient < 0 → SBO:0000632 (sink) + * else → SBO:0000628 (demand) + - Transport reactions → SBO:0000655 (default detector: same + metabolite name appearing in ≥ 2 compartments). + - The reaction whose ``name`` matches ``biomass_rxn_name`` → + SBO:0000629. + - The reaction whose ``name`` matches ``ngam_rxn_name`` → + SBO:0000630. + - Other reactions whose ``name`` contains any + ``pseudoreaction_name_substrings`` → SBO:0000395. + + "fill" semantic: SBO is written to ``annotation['sbo']`` only when + that key is missing or empty — mirrors RAVEN's + ``editMiriam(..., 'fill')`` mode. + + Parameters + ---------- + transport_detector + Optional callable that takes the model and returns the set of + reaction ids classified as transport. When ``None``, the default + same-met-name-in-two-compartments heuristic is used. Pass a + custom detector to use a different convention (e.g. RAVEN's + ``getTransportRxns`` via the MATLAB engine). + only_last_reaction_for_pseudo + **Bug-compatibility flag**, off by default. The legacy yeast-GEM + MATLAB ``addSBOterms.m`` contains a typo (``for i = numel(...)`` + rather than ``for i = 1:numel(...)``) that causes the pseudo- + reaction SBO assignments to run only on the very last reaction + in the model. When ``True``, this flag replicates that bug so + callers can migrate without altering the committed model. Leave + ``False`` for new uses. + """ + biomass_met_names = frozenset(biomass_met_names) + biomass_met_suffixes = tuple(biomass_met_suffixes) + pseudoreaction_name_substrings = tuple(pseudoreaction_name_substrings) + if transport_detector is None: + transport_detector = _default_transport_detector + + _assign_metabolite_sbo(model, biomass_met_names, biomass_met_suffixes) + _assign_reaction_sbo( + model, + biomass_rxn_name=biomass_rxn_name, + ngam_rxn_name=ngam_rxn_name, + pseudo_substrings=pseudoreaction_name_substrings, + transport_detector=transport_detector, + only_last_reaction_for_pseudo=only_last_reaction_for_pseudo, + ) + return model + + +# --- internals --------------------------------------------------------- + +def _assign_metabolite_sbo( + model: cobra.Model, + biomass_met_names: frozenset[str], + biomass_met_suffixes: tuple[str, ...], +) -> None: + for met in model.metabolites: + if met.name in biomass_met_names or met.name.endswith(biomass_met_suffixes): + sbo = "SBO:0000649" + else: + sbo = "SBO:0000247" + _fill_sbo(met, sbo) + + +def _assign_reaction_sbo( + model: cobra.Model, + *, + biomass_rxn_name: str, + ngam_rxn_name: str, + pseudo_substrings: tuple[str, ...], + transport_detector: Callable[[cobra.Model], set[str]], + only_last_reaction_for_pseudo: bool, +) -> None: + """Compute the SBO for each reaction via successive overrides, then + fill into the model in a single pass — mirrors the MATLAB + ``rxnSBO``-array + ``editMiriam(..., 'fill')`` two-step. + """ + transport_set = transport_detector(model) + rxns = list(model.reactions) + + sbo_for_rxn: dict[str, str] = {} + for rxn in rxns: + sbo = "SBO:0000176" + if len(rxn.metabolites) == 1: + (met,) = rxn.metabolites + coef = rxn.metabolites[met] + if met.compartment == "e" or model.compartments.get( + met.compartment + ) == "extracellular": + sbo = "SBO:0000627" + elif coef < 0: + sbo = "SBO:0000632" + else: + sbo = "SBO:0000628" + sbo_for_rxn[rxn.id] = sbo + + # Transport override + for rxn_id in transport_set: + sbo_for_rxn[rxn_id] = "SBO:0000655" + + # Pseudoreaction override. The legacy bug-compat flag scopes this + # loop to the last reaction only; the default walks the whole list. + pseudo_targets = [rxns[-1]] if only_last_reaction_for_pseudo else rxns + for rxn in pseudo_targets: + if rxn.name == biomass_rxn_name: + sbo_for_rxn[rxn.id] = "SBO:0000629" + elif rxn.name == ngam_rxn_name: + sbo_for_rxn[rxn.id] = "SBO:0000630" + elif any(s in rxn.name for s in pseudo_substrings): + sbo_for_rxn[rxn.id] = "SBO:0000395" + + for rxn in rxns: + _fill_sbo(rxn, sbo_for_rxn[rxn.id]) + + +def _fill_sbo(entity, sbo: str) -> None: + """Set ``annotation['sbo']`` only if it is missing or empty.""" + if not entity.annotation.get("sbo"): + entity.annotation["sbo"] = sbo + + +def _default_transport_detector(model: cobra.Model) -> set[str]: + """Same-met-name-in-two-compartments heuristic for transport detection. + + Cheap analogue of RAVEN's ``getTransportRxns``. Pass a custom + callable to :func:`add_sbo_terms` to override. + """ + out: set[str] = set() + for rxn in model.reactions: + by_name: dict[str, set[str | None]] = {} + for met in rxn.metabolites: + by_name.setdefault(met.name, set()).add(met.compartment) + if any(len(comps) >= 2 for comps in by_name.values()): + out.add(rxn.id) + return out diff --git a/src/raven_python/binaries.py b/src/raven_toolbox/binaries.py similarity index 73% rename from src/raven_python/binaries.py rename to src/raven_toolbox/binaries.py index 78ee0ac6..1e0cb375 100644 --- a/src/raven_python/binaries.py +++ b/src/raven_toolbox/binaries.py @@ -3,7 +3,7 @@ Shared across tools (not homology-specific). Resolution order for any executable:: explicit path arg → env var (RAVEN_PYTHON_) → shutil.which (PATH) - → ensure_binary (download the version-pinned ZIP from a raven_python release, + → ensure_binary (download the version-pinned ZIP from a raven_toolbox release, verify SHA256, cache, return the path) → FileNotFoundError with install guidance @@ -55,7 +55,7 @@ def platform_key() -> str: def _cache_dir() -> Path: base = os.environ.get("XDG_CACHE_HOME") or (Path.home() / ".cache") - return Path(base) / "raven_python" / "binaries" + return Path(base) / "raven_toolbox" / "binaries" def _bundle_for(executable: str, registry: dict): @@ -65,6 +65,19 @@ def _bundle_for(executable: str, registry: dict): return None, None +def _maybe_autoload(registry: dict) -> None: + """Populate the default registry from ``$RAVEN_PYTHON_MANIFEST`` on first use, if set. + + Only fires when the caller is using the default (still-empty) ``_REGISTRY`` and the + environment variable points at a manifest; a caller that passes its own ``registry`` + is left untouched. The import is local to avoid a cycle with :mod:`raven_toolbox.manifest`. + """ + if registry is _REGISTRY and not registry and os.environ.get("RAVEN_PYTHON_MANIFEST"): + from raven_toolbox import manifest as _manifest + + _manifest.load_into_registries() + + def _sha256(path: Path) -> str: h = hashlib.sha256() with open(path, "rb") as fh: @@ -73,6 +86,29 @@ def _sha256(path: Path) -> str: return h.hexdigest() +def _safe_extract_zip(zf: zipfile.ZipFile, dest_dir: Path) -> None: + """Extract ``zf`` into ``dest_dir``, rejecting members that escape it. + + ``ZipFile.extractall`` has no path-traversal guard (unlike tarfile's + ``filter="data"`` used in reconstruction/kegg/download.py), so a malicious or + corrupt archive could write outside the cache via absolute paths, ``..``, or + symlink members. SHA256 + HTTPS already make a hostile archive unlikely; this + is defence in depth. + """ + dest = dest_dir.resolve() + for info in zf.infolist(): + member = info.filename + target = (dest_dir / member).resolve() + if target != dest and dest not in target.parents: + raise ValueError(f"unsafe path in archive (path traversal): {member!r}") + # Reject symlink members: extractall recreates them as real symlinks whose + # target the path check above never validates, so one could point outside + # dest_dir (or a later member be written through it). + if (info.external_attr >> 16) & 0o170000 == 0o120000: # stat.S_IFLNK + raise ValueError(f"unsafe symlink in archive: {member!r}") + zf.extractall(dest_dir) + + def ensure_binary(executable: str, *, registry: dict | None = None) -> Path: """Download (if needed) and return the path to a bundled ``executable``. @@ -81,6 +117,7 @@ def ensure_binary(executable: str, *, registry: dict | None = None) -> Path: path. Raises ``FileNotFoundError`` if no bundle for this platform is hosted. """ registry = _REGISTRY if registry is None else registry + _maybe_autoload(registry) bundle_name, bundle = _bundle_for(executable, registry) if bundle is None: raise FileNotFoundError( @@ -108,7 +145,7 @@ def ensure_binary(executable: str, *, registry: dict | None = None) -> Path: # that a later run might mistake for a finished one. Mirrors data.py. part = archive.with_suffix(archive.suffix + ".part") try: - with urlopen(entry["url"]) as resp, open(part, "wb") as out: # noqa: S310 + with urlopen(entry["url"], timeout=60) as resp, open(part, "wb") as out: # noqa: S310 shutil.copyfileobj(resp, out) digest = _sha256(part) if digest != entry["sha256"]: @@ -120,7 +157,7 @@ def ensure_binary(executable: str, *, registry: dict | None = None) -> Path: finally: part.unlink(missing_ok=True) with zipfile.ZipFile(archive) as zf: - zf.extractall(dest_dir) + _safe_extract_zip(zf, dest_dir) archive.unlink(missing_ok=True) if not exe.exists(): raise FileNotFoundError(f"{executable!r} not found in the extracted bundle at {dest_dir}.") diff --git a/src/raven_toolbox/biomass/__init__.py b/src/raven_toolbox/biomass/__init__.py new file mode 100644 index 00000000..10eb0d33 --- /dev/null +++ b/src/raven_toolbox/biomass/__init__.py @@ -0,0 +1,50 @@ +"""Biomass equation manipulation — growth-associated maintenance, amino- +acid ratios, component scaling, and biomass-fraction reporting. + +The yeast-GEM port (see yeast-GEM/code/python/PORTING_PLAN.md) was the +first consumer; the API is parameterised by a :class:`BiomassConfig` +so other GEMs can describe their own component layout. + +A typical caller assembles ``BiomassConfig`` once (often from a +project-level YAML) and passes it to every operation: + +.. code-block:: python + + cfg = BiomassConfig( + biomass_rxn="r_4041", + proton_met="s_0794", + components=[ + BiomassComponent("protein", "protein pseudoreaction", + mass_strategy="mw_minus_2h"), + BiomassComponent("carbohydrate", "carbohydrate pseudoreaction"), + BiomassComponent("lipid_backbone", "lipid backbone pseudoreaction", + mass_strategy="grams"), + BiomassComponent("RNA", "RNA pseudoreaction", + mass_strategy="mw_minus_water"), + BiomassComponent("DNA", "DNA pseudoreaction", + mass_strategy="mw_minus_water"), + BiomassComponent("ion", "ion pseudoreaction"), + BiomassComponent("cofactor", "cofactor pseudoreaction"), + ], + ) + fractions = sum_biomass(model, cfg) # {'protein': 0.46, ...} + scale_biomass(model, cfg, "protein", 0.50, balance_out="carbohydrate") + set_gam(model, 80, biomass_rxn=cfg.biomass_rxn, + cofactor_met_names=("ATP", "ADP", "H2O", "H+", "phosphate")) +""" +from raven_toolbox.biomass.config import BiomassComponent, BiomassConfig +from raven_toolbox.biomass.gam import set_gam +from raven_toolbox.biomass.scale import ( + rescale_pseudoreaction, + scale_biomass, + sum_biomass, +) + +__all__ = [ + "BiomassComponent", + "BiomassConfig", + "rescale_pseudoreaction", + "scale_biomass", + "set_gam", + "sum_biomass", +] diff --git a/src/raven_toolbox/biomass/config.py b/src/raven_toolbox/biomass/config.py new file mode 100644 index 00000000..c0ee453a --- /dev/null +++ b/src/raven_toolbox/biomass/config.py @@ -0,0 +1,95 @@ +"""Configuration types for the biomass module.""" +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass, field +from typing import Literal + +#: Strategies for computing a component's mass contribution from the +#: stoichiometry of its pseudoreaction. See :func:`sum_biomass`. +#: +#: ``"mw"`` multiply each substrate coefficient by the +#: molecular weight derived from its +#: ``formula`` (g/mol), divide by 1000 to get +#: g/gDW. Suits carbohydrate / ion / cofactor. +#: ``"mw_minus_2h"`` as ``"mw"`` but with each MW reduced by +#: 2.016 g/mol (two protons released per +#: charged tRNA on protein-pseudoreaction +#: substrates). +#: ``"mw_minus_water"`` as ``"mw"`` but with each MW reduced by +#: 18.015 g/mol (one water released per +#: polymerisation step; RNA / DNA). +#: ``"grams"`` substrate coefficients are already in +#: g/gDW (e.g. the lipid-backbone +#: pseudoreaction). No MW lookup needed. +MassStrategy = Literal["mw", "mw_minus_2h", "mw_minus_water", "grams"] + + +@dataclass(frozen=True) +class BiomassComponent: + """One component of the biomass equation (protein, carbohydrate, …). + + Attributes + ---------- + name + Canonical short name — also the ``model.metabolites`` name of + the metabolite that the pseudoreaction *produces*. Used by + :func:`rescale_pseudoreaction` to identify the product side. + pseudoreaction_name + ``model.reactions[*].name`` of the pseudoreaction. + cobrapy reactions are looked up by ``name`` (not ``id``) + because that is how the yeast-GEM convention names them. + mass_strategy + How to convert the pseudoreaction's substrates into a mass + fraction. See :data:`MassStrategy` for the choices and their + offsets. + """ + + name: str + pseudoreaction_name: str + mass_strategy: MassStrategy = "mw" + + +@dataclass(frozen=True) +class BiomassConfig: + """Container for the per-organism biomass layout. + + Attributes + ---------- + biomass_rxn + ``model.reactions[*].id`` of the top-level biomass + pseudoreaction (the one whose flux is the growth rate). + proton_met + ``model.metabolites[*].id`` of the cytosolic H+ metabolite — + used by :func:`rescale_pseudoreaction` to keep each + pseudoreaction charge-balanced after rescaling. + components + Ordered tuple of :class:`BiomassComponent` describing every + non-top-level pseudoreaction that contributes to mass. + """ + + biomass_rxn: str + proton_met: str + components: tuple[BiomassComponent, ...] = field(default_factory=tuple) + + def __post_init__(self) -> None: + # Freeze the components into a tuple even if a list was passed. + if not isinstance(self.components, tuple): + object.__setattr__(self, "components", tuple(self.components)) + + @classmethod + def from_components( + cls, + biomass_rxn: str, + proton_met: str, + components: Iterable[BiomassComponent], + ) -> BiomassConfig: + return cls(biomass_rxn=biomass_rxn, proton_met=proton_met, + components=tuple(components)) + + def get(self, component_name: str) -> BiomassComponent: + """Look up a component by name (raises ``KeyError`` if missing).""" + for c in self.components: + if c.name == component_name: + return c + raise KeyError(f"BiomassConfig has no component named {component_name!r}") diff --git a/src/raven_toolbox/biomass/gam.py b/src/raven_toolbox/biomass/gam.py new file mode 100644 index 00000000..15a28be2 --- /dev/null +++ b/src/raven_toolbox/biomass/gam.py @@ -0,0 +1,67 @@ +"""Growth-associated maintenance (GAM) and non-growth maintenance (NGAM). + +Ports the generic part of yeast-GEM's ``changeGAM.m``. +""" +from __future__ import annotations + +from collections.abc import Iterable + +import cobra + + +def set_gam( + model: cobra.Model, + value: float, + *, + biomass_rxn: str, + cofactor_met_names: Iterable[str], + ngam_rxn: str | None = None, + ngam_value: float | None = None, +) -> cobra.Model: + """Set GAM (and optionally NGAM) on a model in place. + + Scales every metabolite participating in the biomass pseudoreaction + whose ``name`` is in ``cofactor_met_names`` (e.g. ATP, ADP, H2O, + H+, phosphate) to ``±value``, preserving the sign of its current + coefficient. + + If both ``ngam_rxn`` and ``ngam_value`` are given, the NGAM + reaction's bounds are fixed at ``(ngam_value, ngam_value)``. + + Parameters + ---------- + model + cobra model to mutate. + value + New GAM value (mmol ATP / gDW per growth unit). + biomass_rxn + Reaction id of the biomass pseudoreaction whose stoichiometry + carries the GAM coefficients. + cofactor_met_names + Iterable of metabolite *names* (not ids) — every metabolite in + the biomass pseudoreaction whose name is in this set will be + rescaled. yeast-GEM uses ``{"ATP", "ADP", "H2O", "H+", "phosphate"}``. + ngam_rxn, ngam_value + Optional NGAM update. If both are provided, the reaction's + bounds are set to ``(ngam_value, ngam_value)`` (equality + constraint). + + Returns the (mutated) model for chaining. + """ + rxn = model.reactions.get_by_id(biomass_rxn) + cofactor_set = set(cofactor_met_names) + + deltas: dict[cobra.Metabolite, float] = {} + for met, coef in rxn.metabolites.items(): + if met.name in cofactor_set: + target = (1 if coef > 0 else -1) * float(value) + # combine=True adds delta → new total = target + deltas[met] = target - coef + if deltas: + rxn.add_metabolites(deltas, combine=True) + + if ngam_rxn is not None and ngam_value is not None: + ngam = model.reactions.get_by_id(ngam_rxn) + ngam.bounds = (float(ngam_value), float(ngam_value)) + + return model diff --git a/src/raven_toolbox/biomass/scale.py b/src/raven_toolbox/biomass/scale.py new file mode 100644 index 00000000..fe809b3f --- /dev/null +++ b/src/raven_toolbox/biomass/scale.py @@ -0,0 +1,207 @@ +"""Biomass component summing and rescaling. + +Ports the generic core of yeast-GEM's ``sumBioMass.m`` / +``scaleBioMass.m`` / ``rescalePseudoReaction.m`` into a single +parameterised module that other GEMs can configure via +:class:`BiomassConfig`. +""" +from __future__ import annotations + +import re + +import cobra + +from raven_toolbox.biomass.config import BiomassComponent, BiomassConfig + +# Elemental atomic weights used to translate formulas into g/mol. +# Mirrors the table baked into yeast-GEM's ``sumBioMass.m``; pseudo- +# element "R" is treated as zero mass (placeholder for residues). +_ELEMENT_MASS_G_PER_MOL: dict[str, float] = { + "C": 12.01, "H": 1.008, "N": 14.007, "O": 15.999, + "P": 30.974, "S": 32.06, "R": 0.0, + "Fe": 55.845, "K": 39.098, "Na": 22.99, "Cl": 35.45, + "Mn": 54.938, "Zn": 65.38, "Ca": 40.078, "Mg": 24.305, "Cu": 63.546, +} + +_FORMULA_TOKEN_RE = re.compile(r"([A-Z][a-z]*)(\d*)") + + +def sum_biomass(model: cobra.Model, config: BiomassConfig) -> dict[str, float]: + """Mass-fraction (g/gDW) per biomass component plus the total. + + Mirrors yeast-GEM's ``sumBioMass.m``. Returns a dict keyed by each + :class:`BiomassComponent` name plus ``"total"``. Components whose + pseudoreaction is missing from the model contribute 0 (logged as a + warning). + + Substrate detection: in each component's pseudoreaction, the + substrate side is the metabolites with negative coefficient — the + same convention yeast-GEM uses. + """ + fractions: dict[str, float] = {} + total = 0.0 + for comp in config.components: + f = _component_fraction(model, comp) + fractions[comp.name] = f + total += f + fractions["total"] = total + return fractions + + +def rescale_pseudoreaction( + model: cobra.Model, + config: BiomassConfig, + component_name: str, + factor: float, +) -> None: + """Multiply the substrate coefficients of one component pseudoreaction + by ``factor`` and rebalance H+ to preserve charge neutrality. + + Ports ``rescalePseudoReaction.m``. "Substrate" here means any + metabolite whose ``name`` is not the component's product name; the + component's own metabolite (e.g. ``protein`` for the protein + pseudoreaction) is left untouched. After the rescaling the + coefficient of ``config.proton_met`` is recomputed so the + reaction's total ionic charge sums to zero. + """ + comp = config.get(component_name) + rxn = _find_pseudoreaction(model, comp) + proton_met = model.metabolites.get_by_id(config.proton_met) + + # Step 1: scale substrate coefficients (everything but the component + # product). Use a fresh dict to avoid mutating during iteration. + deltas: dict[cobra.Metabolite, float] = {} + for met, coef in rxn.metabolites.items(): + if met.name == comp.name: + continue # the product — leave alone + deltas[met] = (factor - 1.0) * coef # combine=True adds (factor-1)*coef + if deltas: + rxn.add_metabolites(deltas, combine=True) + + # Step 2: H+ rebalance. The legacy code first zeroes H+, then sets + # it to negate the rxn's current ionic charge. With combine=True we + # express the same with two `_set_coefficient` calls. + _set_coefficient(rxn, proton_met, 0.0) + total_charge = sum( + (m.charge or 0) * coef for m, coef in rxn.metabolites.items() + ) + _set_coefficient(rxn, proton_met, -total_charge) + + +def scale_biomass( + model: cobra.Model, + config: BiomassConfig, + component_name: str, + new_value: float, + *, + balance_out: str | None = None, +) -> None: + """Rescale a component to a new mass fraction, optionally balancing + out a second component to keep the total at 1 g/gDW. + + Ports ``scaleBioMass.m``. The rescaling factor is derived from + :func:`sum_biomass` on the *current* model state, so call this with + an in-place ``model`` mutation, not on a stale snapshot. + """ + fractions = sum_biomass(model, config) + current = fractions[component_name] + if current == 0: + raise ValueError( + f"Cannot scale {component_name!r} to {new_value}: current " + "fraction is 0 (pseudoreaction missing or empty)." + ) + factor = new_value / current + rescale_pseudoreaction(model, config, component_name, factor) + + if balance_out is not None: + # Recompute X after the first rescaling. + fractions = sum_biomass(model, config) + total = fractions["total"] + balance_current = fractions[balance_out] + if balance_current == 0: + return # nothing we can do + balance_factor = (balance_current + (1.0 - total)) / balance_current + rescale_pseudoreaction(model, config, balance_out, balance_factor) + + +# --- internals --------------------------------------------------------- + +def _component_fraction(model: cobra.Model, comp: BiomassComponent) -> float: + """Compute g/gDW of a single component from its pseudoreaction.""" + try: + rxn = _find_pseudoreaction(model, comp) + except KeyError: + return 0.0 # missing pseudoreaction → contributes 0 + substrates = [(m, c) for m, c in rxn.metabolites.items() if c < 0] + if not substrates: + return 0.0 + + if comp.mass_strategy == "grams": + # Stoichiometry already in g/gDW; just sum the (negative) coefs + # and flip sign. + return -sum(c for _m, c in substrates) + + offset = _mw_offset(comp.mass_strategy) + mass_g = 0.0 + for met, coef in substrates: + mw = _formula_mw(met.formula or "") + if mw == 0: + raise ValueError( + f"Metabolite {met.id} ({met.name!r}) has an empty " + "formula; cannot compute mass for biomass summing." + ) + mw_corrected = mw + offset + mass_g += -coef * mw_corrected + return mass_g / 1000.0 # g/mmol → g/gDW (matching yeast-GEM units) + + +def _mw_offset(strategy: str) -> float: + if strategy == "mw": + return 0.0 + if strategy == "mw_minus_2h": + # 2 × 1.008 = 2.016 g/mol per substrate (charged tRNAs release + # two protons on amino acylation). + return -2.016 + if strategy == "mw_minus_water": + # H2O = 18.015 g/mol per polymerisation step (RNA / DNA). + return -18.015 + raise ValueError(f"Unknown mass_strategy: {strategy!r}") + + +def _formula_mw(formula: str) -> float: + """Molecular weight in g/mol from a Hill-style chemical formula.""" + if not formula: + return 0.0 + mw = 0.0 + for element, count_str in _FORMULA_TOKEN_RE.findall(formula): + if not element: + continue + try: + atomic = _ELEMENT_MASS_G_PER_MOL[element] + except KeyError as exc: + raise ValueError( + f"Unknown element {element!r} in formula {formula!r}; " + "extend raven_toolbox.biomass.scale._ELEMENT_MASS_G_PER_MOL " + "if the element is real." + ) from exc + count = int(count_str) if count_str else 1 + mw += atomic * count + return mw + + +def _find_pseudoreaction(model: cobra.Model, comp: BiomassComponent) -> cobra.Reaction: + for rxn in model.reactions: + if rxn.name == comp.pseudoreaction_name: + return rxn + raise KeyError( + f"Component {comp.name!r}: no reaction named " + f"{comp.pseudoreaction_name!r} in model." + ) + + +def _set_coefficient(rxn: cobra.Reaction, met: cobra.Metabolite, value: float) -> None: + """Land on ``S[met, rxn] = value`` via cobra's combine=True API.""" + current = rxn.metabolites.get(met, 0.0) + delta = float(value) - current + if delta != 0: + rxn.add_metabolites({met: delta}, combine=True) diff --git a/src/raven_toolbox/comparison/__init__.py b/src/raven_toolbox/comparison/__init__.py new file mode 100644 index 00000000..81c6c62e --- /dev/null +++ b/src/raven_toolbox/comparison/__init__.py @@ -0,0 +1,23 @@ +"""Model comparison utilities. + +Two flavours: + +* :func:`compare_models` — N-model presence-matrix overview (RAVEN's + ``compareMultipleModels`` analogue). "How do these models relate?" +* :func:`diff_models` — strict two-model semantic-equality diff for CI + gates. "Are these two models the same?" +""" +from raven_toolbox.comparison.compare import ModelComparison, compare_models +from raven_toolbox.comparison.diff import ( + DEFAULT_ANNOTATION_KEYS, + DiffReport, + diff_models, +) + +__all__ = [ + "DEFAULT_ANNOTATION_KEYS", + "DiffReport", + "ModelComparison", + "compare_models", + "diff_models", +] diff --git a/src/raven_python/comparison/compare.py b/src/raven_toolbox/comparison/compare.py similarity index 87% rename from src/raven_python/comparison/compare.py rename to src/raven_toolbox/comparison/compare.py index c7d38a11..549a0b54 100644 --- a/src/raven_python/comparison/compare.py +++ b/src/raven_toolbox/comparison/compare.py @@ -13,11 +13,14 @@ from collections.abc import Iterable from dataclasses import dataclass, field +from pathlib import Path +from typing import cast import cobra import pandas as pd -from raven_python.tasks import Task, check_tasks +from raven_toolbox.tasks import Task, check_tasks +from raven_toolbox.utils.parse import subsystem_to_str @dataclass @@ -60,12 +63,9 @@ def _subsystem_counts(model: cobra.Model) -> dict[str, int]: """{subsystem_name: reaction_count}. Reactions with empty subsystem fall under '(none)'.""" counts: dict[str, int] = {} for r in model.reactions: - # cobra stores subsystem as a string; RAVEN sometimes uses cell-of-cells (we'd - # already have it as a string here, but guard against list/tuple from messy YAML). - sub = r.subsystem - if isinstance(sub, (list, tuple)): - sub = sub[0] if sub else "" - sub = (sub or "").strip() or "(none)" + # cobra stores subsystem as a string; RAVEN sometimes uses cell-of-cells. + # Coerce to the same ;-joined string used everywhere else (no data lost). + sub = subsystem_to_str(r.subsystem).strip() or "(none)" counts[sub] = counts.get(sub, 0) + 1 return counts @@ -92,7 +92,7 @@ def compare_models( """Compare N cobra models on their reactions / metabolites / genes / subsystems (and tasks, if provided). - ``tasks`` is forwarded to :func:`raven_python.tasks.check_tasks` on each model; pass a + ``tasks`` is forwarded to :func:`raven_toolbox.tasks.check_tasks` on each model; pass a file path or a parsed task list. When omitted, ``ModelComparison.tasks`` is ``None``. Models are identified by ``model.id`` (with a fallback to ``model_`` if missing @@ -130,11 +130,12 @@ def compare_models( task_df: pd.DataFrame | None = None failed: dict[str, list[str]] = {} if tasks is not None: - # raven_python.tasks.check_tasks accepts a path or an iterable of Task; preserve task + # raven_toolbox.tasks.check_tasks accepts a path or an iterable of Task; preserve task # ids for the index. Capture the list once so all models test the same set. - from raven_python.tasks.tasklist import parse_task_list - task_list = (parse_task_list(tasks) if isinstance(tasks, (str, bytes)) - or hasattr(tasks, "__fspath__") else list(tasks)) + from raven_toolbox.tasks.tasklist import parse_task_list + task_list = (parse_task_list(cast("str | Path", tasks)) + if isinstance(tasks, (str, bytes)) or hasattr(tasks, "__fspath__") + else list(tasks)) task_ids = [t.id for t in task_list] task_df = pd.DataFrame(False, index=task_ids, columns=model_ids, dtype=bool) for mid, m in zip(model_ids, models_list, strict=True): diff --git a/src/raven_toolbox/comparison/diff.py b/src/raven_toolbox/comparison/diff.py new file mode 100644 index 00000000..44dee758 --- /dev/null +++ b/src/raven_toolbox/comparison/diff.py @@ -0,0 +1,285 @@ +"""Strict two-model semantic-equality diff. + +Complements :func:`raven_toolbox.comparison.compare.compare_models`, which +takes ≥2 models and produces a presence-matrix overview. Where +``compare_models`` answers *"how do these models relate?"*, +:func:`diff_models` answers *"are these two models the same?"* — used as a +CI gate to verify that two toolchains (e.g. MATLAB RAVEN vs raven_toolbox, +or pre/post refactor of one toolchain) produce equivalent models. + +Diff scope: reaction / metabolite / gene id sets, stoichiometry (within +tolerance), bounds, objective coefficients, GPR rules, metabolite +formula/charge/compartment, and a configurable set of annotation keys. +Formatting differences (key ordering, whitespace, float repr) are +explicitly **not** failures. +""" +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass, field + +import cobra + +# Annotation keys checked by default. Add via ``extra_annotations`` or +# remove via ``ignore_annotations`` in :func:`diff_models`. +DEFAULT_ANNOTATION_KEYS: tuple[str, ...] = ( + "sbo", + "ec-code", + "kegg.reaction", + "kegg.compound", + "metanetx.reaction", + "metanetx.chemical", + "chebi", + "bigg.reaction", + "bigg.metabolite", +) + + +@dataclass +class DiffReport: + """Result of :func:`diff_models`. + + Truthy if the models are semantically equal. Iterate ``differences`` + for human-readable messages. + """ + + equal: bool + differences: list[str] = field(default_factory=list) + + def __bool__(self) -> bool: + return self.equal + + def __str__(self) -> str: + if self.equal: + return "Models are semantically equal." + head = f"Models differ ({len(self.differences)} differences):" + body = "\n".join(f" - {d}" for d in self.differences) + return f"{head}\n{body}" + + +def diff_models( + a: cobra.Model, + b: cobra.Model, + *, + stoichiometry_tol: float = 1e-9, + ignore_annotations: Iterable[str] = (), + extra_annotations: Iterable[str] = (), + max_per_category: int = 50, +) -> DiffReport: + """Compare two cobra models for semantic equality. + + Checks: + + * reaction, metabolite and gene id sets (exact) + * stoichiometry per shared reaction (within ``stoichiometry_tol``) + * lower/upper bounds and objective coefficients (exact) + * GPR rules (whitespace- and case-insensitive string comparison) + * metabolite formula, charge, compartment (exact) + * a default set of annotation keys (see ``DEFAULT_ANNOTATION_KEYS``) + plus any ``extra_annotations``, minus any ``ignore_annotations`` + + ``max_per_category`` caps the per-category diff list so a wholesale + mismatch produces a digestible report rather than 10,000 lines. + """ + diffs: list[str] = [] + keys = (set(DEFAULT_ANNOTATION_KEYS) | set(extra_annotations)) - set(ignore_annotations) + + _diff_id_sets(a, b, diffs) + + common_rxns = {r.id for r in a.reactions} & {r.id for r in b.reactions} + _diff_reactions(a, b, sorted(common_rxns), stoichiometry_tol, keys, diffs, max_per_category) + + common_mets = {m.id for m in a.metabolites} & {m.id for m in b.metabolites} + _diff_metabolites(a, b, sorted(common_mets), keys, diffs, max_per_category) + + return DiffReport(equal=not diffs, differences=diffs) + + +# --- internals --------------------------------------------------------- + +def _diff_id_sets(a: cobra.Model, b: cobra.Model, diffs: list[str]) -> None: + for label, get in ( + ("reactions", lambda m: m.reactions), + ("metabolites", lambda m: m.metabolites), + ("genes", lambda m: m.genes), + ): + a_ids = {x.id for x in get(a)} + b_ids = {x.id for x in get(b)} + only_a = sorted(a_ids - b_ids) + only_b = sorted(b_ids - a_ids) + if only_a: + diffs.append(f"{label} only in A ({len(only_a)}): {only_a[:10]}{_more(only_a, 10)}") + if only_b: + diffs.append(f"{label} only in B ({len(only_b)}): {only_b[:10]}{_more(only_b, 10)}") + + +def _diff_reactions( + a: cobra.Model, + b: cobra.Model, + common: list[str], + tol: float, + anno_keys: set[str], + diffs: list[str], + cap: int, +) -> None: + counters = {"stoich": 0, "bounds": 0, "objective": 0, "gpr": 0, "anno": 0} + for rxn_id in common: + ra = a.reactions.get_by_id(rxn_id) + rb = b.reactions.get_by_id(rxn_id) + + sa = {m.id: c for m, c in ra.metabolites.items()} + sb = {m.id: c for m, c in rb.metabolites.items()} + if set(sa) != set(sb): + _push(diffs, counters, "stoich", cap, f"{rxn_id}: stoichiometry has different mets") + else: + for mid in sa: + if abs(sa[mid] - sb[mid]) > tol: + _push( + diffs, counters, "stoich", cap, + f"{rxn_id}: coef[{mid}] A={sa[mid]:.6g} B={sb[mid]:.6g}", + ) + + if ra.lower_bound != rb.lower_bound or ra.upper_bound != rb.upper_bound: + _push( + diffs, counters, "bounds", cap, + f"{rxn_id}: bounds A=({ra.lower_bound}, {ra.upper_bound}) " + f"B=({rb.lower_bound}, {rb.upper_bound})", + ) + + if ra.objective_coefficient != rb.objective_coefficient: + _push( + diffs, counters, "objective", cap, + f"{rxn_id}: objective A={ra.objective_coefficient} B={rb.objective_coefficient}", + ) + + ga = _normalise_gpr(ra.gene_reaction_rule) + gb = _normalise_gpr(rb.gene_reaction_rule) + if ga != gb: + _push(diffs, counters, "gpr", cap, f"{rxn_id}: GPR A={ga!r} B={gb!r}") + + _diff_annotations( + f"rxn {rxn_id}", ra.annotation, rb.annotation, + anno_keys, diffs, counters, cap, + ) + + +def _diff_metabolites( + a: cobra.Model, + b: cobra.Model, + common: list[str], + anno_keys: set[str], + diffs: list[str], + cap: int, +) -> None: + counters = {"formula": 0, "charge": 0, "compartment": 0, "anno": 0} + for met_id in common: + ma = a.metabolites.get_by_id(met_id) + mb = b.metabolites.get_by_id(met_id) + + if (ma.formula or None) != (mb.formula or None): + _push( + diffs, counters, "formula", cap, + f"met {met_id}: formula A={ma.formula} B={mb.formula}", + ) + + ca = ma.charge if ma.charge is not None else 0 + cb = mb.charge if mb.charge is not None else 0 + if ca != cb: + _push( + diffs, counters, "charge", cap, + f"met {met_id}: charge A={ma.charge} B={mb.charge}", + ) + + if ma.compartment != mb.compartment: + _push( + diffs, counters, "compartment", cap, + f"met {met_id}: compartment A={ma.compartment} B={mb.compartment}", + ) + + _diff_annotations( + f"met {met_id}", ma.annotation, mb.annotation, + anno_keys, diffs, counters, cap, + ) + + +def _diff_annotations( + label: str, + aa: dict, + ba: dict, + keys: set[str], + diffs: list[str], + counters: dict, + cap: int, +) -> None: + for k in keys: + va = _normalise_annotation_value(aa.get(k)) + vb = _normalise_annotation_value(ba.get(k)) + if va != vb: + _push(diffs, counters, "anno", cap, f"{label}.annotation[{k!r}]: A={va} B={vb}") + + +def _normalise_gpr(rule: str) -> str: + """Lowercase + collapse whitespace. + + A more robust comparator would parse to a GPR AST and compare + structures; this is the cheap heuristic that catches the formatting + drift we see between different SBML writers. + """ + if not rule: + return "" + return " ".join(rule.lower().split()) + + +def _normalise_annotation_value(v): + if v is None: + return None + if isinstance(v, list): + return tuple(sorted(str(x) for x in v)) + if isinstance(v, tuple): + return tuple(sorted(str(x) for x in v)) + return str(v) + + +def _push(diffs: list[str], counters: dict, key: str, cap: int, msg: str) -> None: + counters[key] = counters.get(key, 0) + 1 + if counters[key] <= cap: + diffs.append(msg) + elif counters[key] == cap + 1: + diffs.append(f"... ({key} category truncated at {cap} entries)") + + +def _more(seq: list, shown: int) -> str: + return "" if len(seq) <= shown else f" ... (+{len(seq) - shown} more)" + + +# --- CLI --------------------------------------------------------------- + +def _main() -> int: + import argparse + + from cobra.io import read_sbml_model + + parser = argparse.ArgumentParser( + description="Diff two SBML models for semantic equality." + ) + parser.add_argument("a", help="path to first model (SBML)") + parser.add_argument("b", help="path to second model (SBML)") + parser.add_argument( + "--stoichiometry-tol", + type=float, + default=1e-9, + help="absolute tolerance on stoichiometric coefficients (default 1e-9)", + ) + args = parser.parse_args() + + report = diff_models( + read_sbml_model(args.a), + read_sbml_model(args.b), + stoichiometry_tol=args.stoichiometry_tol, + ) + print(report) + return 0 if report.equal else 1 + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(_main()) diff --git a/src/raven_toolbox/conditions/__init__.py b/src/raven_toolbox/conditions/__init__.py new file mode 100644 index 00000000..008fd6aa --- /dev/null +++ b/src/raven_toolbox/conditions/__init__.py @@ -0,0 +1,25 @@ +"""Apply named "condition" presets — bound diffs, biomass edits, etc. + +A *condition* is a YAML file (or pre-parsed dict) describing a set of +deterministic modifications to apply to a cobra model: a prelude that +resets exchange bounds, edits to a cofactor pseudoreaction +(metabolite removals + automatic charge rebalancing), a per-reaction +bounds diff, and an optional biomass-stoichiometry delta. + +Yeast-GEM was the first consumer; the schema is documented in +:func:`apply_condition` and is meant to be reusable for any GEM that +wants to keep its condition presets as data rather than code. +""" +from raven_toolbox.conditions.apply import ( + DEFAULT_RESET_EXCHANGES_UPPER_BOUND, + apply_condition, + load_condition, + set_reaction_bounds, +) + +__all__ = [ + "DEFAULT_RESET_EXCHANGES_UPPER_BOUND", + "apply_condition", + "load_condition", + "set_reaction_bounds", +] diff --git a/src/raven_toolbox/conditions/apply.py b/src/raven_toolbox/conditions/apply.py new file mode 100644 index 00000000..85c72a86 --- /dev/null +++ b/src/raven_toolbox/conditions/apply.py @@ -0,0 +1,192 @@ +"""Generic "apply this YAML condition" loader. + +Schema (all keys optional): + +.. code-block:: yaml + + name: my_condition + description: free-form text. + + # Set every exchange reaction to (0, ub) before any per-rxn override. + prelude: + reset_exchanges: out # str | bool truthy + + # Remove metabolites from a pseudoreaction and rebalance H+ + # (or any other "balance" met) so total charge stays zero. + cofactor_pseudoreaction: + rxn_id: r_4598 + remove_mets: + - { met: s_3714 } # comment field is informational + charge_balance_met: s_0794 # optional + + # Add (combine=True) to a reaction's stoichiometry. + biomass_stoichiometry_delta: + rxn_id: r_4041 + add: + - { met: s_0689, coef: 0.08 } + - { met: s_0687, coef: -0.08 } + - { met: s_0794, coef: -0.16 } + + # Per-reaction bounds. lb / ub omitted means "leave unchanged". + bounds: + - { rxn: r_1654, lb: -1000 } + - { rxn: r_1992, lb: 0 } + - { rxn: r_1663, lb: 0, ub: 0 } + + # Sanity-check counter. Emits a warning if the actual count of + # ``lb == -1000`` bound entries doesn't match. + expected_uptake_count: 15 + +The schema is intentionally narrow: deterministic, idempotent, easy to +diff in code review. Project-specific extensions (e.g. yeast-GEM's +``amino_acid_ratio``) are handled by the *caller* before / after this +function — keep the upstream generic. +""" +from __future__ import annotations + +import warnings +from pathlib import Path +from typing import Any + +import cobra +from ruamel.yaml import YAML + +# A safe loader keeps the parsed document as plain dict / list / scalars, +# which matches what callers expect from ``load_condition``. ruamel.yaml +# is already a transitive dependency via cobra, so we don't take on +# PyYAML on top. +_SAFE_YAML = YAML(typ="safe") + +#: ``prelude.reset_exchanges`` puts every exchange reaction at this +#: upper bound (and lower bound = 0). +DEFAULT_RESET_EXCHANGES_UPPER_BOUND: float = 1000.0 + + +def load_condition(path: str | Path) -> dict[str, Any]: + """Parse a condition YAML file into a plain dict.""" + path = Path(path) + if not path.exists(): + raise FileNotFoundError(f"Condition file not found: {path}") + with open(path) as f: + return _SAFE_YAML.load(f) + + +def apply_condition( + model: cobra.Model, + condition: dict[str, Any] | str | Path, +) -> cobra.Model: + """Apply a parsed condition (or a YAML file path) to ``model`` in place. + + Returns the same model object for chaining. + """ + cfg = condition if isinstance(condition, dict) else load_condition(condition) + + _apply_prelude(model, cfg) + _apply_cofactor_pseudoreaction(model, cfg) + _apply_biomass_stoichiometry_delta(model, cfg) + n_uptake = _apply_bounds(model, cfg) + _check_uptake_count(cfg, n_uptake) + + return model + + +def set_reaction_bounds(rxn: cobra.Reaction, lb: float, ub: float) -> None: + """Set both bounds, bypassing cobra's ``lb <= ub`` validator. + + Some conditions intentionally land on an infeasible ``lb > ub`` state + (e.g. forcing flux via a sentinel bound). cobra's setter rejects + that. This helper writes the underlying private attributes when + needed so the model can match the caller's exact intent. + """ + if lb > ub: + rxn._lower_bound = lb # noqa: SLF001 — intentional bypass + rxn._upper_bound = ub # noqa: SLF001 — intentional bypass + else: + rxn.bounds = (lb, ub) + + +# --- step implementations --------------------------------------------- + +def _apply_prelude(model: cobra.Model, cfg: dict[str, Any]) -> None: + prelude = cfg.get("prelude") or {} + if prelude.get("reset_exchanges"): + # cobrapy doesn't distinguish RAVEN's in / out directions; we + # reset every exchange reaction. + for rxn in model.exchanges: + rxn.lower_bound = 0 + rxn.upper_bound = DEFAULT_RESET_EXCHANGES_UPPER_BOUND + + +def _apply_cofactor_pseudoreaction(model: cobra.Model, cfg: dict[str, Any]) -> None: + cp = cfg.get("cofactor_pseudoreaction") + if not cp: + return + rxn = model.reactions.get_by_id(cp["rxn_id"]) + for entry in cp.get("remove_mets", []): + met = model.metabolites.get_by_id(entry["met"]) + _set_coefficient(rxn, met, 0.0) + balance_met_id = cp.get("charge_balance_met") + if balance_met_id: + balance_met = model.metabolites.get_by_id(balance_met_id) + _set_coefficient(rxn, balance_met, 0.0) + total_charge = sum( + (m.charge or 0) * coef + for m, coef in rxn.metabolites.items() + ) + _set_coefficient(rxn, balance_met, -total_charge) + + +def _apply_biomass_stoichiometry_delta(model: cobra.Model, cfg: dict[str, Any]) -> None: + delta = cfg.get("biomass_stoichiometry_delta") + if not delta: + return + rxn = model.reactions.get_by_id(delta["rxn_id"]) + for entry in delta.get("add", []): + met = model.metabolites.get_by_id(entry["met"]) + rxn.add_metabolites({met: float(entry["coef"])}, combine=True) + + +def _apply_bounds(model: cobra.Model, cfg: dict[str, Any]) -> int: + n_uptake = 0 + for entry in cfg.get("bounds", []): + try: + rxn = model.reactions.get_by_id(entry["rxn"]) + except KeyError: + warnings.warn( + f"Reaction {entry['rxn']!r} not found in model; skipping.", + stacklevel=3, + ) + continue + new_lb = float(entry["lb"]) if "lb" in entry else rxn.lower_bound + new_ub = float(entry["ub"]) if "ub" in entry else rxn.upper_bound + set_reaction_bounds(rxn, new_lb, new_ub) + if entry.get("lb") == -1000: + n_uptake += 1 + return n_uptake + + +def _check_uptake_count(cfg: dict[str, Any], n_uptake: int) -> None: + expected = cfg.get("expected_uptake_count") + if expected is None: + return + if n_uptake != expected: + warnings.warn( + f"Expected {expected} uptake reactions, applied {n_uptake}. " + "Some referenced reactions may be missing from the model.", + stacklevel=3, + ) + + +# --- helpers ---------------------------------------------------------- + +def _set_coefficient(rxn: cobra.Reaction, met: cobra.Metabolite, value: float) -> None: + """Set the stoichiometric coefficient of ``met`` in ``rxn`` to ``value``. + + cobra's only public mutation point is ``add_metabolites``; we go via + ``combine=True`` with ``(value - current)`` to land on the desired + value without depending on cobra's removal-on-zero behaviour. + """ + current = rxn.metabolites.get(met, 0.0) + delta = float(value) - current + if delta != 0: + rxn.add_metabolites({met: delta}, combine=True) diff --git a/src/raven_toolbox/curation/__init__.py b/src/raven_toolbox/curation/__init__.py new file mode 100644 index 00000000..a8ed8658 --- /dev/null +++ b/src/raven_toolbox/curation/__init__.py @@ -0,0 +1,54 @@ +"""Batch curation of metabolites / reactions / genes from data tables. + +Port of yeast-GEM's MATLAB ``curateMetsRxnsGenes`` into a generic +DataFrame-driven engine. Other GEM projects (Human-GEM, custom +reconstructions, …) can use the same machinery with their own TSV +layouts; the only required pieces are the data tables and (optionally) +project-specific id prefixes for fresh metabolites and reactions. + +Public API: + +* :func:`batch_curate` — entrypoint taking pandas DataFrames. +* :func:`batch_curate_from_tsv` — file-path convenience wrapper. +* :class:`CurationResult` — record of what was added / updated. + +Schema (mirrors yeast-GEM's ``data/modelCuration/template/`` layout): + +- **mets_df**: ``metNames, comps, formula, charge, inchi, metNotes`` + + any number of MIRIAM-annotation columns. Match key is + ``(name, comp)``. +- **genes_df**: ``genes, geneShortNames`` + MIRIAM columns. Match key + is ``genes``. +- **rxns_df**: ``rxnNames, grRules, lb, ub, rev, subSystems, eccodes, + rxnNotes, rxnReferences, rxnConfidenceScores`` + MIRIAM columns. + Match key is the reaction's *stoichiometry* — same metabolites and + coefficients ⇒ same reaction. +- **rxns_coeffs_df**: ``rxnNames, metNames, comps, coefficient``. One + row per ``(reaction, metabolite)`` pair. The ``rxnNames`` column + links each coefficient back to a row in ``rxns_df``. An optional + ``index`` first column from the legacy yeast-GEM schema is silently + ignored. + +Everything after the core columns in any of the four tables is +interpreted as a MIRIAM annotation: the column header becomes the +namespace key (``met.annotation[
] = ``). +""" +from raven_toolbox.curation.batch import ( + DEFAULT_CORE_GENE_COLUMNS, + DEFAULT_CORE_MET_COLUMNS, + DEFAULT_CORE_RXN_COEFFS_COLUMNS, + DEFAULT_CORE_RXN_COLUMNS, + CurationResult, + batch_curate, + batch_curate_from_tsv, +) + +__all__ = [ + "DEFAULT_CORE_GENE_COLUMNS", + "DEFAULT_CORE_MET_COLUMNS", + "DEFAULT_CORE_RXN_COEFFS_COLUMNS", + "DEFAULT_CORE_RXN_COLUMNS", + "CurationResult", + "batch_curate", + "batch_curate_from_tsv", +] diff --git a/src/raven_toolbox/curation/batch.py b/src/raven_toolbox/curation/batch.py new file mode 100644 index 00000000..50043494 --- /dev/null +++ b/src/raven_toolbox/curation/batch.py @@ -0,0 +1,460 @@ +"""Generic batch-curation engine driven by tabular data.""" +from __future__ import annotations + +import warnings +from dataclasses import dataclass, field +from pathlib import Path + +import cobra +import pandas as pd + +from raven_toolbox.utils.parse import subsystem_to_str + +#: Core columns recognised in ``mets_df``. Anything else is treated as a +#: MIRIAM annotation column (the header becomes the namespace key). +DEFAULT_CORE_MET_COLUMNS: tuple[str, ...] = ( + "metNames", "comps", "formula", "charge", "inchi", "metNotes", +) +#: Core columns recognised in ``genes_df``. +DEFAULT_CORE_GENE_COLUMNS: tuple[str, ...] = ("genes", "geneShortNames") +#: Core columns recognised in ``rxns_df``. +DEFAULT_CORE_RXN_COLUMNS: tuple[str, ...] = ( + "rxnNames", "grRules", "lb", "ub", "rev", + "subSystems", "eccodes", "rxnNotes", "rxnReferences", "rxnConfidenceScores", +) +#: Required columns in ``rxns_coeffs_df``. (Linkage column ``rxnNames`` +#: + one row per ``(reaction, metabolite)`` pair.) A leading ``index`` +#: column from the legacy yeast-GEM schema is silently ignored. +DEFAULT_CORE_RXN_COEFFS_COLUMNS: tuple[str, ...] = ( + "rxnNames", "metNames", "comps", "coefficient", +) + + +@dataclass +class CurationResult: + """Record of what :func:`batch_curate` added / updated. + + Each list holds the cobra ids of the affected entities, in the + order they were processed. + """ + + added_metabolites: list[str] = field(default_factory=list) + updated_metabolites: list[str] = field(default_factory=list) + added_genes: list[str] = field(default_factory=list) + updated_genes: list[str] = field(default_factory=list) + added_reactions: list[str] = field(default_factory=list) + updated_reactions: list[str] = field(default_factory=list) + + def __bool__(self) -> bool: + return bool( + self.added_metabolites or self.updated_metabolites + or self.added_genes or self.updated_genes + or self.added_reactions or self.updated_reactions + ) + + +# --- public entry points ---------------------------------------------- + +def batch_curate( + model: cobra.Model, + *, + mets_df: pd.DataFrame | None = None, + genes_df: pd.DataFrame | None = None, + rxns_df: pd.DataFrame | None = None, + rxns_coeffs_df: pd.DataFrame | None = None, + met_id_prefix: str = "M_", + rxn_id_prefix: str = "R_", +) -> CurationResult: + """Add or update metabolites / reactions / genes from data tables. + + Each table is optional. ``rxns_df`` and ``rxns_coeffs_df`` must be + provided together (one describes the per-reaction attributes, the + other carries the stoichiometric coefficients). + + Parameters + ---------- + model + Model to curate in place. + mets_df, genes_df, rxns_df, rxns_coeffs_df + Tables matching the schema described in + :mod:`raven_toolbox.curation`. + met_id_prefix, rxn_id_prefix + Prefixes for freshly-generated metabolite / reaction ids + (e.g. ``s_`` and ``r_`` for yeast-GEM, ``M_`` and ``R_`` for + the cobrapy / BiGG default). New entity ids are formed by + finding the largest existing zero-padded suffix matching the + prefix and incrementing from there. + + Returns + ------- + A :class:`CurationResult` summarising the changes. + """ + result = CurationResult() + + if mets_df is not None: + _apply_mets(model, mets_df, met_id_prefix, result) + if genes_df is not None: + _apply_genes(model, genes_df, result) + + if (rxns_df is None) != (rxns_coeffs_df is None): + raise ValueError( + "rxns_df and rxns_coeffs_df must be provided together; got " + f"rxns_df={'set' if rxns_df is not None else 'None'}, " + f"rxns_coeffs_df=" + f"{'set' if rxns_coeffs_df is not None else 'None'}." + ) + if rxns_df is not None: + _apply_reactions(model, rxns_df, rxns_coeffs_df, rxn_id_prefix, result) + return result + + +def batch_curate_from_tsv( + model: cobra.Model, + *, + mets_tsv: str | Path | None = None, + genes_tsv: str | Path | None = None, + rxns_tsv: str | Path | None = None, + rxns_coeffs_tsv: str | Path | None = None, + met_id_prefix: str = "M_", + rxn_id_prefix: str = "R_", +) -> CurationResult: + """File-path convenience wrapper for :func:`batch_curate`. + + Each path is optional. TSVs are read with pandas' default + ``read_csv(sep='\\t')``; empty cells become ``NaN`` (not the + empty string), which the engine treats as "skip this field". + """ + def _read(path): + return pd.read_csv(path, sep="\t") if path is not None else None + + return batch_curate( + model, + mets_df=_read(mets_tsv), + genes_df=_read(genes_tsv), + rxns_df=_read(rxns_tsv), + rxns_coeffs_df=_read(rxns_coeffs_tsv), + met_id_prefix=met_id_prefix, + rxn_id_prefix=rxn_id_prefix, + ) + + +# --- metabolites ------------------------------------------------------ + +def _apply_mets( + model: cobra.Model, + df: pd.DataFrame, + id_prefix: str, + result: CurationResult, +) -> None: + miriam_cols = [c for c in df.columns if c not in DEFAULT_CORE_MET_COLUMNS] + name_index = _name_compartment_index(model) + + new_rows: list[pd.Series] = [] + for _, row in df.iterrows(): + name = row["metNames"] + comp = row["comps"] + existing = name_index.get((name, comp)) + if existing is not None: + _update_metabolite(existing, row, miriam_cols) + result.updated_metabolites.append(existing.id) + else: + new_rows.append(row) + + if new_rows: + new_mets = _build_new_metabolites(model, new_rows, miriam_cols, id_prefix) + model.add_metabolites(new_mets) + result.added_metabolites.extend(m.id for m in new_mets) + + if result.updated_metabolites: + warnings.warn( + f"{len(result.updated_metabolites)} metabolite(s) already " + "existed and were overwritten: " + f"{result.updated_metabolites[:5]}" + f"{'...' if len(result.updated_metabolites) > 5 else ''}", + stacklevel=4, + ) + + +def _name_compartment_index(model: cobra.Model) -> dict[tuple[str, str], cobra.Metabolite]: + return {(m.name, m.compartment): m for m in model.metabolites} + + +def _update_metabolite(met: cobra.Metabolite, row: pd.Series, miriam_cols: list[str]) -> None: + if _has_value(row.get("formula")): + met.formula = str(row["formula"]) + if _has_value(row.get("charge")): + met.charge = _coerce_int(row["charge"]) + if _has_value(row.get("inchi")): + met.annotation["inchi"] = str(row["inchi"]) + if _has_value(row.get("metNotes")): + met.notes["metNotes"] = str(row["metNotes"]) + _apply_miriam(met, row, miriam_cols) + + +def _build_new_metabolites( + model: cobra.Model, + rows: list[pd.Series], + miriam_cols: list[str], + id_prefix: str, +) -> list[cobra.Metabolite]: + ids = _generate_ids(model.metabolites, id_prefix, len(rows)) + new_mets: list[cobra.Metabolite] = [] + for new_id, row in zip(ids, rows, strict=True): + met = cobra.Metabolite( + id=new_id, + name=str(row["metNames"]), + compartment=str(row["comps"]), + ) + if _has_value(row.get("formula")): + met.formula = str(row["formula"]) + if _has_value(row.get("charge")): + met.charge = _coerce_int(row["charge"]) + if _has_value(row.get("inchi")): + met.annotation["inchi"] = str(row["inchi"]) + if _has_value(row.get("metNotes")): + met.notes["metNotes"] = str(row["metNotes"]) + _apply_miriam(met, row, miriam_cols) + new_mets.append(met) + return new_mets + + +# --- genes ------------------------------------------------------------ + +def _apply_genes(model: cobra.Model, df: pd.DataFrame, result: CurationResult) -> None: + miriam_cols = [c for c in df.columns if c not in DEFAULT_CORE_GENE_COLUMNS] + existing_genes = {g.id: g for g in model.genes} + + new_rows: list[pd.Series] = [] + for _, row in df.iterrows(): + gid = str(row["genes"]) + existing = existing_genes.get(gid) + if existing is not None: + _update_gene(existing, row, miriam_cols) + result.updated_genes.append(gid) + else: + new_rows.append(row) + + if new_rows: + # cobrapy doesn't have a direct "add a free-standing gene" API; + # genes are typically added via reactions. Use cobra.Gene + an + # explicit registration through the DictList. + from cobra.core.gene import Gene + + for row in new_rows: + g = Gene(str(row["genes"])) + if _has_value(row.get("geneShortNames")): + g.name = str(row["geneShortNames"]) + _apply_miriam(g, row, miriam_cols) + model.genes.append(g) + result.added_genes.append(g.id) + + if result.updated_genes: + warnings.warn( + f"{len(result.updated_genes)} gene(s) already existed and " + f"were overwritten: {result.updated_genes[:5]}" + f"{'...' if len(result.updated_genes) > 5 else ''}", + stacklevel=4, + ) + + +def _update_gene(gene, row: pd.Series, miriam_cols: list[str]) -> None: + if _has_value(row.get("geneShortNames")): + gene.name = str(row["geneShortNames"]) + _apply_miriam(gene, row, miriam_cols) + + +# --- reactions -------------------------------------------------------- + +def _apply_reactions( + model: cobra.Model, + rxns_df: pd.DataFrame, + coeffs_df: pd.DataFrame, + id_prefix: str, + result: CurationResult, +) -> None: + miriam_cols = [c for c in rxns_df.columns if c not in DEFAULT_CORE_RXN_COLUMNS] + _validate_rxns_coeffs(coeffs_df) + coeffs_by_name = _group_coeffs_by_rxn_name(model, coeffs_df) + + # Build {stoichiometry_signature: existing_reaction} for the lookup + # of "is this reaction already in the model?". + by_signature = { + _stoich_signature(r): r for r in model.reactions + } + + new_rows: list[tuple[pd.Series, dict[cobra.Metabolite, float]]] = [] + for _, row in rxns_df.iterrows(): + rxn_name = row["rxnNames"] + if rxn_name not in coeffs_by_name: + raise ValueError( + f"Reaction {rxn_name!r} in rxns_df has no matching " + "row(s) in rxns_coeffs_df." + ) + stoich = coeffs_by_name[rxn_name] + sig = _stoich_signature_from_dict(stoich) + existing = by_signature.get(sig) + if existing is not None: + _update_reaction(existing, row, miriam_cols) + result.updated_reactions.append(existing.id) + else: + new_rows.append((row, stoich)) + + if new_rows: + new_rxns = _build_new_reactions(model, new_rows, miriam_cols, id_prefix) + model.add_reactions(new_rxns) + result.added_reactions.extend(r.id for r in new_rxns) + + if result.updated_reactions: + warnings.warn( + f"{len(result.updated_reactions)} reaction(s) had matching " + "stoichiometry in the model and were overwritten: " + f"{result.updated_reactions[:5]}" + f"{'...' if len(result.updated_reactions) > 5 else ''}", + stacklevel=4, + ) + + +def _validate_rxns_coeffs(df: pd.DataFrame) -> None: + missing = set(DEFAULT_CORE_RXN_COEFFS_COLUMNS) - set(df.columns) + if missing: + raise ValueError( + f"rxns_coeffs_df is missing required columns: {sorted(missing)}" + ) + + +def _group_coeffs_by_rxn_name( + model: cobra.Model, + coeffs_df: pd.DataFrame, +) -> dict[str, dict[cobra.Metabolite, float]]: + """Group coefficient rows by rxn name and resolve the metabolites. + + Each value is a ``{metabolite_object: coefficient}`` dict ready for + ``cobra.Reaction.add_metabolites``. Metabolites are looked up by + ``(name, comp)``; if a coefficient references an unknown + metabolite, that's an error (the caller should add the metabolite + via ``mets_df`` first). + """ + name_index = _name_compartment_index(model) + grouped: dict[str, dict[cobra.Metabolite, float]] = {} + for _, row in coeffs_df.iterrows(): + rxn_name = str(row["rxnNames"]) + met_name = str(row["metNames"]) + comp = str(row["comps"]) + coef = float(row["coefficient"]) + met = name_index.get((met_name, comp)) + if met is None: + raise ValueError( + f"Reaction {rxn_name!r} references metabolite " + f"{met_name!r}[{comp}] which is not in the model. " + "Add it via mets_df first, or include it in the same " + "batch_curate call." + ) + grouped.setdefault(rxn_name, {})[met] = coef + return grouped + + +def _stoich_signature(rxn: cobra.Reaction) -> frozenset: + return frozenset((m.id, c) for m, c in rxn.metabolites.items()) + + +def _stoich_signature_from_dict(stoich: dict[cobra.Metabolite, float]) -> frozenset: + return frozenset((m.id, c) for m, c in stoich.items()) + + +def _update_reaction(rxn: cobra.Reaction, row: pd.Series, miriam_cols: list[str]) -> None: + if _has_value(row.get("rxnNames")): + rxn.name = str(row["rxnNames"]) + if _has_value(row.get("grRules")): + rxn.gene_reaction_rule = str(row["grRules"]) + if _has_value(row.get("lb")): + rxn.lower_bound = float(row["lb"]) + if _has_value(row.get("ub")): + rxn.upper_bound = float(row["ub"]) + if _has_value(row.get("subSystems")): + rxn.subsystem = subsystem_to_str(row["subSystems"]) + if _has_value(row.get("eccodes")): + rxn.annotation["ec-code"] = str(row["eccodes"]) + if _has_value(row.get("rxnNotes")): + rxn.notes["rxnNotes"] = str(row["rxnNotes"]) + if _has_value(row.get("rxnReferences")): + rxn.notes["rxnReferences"] = str(row["rxnReferences"]) + if _has_value(row.get("rxnConfidenceScores")): + rxn.notes["rxnConfidenceScores"] = str(row["rxnConfidenceScores"]) + _apply_miriam(rxn, row, miriam_cols) + + +def _build_new_reactions( + model: cobra.Model, + rows: list[tuple[pd.Series, dict[cobra.Metabolite, float]]], + miriam_cols: list[str], + id_prefix: str, +) -> list[cobra.Reaction]: + ids = _generate_ids(model.reactions, id_prefix, len(rows)) + new_rxns: list[cobra.Reaction] = [] + for new_id, (row, stoich) in zip(ids, rows, strict=True): + rxn = cobra.Reaction( + id=new_id, + name=str(row["rxnNames"]), + lower_bound=float(row["lb"]) if _has_value(row.get("lb")) else -1000.0, + upper_bound=float(row["ub"]) if _has_value(row.get("ub")) else 1000.0, + ) + rxn.add_metabolites(stoich) + if _has_value(row.get("grRules")): + rxn.gene_reaction_rule = str(row["grRules"]) + if _has_value(row.get("subSystems")): + rxn.subsystem = subsystem_to_str(row["subSystems"]) + if _has_value(row.get("eccodes")): + rxn.annotation["ec-code"] = str(row["eccodes"]) + if _has_value(row.get("rxnNotes")): + rxn.notes["rxnNotes"] = str(row["rxnNotes"]) + if _has_value(row.get("rxnReferences")): + rxn.notes["rxnReferences"] = str(row["rxnReferences"]) + if _has_value(row.get("rxnConfidenceScores")): + rxn.notes["rxnConfidenceScores"] = str(row["rxnConfidenceScores"]) + _apply_miriam(rxn, row, miriam_cols) + new_rxns.append(rxn) + return new_rxns + + +# --- shared helpers --------------------------------------------------- + +def _apply_miriam(entity, row: pd.Series, miriam_cols: list[str]) -> None: + for col in miriam_cols: + value = row.get(col) + if _has_value(value): + entity.annotation[col] = str(value).strip() + + +def _has_value(v) -> bool: + """Return True if ``v`` is a non-empty, non-NaN cell value.""" + if v is None: + return False + if isinstance(v, float) and v != v: # NaN + return False + if isinstance(v, str) and v.strip() == "": + return False + return True + + +def _coerce_int(v) -> int: + """Pandas reads integer-like strings as floats. Recover the int.""" + return int(round(float(v))) + + +def _generate_ids(existing, prefix: str, count: int) -> list[str]: + """Generate ``count`` fresh ids by incrementing from the largest + existing zero-padded suffix matching ``prefix``. + + e.g. with prefix ``s_`` and existing ``s_4100``, the next ids are + ``s_4101``, ``s_4102``, … Width is preserved. + """ + max_n = 0 + width = 1 + for entity in existing: + if not entity.id.startswith(prefix): + continue + suffix = entity.id[len(prefix):] + if suffix.isdigit(): + max_n = max(max_n, int(suffix)) + width = max(width, len(suffix)) + return [f"{prefix}{(max_n + i):0{width}d}" for i in range(1, count + 1)] diff --git a/src/raven_toolbox/data.py b/src/raven_toolbox/data.py new file mode 100644 index 00000000..36e6bcd2 --- /dev/null +++ b/src/raven_toolbox/data.py @@ -0,0 +1,215 @@ +"""Fetch and cache published data artefacts (KEGG reference model, tables, HMMs). + +The mirror of :mod:`raven_toolbox.binaries` for *data*: a version-pinned registry of +downloadable artefacts, fetched on first use, SHA256-verified, and cached under +platformdirs so end users never rebuild them from a KEGG dump (that is the +maintainer's job — see docs/maintaining_kegg_data.md). + +Resolution for any artefact file: + + explicit local dir → cached copy → download from the registry (verify, + cache) → FileNotFoundError with guidance + +The registry is **empty until the artefacts are published** (same as +``binaries._REGISTRY``); until then ``ensure_data_file`` raises an actionable +error. Cache layout:: + + $XDG_CACHE_HOME/raven_toolbox/data/-/ + (or ~/.cache/raven_toolbox/data/... if XDG_CACHE_HOME is unset) +""" +from __future__ import annotations + +import gzip +import os +import shutil +import tarfile +from pathlib import Path +from urllib.request import urlopen + +from raven_toolbox.binaries import _sha256 + +# dataset -> {"version": str, "files": {filename: {"url": str, "sha256": str}}} +# Mirrors data/manifest.json (the cross-language source of truth); regenerate the +# block with scripts/make_registry_snippet.py when publishing a new KEGG release. +_DATA_REGISTRY: dict = { + "kegg": { + "version": "kegg116", + "files": { + "kegg116_core.tar.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_core.tar.gz", + "sha256": "155d5806d43db2fde5783fb124f8782bbcad390a1dd80879c520d2eac9d780e7", + }, + "kegg116_eukaryotes.hmm.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_eukaryotes.hmm.gz", + "sha256": "2d48bc9935575d0f9ba4178bf2df19279bff866b49c1bf83a8e15787b11d6708", + }, + "kegg116_prokaryotes.hmm.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_prokaryotes.hmm.gz", + "sha256": "d80cb2a22dec9fd8336b3998e3b96ee121672f63f4041cddaf09624fe739f1af", + }, + "kegg116_taxonomy.gz": { + "url": "https://github.com/SysBioChalmers/raven-toolbox/releases/download/v0.1.0/kegg116_taxonomy.gz", + "sha256": "1edc56da94d71433e5f08c133600292c311baaf33279a959518ab08389b0e538", + }, + }, + }, +} + +# The core KEGG artefacts needed to build a model (no HMM libraries). These are +# the *base* names of the files bundled into the published ``_core.tar.gz`` +# (each stored version-prefixed inside the archive); ``ensure_kegg_data`` fetches the +# bundle and extracts these, and the build groups exactly this set. +CORE_KEGG_FILES = ( + "reference_model.yml.gz", + "ko_reaction.tsv.gz", + "ko_names.tsv.gz", + "organism_gene_ko.tsv.gz", + "rxn_flags.tsv.gz", +) + + +def _data_cache_dir() -> Path: + base = os.environ.get("XDG_CACHE_HOME") or (Path.home() / ".cache") + return Path(base) / "raven_toolbox" / "data" + + +def _maybe_autoload(registry: dict) -> None: + """Populate the default registry from ``$RAVEN_PYTHON_MANIFEST`` on first use, if set. + + Fires only when the caller relies on the default (still-empty) ``_DATA_REGISTRY`` and + the environment variable points at a manifest. Local import avoids an import cycle with + :mod:`raven_toolbox.manifest`. + """ + if registry is _DATA_REGISTRY and not registry and os.environ.get("RAVEN_PYTHON_MANIFEST"): + from raven_toolbox import manifest as _manifest + + _manifest.load_into_registries() + + +def _bundle(dataset: str, registry: dict) -> dict: + _maybe_autoload(registry) + bundle = registry.get(dataset) + if bundle is None: + raise FileNotFoundError( + f"No data artefacts registered for {dataset!r}. Either pass a local " + f"directory of artefacts, or build them per docs/maintaining_kegg_data.md." + ) + return bundle + + +def ensure_data_file( + dataset: str, + filename: str, + *, + version: str | None = None, + registry: dict | None = None, + verify: bool = False, +) -> Path: + """Download (if needed) and return the cached path to one artefact file. + + Looks the file up in the registry for ``dataset`` (at ``version`` or the + registry's default), downloads it to the version-pinned cache directory, + verifies its SHA256, and returns the path. Re-uses an already-cached copy. + + A freshly downloaded file is always SHA256-checked. ``verify`` additionally + re-checks an *already-cached* file's SHA256 (a mismatch — i.e. a corrupted + cache — discards it and re-downloads); it is off by default so the common + cache-hit path stays fast. + """ + registry = _DATA_REGISTRY if registry is None else registry + bundle = _bundle(dataset, registry) + ver = version or bundle["version"] + entry = bundle.get("files", {}).get(filename) + if entry is None: + raise FileNotFoundError( + f"{filename!r} is not registered for {dataset!r} {ver}. " + f"Available: {sorted(bundle.get('files', {}))}." + ) + + dest_dir = _data_cache_dir() / f"{dataset}-{ver}" + dest = dest_dir / filename + if dest.exists(): + if not verify or _sha256(dest) == entry["sha256"]: + return dest + dest.unlink() # corrupted cache → fall through and re-download + + dest_dir.mkdir(parents=True, exist_ok=True) + tmp = dest.with_name(dest.name + ".part") + with urlopen(entry["url"], timeout=60) as resp, open(tmp, "wb") as out: # noqa: S310 (trusted registry URLs) + shutil.copyfileobj(resp, out) + digest = _sha256(tmp) + if digest != entry["sha256"]: + tmp.unlink(missing_ok=True) + raise ValueError( + f"SHA256 mismatch for {dataset}/{filename} ({ver}): " + f"expected {entry['sha256']}, got {digest}." + ) + tmp.replace(dest) + return dest + + +def ensure_kegg_data(*, version: str | None = None, registry: dict | None = None) -> Path: + """Ensure the core KEGG artefacts are cached; return their directory. + + Fetches the single ``_core.tar.gz`` bundle (the gene-free reference + model + the KO/reaction/organism-gene tables of :data:`CORE_KEGG_FILES`), + SHA256-verifies it, and extracts the version-prefixed members into the cache + directory on first use — ready to pass as the ``artefact_dir`` of + :func:`get_kegg_model_for_organism_from_artefacts`. The HMM libraries and the + taxonomy file are *separate* artefacts (see :func:`ensure_kegg_hmm_library`, + :func:`ensure_kegg_taxonomy`). + """ + registry = _DATA_REGISTRY if registry is None else registry + ver = version or _bundle("kegg", registry)["version"] + dest_dir = _data_cache_dir() / f"kegg-{ver}" + archive = ensure_data_file("kegg", f"{ver}_core.tar.gz", version=ver, registry=registry) + # Extract once; a marker avoids re-extracting (and re-reading the archive) per call. + marker = dest_dir / ".core-extracted" + if not marker.exists(): + with tarfile.open(archive, "r:gz") as tar: + tar.extractall(dest_dir, filter="data") # safe extraction (matches download.py) + marker.touch() + return dest_dir + + +def ensure_kegg_hmm_library( + domain: str, + *, + version: str | None = None, + registry: dict | None = None, +) -> Path: + """Ensure a domain HMM library is cached and decompressed; return the ``.hmm`` path. + + ``domain`` is ``"prokaryotes"`` or ``"eukaryotes"``. Fetches the gzipped + concatenated library ``_.hmm.gz`` and decompresses it once + (cached). Returns the path to the ``.hmm`` flatfile — the argument for + :func:`run_hmmsearch`, which searches it directly (no ``hmmpress`` needed). + + Shipping the gzip flatfile keeps the download ~10x smaller than a binary index, + stays portable across HMMER versions/platforms, and lets the same artefact serve + MATLAB RAVEN. + """ + registry = _DATA_REGISTRY if registry is None else registry + ver = version or _bundle("kegg", registry)["version"] + archive = ensure_data_file("kegg", f"{ver}_{domain}.hmm.gz", version=ver, registry=registry) + library = archive.with_suffix("") # strip ".gz" -> _.hmm + if not library.exists(): + tmp = library.with_name(library.name + ".part") + with gzip.open(archive, "rb") as src, open(tmp, "wb") as out: + shutil.copyfileobj(src, out) + tmp.replace(library) + return library + + +def ensure_kegg_taxonomy(*, version: str | None = None, registry: dict | None = None) -> Path: + """Ensure the KEGG ``taxonomy`` artefact is cached; return its (gzipped) path. + + The gzipped KEGG ``taxonomy`` file is the source for domain classification and for + regenerating the phylogenetic distance matrix — RAVEN's ``keggPhylDist``, which GECKO + uses to pick the closest organism for kcat assignment — via + :func:`raven_toolbox.reconstruction.kegg.phyl_dist` (which reads ``.gz`` directly). So + that capability needs only this published artefact, no MATLAB ``.mat`` file. + """ + registry = _DATA_REGISTRY if registry is None else registry + ver = version or _bundle("kegg", registry)["version"] + return ensure_data_file("kegg", f"{ver}_taxonomy.gz", version=ver, registry=registry) diff --git a/src/raven_python/gapfilling/__init__.py b/src/raven_toolbox/gapfilling/__init__.py similarity index 81% rename from src/raven_python/gapfilling/__init__.py rename to src/raven_toolbox/gapfilling/__init__.py index 747b2939..d7bf8d17 100644 --- a/src/raven_python/gapfilling/__init__.py +++ b/src/raven_toolbox/gapfilling/__init__.py @@ -4,6 +4,6 @@ reactions blocked in a draft can carry flux. For the other gap-fill flavour (fill until the objective is feasible) use ``cobra.flux_analysis.gapfill``. """ -from raven_python.gapfilling.fill import GapFillResult, connect_blocked_reactions +from raven_toolbox.gapfilling.fill import GapFillResult, connect_blocked_reactions __all__ = ["GapFillResult", "connect_blocked_reactions"] diff --git a/src/raven_python/gapfilling/fill.py b/src/raven_toolbox/gapfilling/fill.py similarity index 85% rename from src/raven_python/gapfilling/fill.py rename to src/raven_toolbox/gapfilling/fill.py index ba3418d7..01b15c49 100644 --- a/src/raven_python/gapfilling/fill.py +++ b/src/raven_toolbox/gapfilling/fill.py @@ -15,13 +15,15 @@ """ from __future__ import annotations +import math from collections.abc import Iterable from dataclasses import dataclass import cobra from cobra.flux_analysis import find_blocked_reactions, flux_variability_analysis +from optlang.interface import Variable # untyped (resolves to Any); used for container hints -from raven_python.manipulation.transfer import add_reactions_from_model +from raven_toolbox.manipulation.transfer import add_reactions_from_model @dataclass @@ -74,15 +76,27 @@ def _solve_min_templates( the problem is infeasible. """ prob = working.problem - indicators: dict[str, object] = {} + indicators: dict[str, Variable] = {} extra = [] + # Big-M for the indicator constraints. A template reaction with an infinite + # bound would otherwise put an infinite coefficient into the MILP; clamp it to + # the largest finite bound magnitude in the model (>= any flux it can carry). + finite_bounds = [ + abs(b) + for r in working.reactions + for b in (r.lower_bound, r.upper_bound) + if math.isfinite(b) + ] + big_m = max(finite_bounds) if finite_bounds else 1000.0 for rid in template_ids: rxn = working.reactions.get_by_id(rid) y = prob.Variable(f"_gf_keep_{rid}", type="binary") indicators[rid] = y # Flux is confined to [lb*y, ub*y]: zero unless the reaction is kept (y=1). - extra.append(prob.Constraint(rxn.flux_expression - rxn.upper_bound * y, ub=0, name=f"_gf_ub_{rid}")) - extra.append(prob.Constraint(rxn.flux_expression - rxn.lower_bound * y, lb=0, name=f"_gf_lb_{rid}")) + ub = rxn.upper_bound if math.isfinite(rxn.upper_bound) else big_m + lb = rxn.lower_bound if math.isfinite(rxn.lower_bound) else -big_m + extra.append(prob.Constraint(rxn.flux_expression - ub * y, ub=0, name=f"_gf_ub_{rid}")) + extra.append(prob.Constraint(rxn.flux_expression - lb * y, lb=0, name=f"_gf_lb_{rid}")) working.add_cons_vars(list(indicators.values()) + extra) if allow_net_production: # relax steady state to Sv >= 0 (mets may accumulate) @@ -137,6 +151,8 @@ def connect_blocked_reactions( The draft is expected to have exchange reactions for its nutrients (otherwise most reactions are trivially blocked). """ + if penalty <= 0: + raise ValueError(f"penalty must be > 0 (a non-positive cost malforms the MILP); got {penalty}.") templates = _as_models(templates) blocked = set(find_blocked_reactions(model)) candidates = [r for r in blocked if model.reactions.get_by_id(r).lower_bound >= 0] diff --git a/src/raven_python/init/__init__.py b/src/raven_toolbox/init/__init__.py similarity index 63% rename from src/raven_python/init/__init__.py rename to src/raven_toolbox/init/__init__.py index 8ef24ad2..ae9744ae 100644 --- a/src/raven_python/init/__init__.py +++ b/src/raven_toolbox/init/__init__.py @@ -14,15 +14,15 @@ * :func:`ftinit` — the full pipeline (``prep_init_model`` → staged ``run_ftinit`` → ``fill_tasks`` → ``remove_low_score_genes``). """ -from raven_python.init.build import InitModelResult, get_init_model -from raven_python.init.ftinit import FtInitResult, ftinit, run_ftinit -from raven_python.init.genes import remove_low_score_genes -from raven_python.init.init import InitResult, run_init -from raven_python.init.merge import group_rxn_scores, merge_linear -from raven_python.init.prep import PrepData, ReactionMasks, classify_reactions, prep_init_model -from raven_python.init.score import gene_scores_from_expression, score_reactions_from_genes -from raven_python.init.steps import InitStep, get_init_steps -from raven_python.init.taskfill import TaskFillResult, fill_tasks +from raven_toolbox.init.build import InitModelResult, get_init_model +from raven_toolbox.init.ftinit import FtInitResult, ftinit, run_ftinit +from raven_toolbox.init.genes import remove_low_score_genes +from raven_toolbox.init.init import InitResult, run_init +from raven_toolbox.init.merge import group_rxn_scores, merge_linear +from raven_toolbox.init.prep import PrepData, ReactionMasks, classify_reactions, prep_init_model +from raven_toolbox.init.score import gene_scores_from_expression, score_reactions_from_genes +from raven_toolbox.init.steps import InitStep, get_init_steps +from raven_toolbox.init.taskfill import TaskFillResult, fill_tasks __all__ = [ "FtInitResult", diff --git a/src/raven_python/init/build.py b/src/raven_toolbox/init/build.py similarity index 92% rename from src/raven_python/init/build.py rename to src/raven_toolbox/init/build.py index a0d05385..a725cb12 100644 --- a/src/raven_python/init/build.py +++ b/src/raven_toolbox/init/build.py @@ -6,7 +6,7 @@ loaders) or reaction scores directly. ``essential_rxns`` are forced kept. For task-aware gap-filling on top of the resulting model, use ftINIT -(:func:`raven_python.init.ftinit`); ``get_init_model`` itself does not run the task layer. +(:func:`raven_toolbox.init.ftinit`); ``get_init_model`` itself does not run the task layer. """ from __future__ import annotations @@ -16,8 +16,8 @@ import cobra from cobra.flux_analysis import find_blocked_reactions -from raven_python.init.init import run_init -from raven_python.init.score import score_reactions_from_genes +from raven_toolbox.init.init import run_init +from raven_toolbox.init.score import score_reactions_from_genes @dataclass @@ -69,8 +69,8 @@ def get_init_model( model, gene_scores, isozyme_scoring=isozyme_scoring, complex_scoring=complex_scoring, no_gene_score=no_gene_score, ) - else: - scores = dict(rxn_scores) + else: # rxn_scores is not None here (exactly-one-of check above); `or {}` placates mypy + scores = dict(rxn_scores or {}) deleted_dead_end: list[str] = [] if remove_dead_ends: diff --git a/src/raven_python/init/ftinit.py b/src/raven_toolbox/init/ftinit.py similarity index 93% rename from src/raven_python/init/ftinit.py rename to src/raven_toolbox/init/ftinit.py index b355e457..9cf2be45 100644 --- a/src/raven_python/init/ftinit.py +++ b/src/raven_toolbox/init/ftinit.py @@ -23,7 +23,7 @@ oriented irreversible in its forced direction (``prepINITModel`` does this). Objective: **maximise** ``Σ score·indicator``. Unlike classic INIT -(:func:`raven_python.init.run_init`), ftINIT does **not** reward production of every +(:func:`raven_toolbox.init.run_init`), ftINIT does **not** reward production of every metabolite — ``prod_weight`` applies only to metabolomics-detected metabolites (not yet implemented; passing a non-empty ``metabolomics`` argument raises ``NotImplementedError``). Connectivity comes solely from the flux gates plus any @@ -51,12 +51,14 @@ from dataclasses import dataclass, field import cobra +from cobra.exceptions import OptimizationError +from optlang.interface import Variable # untyped (resolves to Any); used for container hints from optlang.symbolics import Real, add, mul -from raven_python.init.genes import remove_low_score_genes -from raven_python.init.merge import group_rxn_scores -from raven_python.init.steps import get_init_steps -from raven_python.init.taskfill import fill_tasks +from raven_toolbox.init.genes import remove_low_score_genes +from raven_toolbox.init.merge import group_rxn_scores +from raven_toolbox.init.steps import get_init_steps +from raven_toolbox.init.taskfill import fill_tasks _FORCE_ON = 0.1 # min flux for a reaction to count as "on" (RAVEN forceOnLim) _BIG_M = 100.0 # indicator/direction big-M cap on a *scored* reaction's flux (RAVEN's 100) @@ -99,7 +101,7 @@ def run_ftinit( ``ignore_mets`` are metabolite **names** whose mass balance is dropped (RAVEN's per-step "simple metabolite" removal, e.g. H2O/H+). See the module docstring for the formulation. This is the single-step variant; the staged schedule - (:func:`raven_python.init.ftinit`) calls it per step. + (:func:`raven_toolbox.init.ftinit`) calls it per step. """ scores = dict(rxn_scores or {}) essential = set(essential_rxns or []) @@ -111,8 +113,8 @@ def run_ftinit( variables: list = [] constraints: list = [] - flux_terms: dict[str, list[tuple[object, float]]] = {} # rxn id -> [(var, sign)] - indicators: dict[str, tuple[object, float]] = {} # rxn id -> (indicator var, score) + flux_terms: dict[str, list[tuple[Variable, float]]] = {} # rxn id -> [(var, sign)] + indicators: dict[str, tuple[Variable, float]] = {} # rxn id -> (indicator var, score) free_or_essential: set[str] = set() # kept regardless of an indicator def add_constraint(expr, **kw): @@ -210,14 +212,14 @@ def add_constraint(expr, **kw): opt.optimize() # Accept a near-optimal incumbent (when a MIP gap / time limit is set), as RAVEN does. if opt.status not in ("optimal", "feasible", "suboptimal", "time_limit"): - raise RuntimeError(f"ftINIT MILP did not solve (status: {opt.status}).") + raise OptimizationError(f"ftINIT MILP did not solve (status: {opt.status}).") # RAVEN: a reaction is "on" iff its indicator ≥ 0.5 (positive indicators are # continuous and can land fractionally when a reaction can carry only tiny flux). on = {rid for rid, (ind, _) in indicators.items() if (ind.primal or 0.0) >= 0.5} kept = free_or_essential | on deleted = [r.id for r in model.reactions if r.id not in kept] - fluxes = { + fluxes: dict[str, float] = { rid: sum(sign * (var.primal or 0.0) for var, sign in terms) for rid, terms in flux_terms.items() } @@ -244,18 +246,18 @@ def ftinit( ) -> cobra.Model: """Run the full ftINIT pipeline on prepData and return the context-specific model. - ``prep`` is a :class:`raven_python.init.PrepData`. ``rxn_scores`` maps **original** + ``prep`` is a :class:`raven_toolbox.init.PrepData`. ``rxn_scores`` maps **original** reaction id → score (e.g. from :func:`score_reactions_from_genes` on the template). - Each step (:func:`raven_python.init.get_init_steps`) regroups scores under its + Each step (:func:`raven_toolbox.init.get_init_steps`) regroups scores under its ``ignore_mask``, fixes the reactions turned on by earlier steps as essential (in their flux direction), and solves :func:`run_ftinit` on the merged model. Reactions never turned on (and not essential or left-in) are removed from the reference model; exchange reactions are always kept (RAVEN re-adds them). If ``fill_gaps`` and ``prep`` carries tasks, reactions are added back so every task - is feasible (:func:`raven_python.init.fill_tasks`). If ``gene_scores`` is given, + is feasible (:func:`raven_toolbox.init.fill_tasks`). If ``gene_scores`` is given, negative-scoring genes are pruned from the GPRs at the end - (:func:`raven_python.init.remove_low_score_genes`). + (:func:`raven_toolbox.init.remove_low_score_genes`). Essential reactions are forced to carry ``force_on`` (default 0.1) of flux in the forced direction. On genome-scale models a stricter regime is needed (the previous diff --git a/src/raven_python/init/genes.py b/src/raven_toolbox/init/genes.py similarity index 84% rename from src/raven_python/init/genes.py rename to src/raven_toolbox/init/genes.py index ceed3dae..cbee6fdc 100644 --- a/src/raven_python/init/genes.py +++ b/src/raven_toolbox/init/genes.py @@ -11,13 +11,12 @@ from __future__ import annotations import ast -import statistics from collections.abc import Mapping import cobra from cobra.manipulation import remove_genes -_AGG = {"min": min, "max": max, "median": statistics.median, "average": statistics.fmean} +from raven_toolbox.utils.gpr import resolve_aggregators def _prune(node, scores, iso, cplx) -> tuple[str | None, float | None]: @@ -27,15 +26,15 @@ def _prune(node, scores, iso, cplx) -> tuple[str | None, float | None]: if not isinstance(node, ast.BoolOp): return None, None - children = [_prune(v, scores, iso, cplx) for v in node.values] - children = [(s, sc) for s, sc in children if s is not None] + pruned = [_prune(v, scores, iso, cplx) for v in node.values] + children: list[tuple[str, float | None]] = [(s, sc) for s, sc in pruned if s is not None] if isinstance(node.op, ast.And): # complex: keep every subunit, prune nested ORs kept = children else: # OR / isozymes: drop negative-scoring alternatives, keep at least one kept = [(s, sc) for s, sc in children if sc is None or sc >= 0] - if not kept: # all negative → keep the least-negative - kept = [max(children, key=lambda c: c[1])] + if not kept: # all negative → keep the least-negative (every sc is non-None here) + kept = [max(children, key=lambda c: c[1] if c[1] is not None else float("-inf"))] parts = [s for s, _ in kept] score_vals = [sc for _, sc in kept if sc is not None] @@ -64,10 +63,7 @@ def remove_low_score_genes( **deterministically** (first on a tie), unlike RAVEN's random tie-break — same quality, reproducible. """ - for name, value in (("isozyme_scoring", isozyme_scoring), ("complex_scoring", complex_scoring)): - if value not in _AGG: - raise ValueError(f"{name} must be one of {sorted(_AGG)}; got {value!r}.") - iso, cplx = _AGG[isozyme_scoring], _AGG[complex_scoring] + iso, cplx = resolve_aggregators(isozyme_scoring, complex_scoring) out = model.copy() for rxn in out.reactions: diff --git a/src/raven_python/init/init.py b/src/raven_toolbox/init/init.py similarity index 97% rename from src/raven_python/init/init.py rename to src/raven_toolbox/init/init.py index f23e17a8..05b6d84a 100644 --- a/src/raven_python/init/init.py +++ b/src/raven_toolbox/init/init.py @@ -39,6 +39,8 @@ from dataclasses import dataclass import cobra +from cobra.exceptions import OptimizationError +from optlang.interface import Variable # untyped (resolves to Any); used for container hints from optlang.symbolics import Real, add, mul _EPS = 1.0 # flux an included reaction must carry (RAVEN's fake-met unit) @@ -136,7 +138,7 @@ def run_init( flux = {d.key: prob.Variable(f"v_{d.key}", lb=0.0, ub=d.ub) for d in directed} # Binary include-indicators for non-essential reactions; eps*x <= v <= ub*x. - keep: dict[str, object] = {} + keep: dict[str, Variable] = {} gates = [] for d in directed: if d.essential: @@ -203,7 +205,7 @@ def run_init( opt.optimize() # With a MIP gap / time limit set, accept a near-optimal incumbent (as RAVEN does). if opt.status not in ("optimal", "feasible", "suboptimal", "time_limit"): - raise RuntimeError(f"INIT MILP did not solve (status: {opt.status}).") + raise OptimizationError(f"INIT MILP did not solve (status: {opt.status}).") # A reaction is kept if any of its directed parts is essential or has x≈1. kept_origins = {d.origin for d in directed if d.essential} diff --git a/src/raven_python/init/merge.py b/src/raven_toolbox/init/merge.py similarity index 100% rename from src/raven_python/init/merge.py rename to src/raven_toolbox/init/merge.py diff --git a/src/raven_python/init/prep.py b/src/raven_toolbox/init/prep.py similarity index 97% rename from src/raven_python/init/prep.py rename to src/raven_toolbox/init/prep.py index 8ed4b895..3694fdf3 100644 --- a/src/raven_python/init/prep.py +++ b/src/raven_toolbox/init/prep.py @@ -7,7 +7,7 @@ :func:`classify_reactions` is the reaction taxonomy: exchange, GPR-less import / simple / advanced transport, spontaneous, GPR-less extracellular, custom, and -"any without a GPR". The staged schedule (:func:`raven_python.init.get_init_steps`) selects +"any without a GPR". The staged schedule (:func:`raven_toolbox.init.get_init_steps`) selects which categories to keep out of each MILP step via an 8-bit pattern. """ from __future__ import annotations @@ -17,8 +17,8 @@ import cobra -from raven_python.init.merge import merge_linear -from raven_python.tasks import Task, find_task_essential_reactions +from raven_toolbox.init.merge import merge_linear +from raven_toolbox.tasks import Task, find_task_essential_reactions @dataclass diff --git a/src/raven_python/init/score.py b/src/raven_toolbox/init/score.py similarity index 82% rename from src/raven_python/init/score.py rename to src/raven_toolbox/init/score.py index 6e14f867..7beef0f2 100644 --- a/src/raven_python/init/score.py +++ b/src/raven_toolbox/init/score.py @@ -6,21 +6,20 @@ **AND** (complexes) with ``complex_scoring`` (default ``min``). Genes missing from ``gene_scores`` are *omitted*; a reaction with no genes — or whose genes are all missing — gets ``no_gene_score`` (default −2). These reaction scores feed -:func:`raven_python.init.run_init` and :func:`raven_python.init.ftinit`. +:func:`raven_toolbox.init.run_init` and :func:`raven_toolbox.init.ftinit`. Upstream — the omics-data → gene-score step (thresholding, expression levels) — lives -in :mod:`raven_python.omics`; this function takes gene scores as given. +in :mod:`raven_toolbox.omics`; this function takes gene scores as given. """ from __future__ import annotations import ast import math -import statistics from collections.abc import Mapping import cobra -_AGG = {"min": min, "max": max, "median": statistics.median, "average": statistics.fmean} +from raven_toolbox.utils.gpr import resolve_aggregators def gene_scores_from_expression( @@ -40,10 +39,9 @@ def gene_scores_from_expression( score is clamped to ``[min_score, max_score]``; non-positive level/reference (and missing reference) → ``min_score`` (RAVEN maps these NaNs to -5). """ - scalar = isinstance(reference, (int, float)) scores: dict[str, float] = {} for gene, level in expression.items(): - ref = reference if scalar else reference.get(gene) + ref = reference if isinstance(reference, (int, float)) else reference.get(gene) if not level or not ref or level <= 0 or ref <= 0: scores[gene] = min_score else: @@ -70,10 +68,7 @@ def score_reactions_from_genes( no_gene_score: float = -2.0, ) -> dict[str, float]: """Return ``{reaction_id: score}`` from per-gene scores via each reaction's GPR.""" - for name, value in (("isozyme_scoring", isozyme_scoring), ("complex_scoring", complex_scoring)): - if value not in _AGG: - raise ValueError(f"{name} must be one of {sorted(_AGG)}; got {value!r}.") - iso, cplx = _AGG[isozyme_scoring], _AGG[complex_scoring] + iso, cplx = resolve_aggregators(isozyme_scoring, complex_scoring) scores: dict[str, float] = {} for rxn in model.reactions: diff --git a/src/raven_python/init/steps.py b/src/raven_toolbox/init/steps.py similarity index 96% rename from src/raven_python/init/steps.py rename to src/raven_toolbox/init/steps.py index d8a7b86e..4eb7ea8a 100644 --- a/src/raven_python/init/steps.py +++ b/src/raven_toolbox/init/steps.py @@ -2,7 +2,7 @@ ftINIT runs as a short sequence of MILP steps instead of one big MILP. Each step (:class:`InitStep`) chooses which reaction categories to hold out of the problem -(``ignore_mask``, an 8-bit pattern over :class:`raven_python.init.ReactionMasks`), whether +(``ignore_mask``, an 8-bit pattern over :class:`raven_toolbox.init.ReactionMasks`), whether to drop positive reversibles and allow metabolite secretion, and how to treat the reactions turned on by previous steps (``'ignore'`` for the first step, ``'essential'`` to fix them on). :func:`get_init_steps` builds the standard schedules. diff --git a/src/raven_python/init/taskfill.py b/src/raven_toolbox/init/taskfill.py similarity index 94% rename from src/raven_python/init/taskfill.py rename to src/raven_toolbox/init/taskfill.py index 58501ced..0bcd6fa6 100644 --- a/src/raven_python/init/taskfill.py +++ b/src/raven_toolbox/init/taskfill.py @@ -18,10 +18,11 @@ from dataclasses import dataclass import cobra +from cobra.exceptions import OptimizationError from optlang.symbolics import Real, add, mul -from raven_python.tasks import Task -from raven_python.tasks.check import ( +from raven_toolbox.tasks import Task +from raven_toolbox.tasks.check import ( _metabolite_bounds, _set_constraint_bounds, apply_task_constraints, @@ -87,13 +88,15 @@ def _fill_one_task( right trade for tractability, exactly as for the main ftINIT MILP. """ if not candidates: # nothing left to add → task cannot be made feasible - raise RuntimeError(f"gap-filling found no candidates for task {task.id!r}.") + raise OptimizationError(f"gap-filling found no candidates for task {task.id!r}.") combined = _closed_copy(model) # task I/O via the task's b, not the model's exchanges combined.add_reactions([r.copy() for r in candidates]) name_to_id, comp_to_ids = task_name_maps(combined) _, error = apply_task_constraints(combined, task, name_to_id, comp_to_ids) if error is not None: - raise RuntimeError(f"task {task.id!r} could not be applied to the reference: {error}") + raise OptimizationError( + f"task {task.id!r} could not be applied to the reference: {error}" + ) prob = combined.problem extras = [] @@ -126,7 +129,7 @@ def _fill_one_task( # (no incumbent) means the task cannot be satisfied from the reference. if combined.solver.status not in ("optimal", "feasible", "suboptimal", "time_limit") or \ combined.variables[f"_fill_{candidates[0].id}"].primal is None: - raise RuntimeError(f"gap-filling found no way to make task {task.id!r} feasible.") + raise OptimizationError(f"gap-filling found no way to make task {task.id!r} feasible.") return [c.id for c in candidates if (combined.variables[f"_fill_{c.id}"].primal or 0.0) > 0.5] @@ -174,7 +177,7 @@ def fill_tasks( avail = [r for r in candidates if r.id not in present] try: chosen = _fill_one_task(out, avail, task, costs, mip_gap=mip_gap, time_limit=time_limit) - except RuntimeError: + except OptimizationError: failed.append(task.id) continue if chosen: diff --git a/src/raven_python/io/__init__.py b/src/raven_toolbox/io/__init__.py similarity index 56% rename from src/raven_python/io/__init__.py rename to src/raven_toolbox/io/__init__.py index 1fc937a0..312d68b8 100644 --- a/src/raven_python/io/__init__.py +++ b/src/raven_toolbox/io/__init__.py @@ -2,11 +2,11 @@ the GECKO ec-model substructure), SIF, Excel export, and the Standard-GEM ``model//…`` git layout. """ -from raven_python.io.ec_data import EcData -from raven_python.io.excel import export_to_excel -from raven_python.io.git import export_for_git -from raven_python.io.sif import export_model_to_sif -from raven_python.io.yaml import read_yaml_model, write_yaml_model +from raven_toolbox.io.ec_data import EcData +from raven_toolbox.io.excel import export_to_excel +from raven_toolbox.io.git import export_for_git +from raven_toolbox.io.sif import export_model_to_sif +from raven_toolbox.io.yaml import read_yaml_model, write_yaml_model __all__ = [ "EcData", diff --git a/src/raven_python/io/ec_data.py b/src/raven_toolbox/io/ec_data.py similarity index 96% rename from src/raven_python/io/ec_data.py rename to src/raven_toolbox/io/ec_data.py index 850c8e8b..ef030bc8 100644 --- a/src/raven_python/io/ec_data.py +++ b/src/raven_toolbox/io/ec_data.py @@ -15,7 +15,7 @@ `ec_data_to_yaml_sections`. It does NOT touch the cobra-shaped portion of the document — that stays -with ``raven_python.io.yaml``, which calls into here when ec sections are +with ``raven_toolbox.io.yaml``, which calls into here when ec sections are present. The split mirrors RAVEN MATLAB: ``readYAMLmodel.m`` populates ``model.ec`` whenever the YAML defines it; downstream consumers (geckopy / GECKO) operate on the populated struct. @@ -45,6 +45,7 @@ from __future__ import annotations import math +import warnings from dataclasses import dataclass, field from typing import Any @@ -263,6 +264,14 @@ def _build_ec_data( ) mat[i, j] = float(stoich) + if n_r > 0 and mat.nnz == 0: # reactions present but none coupled to an enzyme + warnings.warn( + f"ec-rxns has {n_r} entries but none reference an enzyme; the " + "reaction↔enzyme coupling matrix is all-zero (likely malformed " + "ec-rxns/ec-enzymes).", + stacklevel=2, + ) + return EcData( gecko_light=gecko_light, rxns=rxns, @@ -379,5 +388,5 @@ def _eccodes_to_yaml(eccodes: str): a scalar string for one EC, a list for multiple.""" parts = [p for p in eccodes.split(";") if p] if len(parts) <= 1: - return eccodes + return ";".join(parts) return parts diff --git a/src/raven_python/io/excel.py b/src/raven_toolbox/io/excel.py similarity index 92% rename from src/raven_python/io/excel.py rename to src/raven_toolbox/io/excel.py index cf6196ed..5ddf8e9b 100644 --- a/src/raven_python/io/excel.py +++ b/src/raven_toolbox/io/excel.py @@ -2,9 +2,9 @@ Writes the five-sheet RAVEN xlsx layout — RXNS, METS, COMPS, GENES, MODEL — pulling RAVEN-specific values back out of cobra's ``annotation`` / ``notes`` (where the -raven_python YAML reader stashes them). Excel *import* is intentionally not provided. +raven_toolbox YAML reader stashes them). Excel *import* is intentionally not provided. -Requires the optional ``openpyxl`` dependency (``pip install raven_python[excel]``). +Requires the optional ``openpyxl`` dependency (``pip install raven_toolbox[excel]``). """ from __future__ import annotations @@ -12,10 +12,12 @@ import cobra +from raven_toolbox.utils.parse import subsystem_to_str + def _miriam_string(annotation: dict, exclude: tuple[str, ...] = ()) -> str: """RAVEN MIRIAM column: ``namespace/id;namespace/id2;...`` (sorted).""" - parts = [] + parts: list[str] = [] for namespace in sorted(annotation): if namespace in exclude: continue @@ -63,7 +65,7 @@ def export_to_excel( except ImportError as exc: # pragma: no cover - exercised only without openpyxl raise ImportError( "export_to_excel requires openpyxl. Install it with " - "`pip install raven_python[excel]` (or `pip install openpyxl`)." + "`pip install raven_toolbox[excel]` (or `pip install openpyxl`)." ) from exc reactions = sorted(model.reactions, key=lambda r: r.id) if sort_ids else list(model.reactions) @@ -84,9 +86,7 @@ def export_to_excel( "REPLACEMENT ID", "NOTE", "REFERENCE", "CONFIDENCE SCORE"] ) for r in reactions: - subsystem = r.subsystem - if isinstance(subsystem, (list, tuple)): - subsystem = ";".join(subsystem) + subsystem = subsystem_to_str(r.subsystem) ws.append([ None, r.id, r.name, _equation(r), _ec_codes(r), r.gene_reaction_rule, r.lower_bound, r.upper_bound, @@ -104,7 +104,7 @@ def export_to_excel( ws.append([ None, f"{m.name}[{m.compartment}]", m.name, None, _miriam_string(m.annotation, exclude=("smiles",)), - None if inchi else m.formula, inchi, m.compartment, m.id, m.charge, + m.formula, inchi, m.compartment, m.id, m.charge, ]) # --- COMPS --- diff --git a/src/raven_python/io/git.py b/src/raven_toolbox/io/git.py similarity index 91% rename from src/raven_python/io/git.py rename to src/raven_toolbox/io/git.py index 80bf8e8e..735a1275 100644 --- a/src/raven_python/io/git.py +++ b/src/raven_toolbox/io/git.py @@ -5,7 +5,7 @@ Git-maintained model repository (Metabolic Atlas / Human-GEM / yeast-GEM style), plus a ``dependencies.txt`` recording tool versions. -Thin orchestration over the writers raven_python already exposes: ``write_yaml_model``, +Thin orchestration over the writers raven_toolbox already exposes: ``write_yaml_model``, cobra's ``write_sbml_model`` and ``save_matlab_model``, ``export_to_excel``, plus a single-file reaction table (txt). """ @@ -18,9 +18,9 @@ import cobra -from raven_python.io.excel import _equation, export_to_excel -from raven_python.io.yaml import write_yaml_model -from raven_python.utils.sort import sort_identifiers +from raven_toolbox.io.excel import _equation, export_to_excel +from raven_toolbox.io.yaml import write_yaml_model +from raven_toolbox.utils.sort import sort_identifiers _ALL_FORMATS = ("yml", "xml", "mat", "xlsx", "txt") @@ -101,6 +101,6 @@ def target(fmt: str) -> Path: with open(root / "dependencies.txt", "w", encoding="utf-8") as fh: fh.write(f"python\t{platform.python_version()}\n") fh.write(f"cobra\t{_version('cobra')}\n") - fh.write(f"raven_python\t{_version('raven_python')}\n") + fh.write(f"raven_toolbox\t{_version('raven_toolbox')}\n") return root diff --git a/src/raven_python/io/sif.py b/src/raven_toolbox/io/sif.py similarity index 97% rename from src/raven_python/io/sif.py rename to src/raven_toolbox/io/sif.py index 9e73efa9..49ce2106 100644 --- a/src/raven_python/io/sif.py +++ b/src/raven_toolbox/io/sif.py @@ -19,7 +19,7 @@ import cobra -from raven_python.manipulation.irreversible import convert_to_irreversible +from raven_toolbox.manipulation.irreversible import convert_to_irreversible _GRAPH_TYPES = ("rc", "rr", "cc") diff --git a/src/raven_python/io/yaml.py b/src/raven_toolbox/io/yaml.py similarity index 68% rename from src/raven_python/io/yaml.py rename to src/raven_toolbox/io/yaml.py index f24c642a..eed439c3 100644 --- a/src/raven_python/io/yaml.py +++ b/src/raven_toolbox/io/yaml.py @@ -19,7 +19,7 @@ * **Model-level extras** cobra ignores: ``version`` and the ``metaData`` provenance block, preserved on ``model.notes`` for round-tripping. * **The GECKO ec sections** (``ec-rxns``/``ec-enzymes``/``gecko_light``): on - read, parsed into a typed :class:`~raven_python.io.ec_data.EcData` and + read, parsed into a typed :class:`~raven_toolbox.io.ec_data.EcData` and attached as ``model.ec``; on write, serialised back to top-level sections whenever ``model.ec`` is present. This mirrors RAVEN ``readYAMLmodel.m`` / ``writeYAMLmodel.m``, which populate the ``model.ec`` struct when the YAML @@ -30,6 +30,8 @@ * older RAVEN files with ``id`` / ``name`` nested in ``metaData``; * per-metabolite top-level ``smiles`` (lifted into ``annotation['smiles']``); +* per-reaction top-level ``eccodes`` (lifted into ``annotation['ec-code']`` — + the cobra-standard place where geckopy reads EC numbers); * very old RAVEN files written as a bare ``-`` sequence of single-key mappings rather than one big mapping; * MATLAB GECKO ecModels whose ``usage_prot_*`` and ``prot_pool_exchange`` @@ -48,7 +50,7 @@ from cobra.io.dict import model_from_dict, model_to_dict from cobra.io.yaml import yaml as _cobra_yaml # ruamel round-trip YAML (handles !!omap) -from raven_python.io.ec_data import ( +from raven_toolbox.io.ec_data import ( EcData, ec_data_from_yaml_sections, ec_data_to_yaml_sections, @@ -66,12 +68,16 @@ def _open_text(path: str | Path, mode: str): # 'note' to avoid colliding with the notes container itself.) _MET_FIELDS = (("inchis", "inchis"), ("deltaG", "deltaG"), ("metFrom", "metFrom"), ("notes", "note")) _RXN_FIELDS = ( - ("confidence_score", "confidence_score"), ("references", "references"), ("rxnFrom", "rxnFrom"), ("deltaG", "deltaG"), + ("confidence_score", "confidence_score"), ("notes", "note"), ) +# Legacy YAML keys accepted on READ for reaction notes. Old RAVEN MATLAB writers +# used "rxnNotes"; the canonical key (matching cobrapy and the current MATLAB +# writer) is "notes". When both appear, "notes" wins. +_LEGACY_RXN_KEY_ALIASES = (("rxnNotes", "notes"),) _GENE_FIELDS = (("protein", "protein"),) _COBRA_TOP_KEYS = frozenset({"metabolites", "reactions", "genes", "compartments", "id", "name"}) @@ -82,8 +88,17 @@ def _open_text(path: str | Path, mode: str): def _to_plain(obj): + """Coerce ruamel/numpy scalars to plain Python primitives. + + Dicts are returned as ``OrderedDict`` so that the round-trip dumper + emits them with the ``!!omap`` tag (ruamel's CommentedMap subclass + is replaced by a plain ``OrderedDict`` to avoid carrying ruamel's + own type tags through). Returning a plain ``dict`` instead would + drop the ``!!omap`` tag and produce files that RAVEN's MATLAB + reader (a line-based parser keyed on ``!!omap``) cannot load. + """ if isinstance(obj, dict): - return {str(k): _to_plain(v) for k, v in obj.items()} + return OrderedDict((str(k), _to_plain(v)) for k, v in obj.items()) if isinstance(obj, (list, tuple)): return [_to_plain(v) for v in obj] if isinstance(obj, bool) or obj is None: @@ -154,8 +169,9 @@ def model_from_yaml_data(raw: dict) -> cobra.Model: out of legacy ``metaData``; preserves ``version`` and ``metaData`` on ``model.notes`` for round-trip. 2. **legacy quirks:** lifts per-metabolite top-level ``smiles`` into - ``annotation['smiles']`` (older MATLAB GECKO ecModels emitted it - at the top level); flips the older reverse-direction + ``annotation['smiles']`` and per-reaction top-level ``eccodes`` into + ``annotation['ec-code']`` (older RAVEN/MATLAB GECKO files emitted + these at the top level); flips the older reverse-direction ``usage_prot_*`` / ``prot_pool_exchange`` convention to the forward convention. 3. **GECKO ec sections:** when ``ec-rxns`` / ``ec-enzymes`` are @@ -175,6 +191,26 @@ def model_from_yaml_data(raw: dict) -> cobra.Model: # canonical place. No-op on current files. _lift_smiles_to_annotation(raw.get("metabolites")) + # Legacy quirk: per-reaction top-level `eccodes` -> annotation['ec-code']. + # EC numbers are standard cobra annotation; older RAVEN/MATLAB files put + # them at the reaction top level, which hid them from cobra/geckopy (which + # read annotation['ec-code']). Lift before model_from_dict. No-op on + # current cobra-shaped files. + _lift_eccodes_to_annotation(raw.get("reactions")) + + # Normalise legacy reaction-side YAML keys (e.g. RAVEN MATLAB's + # ``rxnNotes`` -> the canonical ``notes``) before any field capture so + # the capture step sees a single key per concept. + for entry in raw.get("reactions", []): + if not isinstance(entry, dict): + continue + for legacy_key, canonical_key in _LEGACY_RXN_KEY_ALIASES: + if legacy_key in entry and canonical_key not in entry: + entry[canonical_key] = entry.pop(legacy_key) + elif legacy_key in entry: + # Canonical wins; just drop the legacy duplicate. + entry.pop(legacy_key) + met_notes = _capture_entry_fields(raw.get("metabolites", []), _MET_FIELDS) rxn_notes = _capture_entry_fields(raw.get("reactions", []), _RXN_FIELDS) gene_notes = _capture_entry_fields(raw.get("genes", []), _GENE_FIELDS) @@ -192,18 +228,33 @@ def model_from_yaml_data(raw: dict) -> cobra.Model: # No-op when none match. _flip_legacy_prot_direction(model) - # Legacy files keep id/name inside metaData; restore them if cobra found none. + # RAVEN convention keeps id/name/version inside metaData; lift them + # onto the model so cobra-shaped accessors find them too. Older + # cobra-style files (id/name/version at root) are handled by the + # cobra reader; the metaData lookups below are pure fallbacks. if metadata.get("id") and not model.id: model.id = metadata["id"] if metadata.get("name") and not model.name: model.name = metadata["name"] + if version is None and metadata.get("version") is not None: + version = metadata["version"] if metadata: model.notes["metaData"] = metadata if version is not None: model.notes["version"] = version # Pop the ec sections out of `foreign` and into a typed EcData. - # The remaining unknown keys round-trip opaquely. + # The remaining unknown keys round-trip opaquely. Pre-shim RAVEN + # MATLAB writes wrote `geckoLight: "true"` inside metaData (rather + # than the current top-level `gecko_light`); honour the legacy + # placement too — keep the metaData entry untouched (round-trip) + # and surface it at the top level so EcData picks it up. + legacy_gecko = metadata.get("geckoLight") + if legacy_gecko is not None and "gecko_light" not in foreign: + if isinstance(legacy_gecko, str): + foreign["gecko_light"] = legacy_gecko.lower() == "true" + else: + foreign["gecko_light"] = bool(legacy_gecko) ec_sections = {k: foreign.pop(k) for k in list(foreign) if k in _EC_TOP_KEYS} ec_data = ec_data_from_yaml_sections(ec_sections) if ec_data is not None: @@ -242,6 +293,45 @@ def _lift_smiles_to_annotation(metabolites) -> None: ) +def _lift_eccodes_to_annotation(reactions) -> None: + """Move a reaction's legacy top-level ``eccodes`` into ``annotation['ec-code']``. + + EC numbers are a standard MIRIAM cross-reference, so the cobra/raven + convention is to carry them inside ``annotation`` under the ``ec-code`` + key — where cobra and geckopy read them — not as a RAVEN-only top-level + field. Older RAVEN/MATLAB writers emitted a top-level ``eccodes`` (a + ``;``-joined string or a list of codes); lift it into the canonical + place. Normalises in place; no-op when no reaction carries a top-level + ``eccodes`` key. A native ``annotation['ec-code']`` (if already present) + wins and is left untouched. + """ + if not isinstance(reactions, list): + return + for rxn in reactions: + if not (isinstance(rxn, dict) and "eccodes" in rxn): + continue + codes = _eccodes_to_list(rxn.pop("eccodes")) + if not codes: + continue + annotation = rxn.get("annotation") + if not isinstance(annotation, dict): + annotation = {} + rxn["annotation"] = annotation + annotation.setdefault("ec-code", codes) + + +def _eccodes_to_list(value) -> list: + """Normalise a RAVEN ``eccodes`` value to a list of trimmed code strings. + + Accepts a ``;``-joined string (RAVEN MATLAB's ``getECstring`` form) or an + already-split list; drops empty tokens. + """ + if value is None: + return [] + items = value if isinstance(value, list) else str(value).split(";") + return [str(s).strip() for s in items if str(s).strip()] + + def _flip_legacy_prot_direction(model: cobra.Model) -> None: """Flip pre-forward-direction protein reactions in place. @@ -344,6 +434,13 @@ def write_yaml_model( _emit_entry_fields(doc.get("reactions", []), _RXN_FIELDS) _emit_entry_fields(doc.get("genes", []), _GENE_FIELDS) + # cobra's _gene_to_dict always emits `name: ''` because name is a + # required attribute; RAVEN MATLAB skips empty names. Drop the + # empty-string entry so the two writers produce identical genes. + for gene in doc.get("genes", []) or (): + if isinstance(gene, dict) and gene.get("name") == "": + gene.pop("name", None) + # ec sections come from the typed model.ec (when present), not from the # opaque foreign-keys stash. Drop any stale ec-* entries in `foreign` so # they can't conflict with the EcData-derived ones. @@ -356,24 +453,42 @@ def write_yaml_model( else None ) - # cobra dict order is metabolites, reactions, genes, id, name, compartments; - # append version / gecko_light / metaData / ec-* like RAVEN's writer. - if version is not None: - doc["version"] = version - metadata = dict(stored_meta) + # Final document order (matches RAVEN MATLAB writeYAMLmodel): + # metaData, metabolites, reactions, genes, compartments, + # gecko_light, ec-rxns, ec-enzymes, + # id/name/version live inside metaData (the RAVEN convention) — they + # are NOT emitted at root level. Cobra reading such a file recovers + # the lists and compartments and leaves model.id empty (consistent + # with how RAVEN MATLAB has always laid these files out). + metadata = OrderedDict(stored_meta) if model.id: metadata.setdefault("id", model.id) if model.name: metadata.setdefault("name", model.name) - if ec_sections is not None: - doc["gecko_light"] = ec_sections["gecko_light"] + if version is not None: + metadata.setdefault("version", version) + + # cobra's model_to_dict put id / name at root level; drop them so they + # don't duplicate the metaData copy. + doc.pop("id", None) + doc.pop("name", None) + + ordered = OrderedDict() if metadata: - doc["metaData"] = metadata + ordered["metaData"] = metadata + for key in ("metabolites", "reactions", "genes", "compartments", "notes"): + if key in doc: + ordered[key] = doc.pop(key) if ec_sections is not None: - doc["ec-rxns"] = ec_sections["ec-rxns"] - doc["ec-enzymes"] = ec_sections["ec-enzymes"] + ordered["gecko_light"] = ec_sections["gecko_light"] + ordered["ec-rxns"] = ec_sections["ec-rxns"] + ordered["ec-enzymes"] = ec_sections["ec-enzymes"] + # Carry over any cobra-shaped fields we didn't classify above + # (defensive: keeps forward compatibility with future cobra additions). + for key, value in doc.items(): + ordered.setdefault(key, value) for key, value in foreign.items(): - doc[key] = value + ordered.setdefault(key, value) with _open_text(path, "w") as handle: - _cobra_yaml.dump(doc, handle) + _cobra_yaml.dump(ordered, handle) diff --git a/src/raven_python/localization/__init__.py b/src/raven_toolbox/localization/__init__.py similarity index 87% rename from src/raven_python/localization/__init__.py rename to src/raven_toolbox/localization/__init__.py index c6071e0f..6d9f8215 100644 --- a/src/raven_python/localization/__init__.py +++ b/src/raven_toolbox/localization/__init__.py @@ -4,13 +4,13 @@ :func:`load_wolfpsort` / :func:`load_deeploc` parse predictor outputs into the ``gene × compartment`` :class:`LocalizationScores` DataFrame the algorithm consumes. """ -from raven_python.localization.predict import ( +from raven_toolbox.localization.predict import ( LocalizationProposal, LocalizationResult, apply_localization, predict_localization, ) -from raven_python.localization.scores import ( +from raven_toolbox.localization.scores import ( LocalizationScores, load_deeploc, load_wolfpsort, diff --git a/src/raven_python/localization/predict.py b/src/raven_toolbox/localization/predict.py similarity index 97% rename from src/raven_python/localization/predict.py rename to src/raven_toolbox/localization/predict.py index 0fb8596d..d0175182 100644 --- a/src/raven_python/localization/predict.py +++ b/src/raven_toolbox/localization/predict.py @@ -21,7 +21,7 @@ * Isozyme separation is *not* applied internally — a reaction with isozymes is treated as "all listed genes must share its compartment". For per-isozyme placement, call - :func:`raven_python.manipulation.expand_model` first. + :func:`raven_toolbox.manipulation.expand_model` first. * Transports are routed through ``default_compartment``. """ from __future__ import annotations @@ -31,9 +31,10 @@ import cobra import pandas as pd +from optlang.interface import Variable # untyped (resolves to Any); used for container hints from optlang.symbolics import Real, add, mul -from raven_python.localization.scores import LocalizationScores +from raven_toolbox.localization.scores import LocalizationScores @dataclass @@ -165,12 +166,12 @@ def _met_cost(m_id: str) -> float: opt = prob.Model() # x[r, c] = 1 iff reaction r placed in c (only for r ∈ placeable) - x: dict[tuple[str, str], object] = { + x: dict[tuple[str, str], Variable] = { (r.id, c): prob.Variable(f"x_{r.id}_{c}", type="binary") for r in placeable for c in compartments } # y[g, c] = 1 iff gene g assigned to c - y: dict[tuple[str, str], object] = { + y: dict[tuple[str, str], Variable] = { (g, c): prob.Variable(f"y_{g}_{c}", type="binary") for g in genes_in_scope for c in compartments } @@ -182,7 +183,7 @@ def _met_cost(m_id: str) -> float: for c in compartments: if c != default_compartment: met_keys.add((m.id, c)) - t: dict[tuple[str, str], object] = { + t: dict[tuple[str, str], Variable] = { k: prob.Variable(f"t_{k[0]}_{k[1]}", type="binary") for k in met_keys } @@ -223,7 +224,10 @@ def _met_cost(m_id: str) -> float: score_lookup = scores.df # gene_id × compartment → float for g in genes_in_scope: for c in compartments: - s = float(score_lookup.at[g, c]) if c in score_lookup.columns and not pd.isna(score_lookup.at[g, c]) else 0.0 + if c not in score_lookup.columns: + continue + val = score_lookup.at[g, c] + s = 0.0 if pd.isna(val) else float(val) if s: obj_terms.append(mul([Real(s), y[g, c]])) # − transport cost per added transport diff --git a/src/raven_python/localization/scores.py b/src/raven_toolbox/localization/scores.py similarity index 96% rename from src/raven_python/localization/scores.py rename to src/raven_toolbox/localization/scores.py index 4fabc031..2162571e 100644 --- a/src/raven_python/localization/scores.py +++ b/src/raven_toolbox/localization/scores.py @@ -1,6 +1,6 @@ """Loaders for gene → compartment localisation predictors (WoLF PSORT, DeepLoc, …). -The localisation algorithm in :mod:`raven_python.localization.predict` consumes a +The localisation algorithm in :mod:`raven_toolbox.localization.predict` consumes a *gene × compartment* score table (:class:`LocalizationScores`) where higher = stronger evidence. Each predictor produces this differently; loaders here normalise them. The format is open — a user can build a :class:`LocalizationScores` from any source by @@ -25,7 +25,7 @@ class LocalizationScores: compartment id; values are floats (higher = stronger evidence for that compartment). Genes absent from ``df`` and NaN entries are treated as "no signal" by - :func:`raven_python.localization.predict_localization` (uniform prior contribution). + :func:`raven_toolbox.localization.predict_localization` (uniform prior contribution). """ df: pd.DataFrame diff --git a/src/raven_toolbox/manifest.py b/src/raven_toolbox/manifest.py new file mode 100644 index 00000000..7eb00869 --- /dev/null +++ b/src/raven_toolbox/manifest.py @@ -0,0 +1,123 @@ +"""Load a raven-data manifest into the runtime resolver registries. + +A *manifest* is the single, language-agnostic source of truth for every downloadable +artefact (KEGG tables / HMMs, …) and external-binary bundle (BLAST / DIAMOND / HMMER). +It lives in the data repository (and/or a Zenodo record); raven-toolbox and MATLAB RAVEN +both read the same JSON and verify each file's SHA256 after download. See +``data/manifest.schema.json`` for the format and ``data/manifest.example.json`` for a +worked example. + +The manifest is a superset of the two runtime registries: + +* ``manifest["data"]`` → :data:`raven_toolbox.data._DATA_REGISTRY` +* ``manifest["binaries"]`` → :data:`raven_toolbox.binaries._REGISTRY` + +Usage:: + + from raven_toolbox import manifest + manifest.load_into_registries("https://github.com/SysBioChalmers/raven-data/releases/download/manifest-v1/manifest.json") + +or set ``RAVEN_PYTHON_MANIFEST`` (a path or URL) and the resolvers load it lazily on first +use:: + + export RAVEN_PYTHON_MANIFEST=/path/to/manifest.json +""" +from __future__ import annotations + +import json +import os +from pathlib import Path +from urllib.request import urlopen + +#: Environment variable holding a manifest path or URL; consulted lazily by the +#: resolvers when their registry is still empty. +ENV_MANIFEST = "RAVEN_PYTHON_MANIFEST" + +#: Manifest format version this module understands. +SUPPORTED_VERSION = 1 + + +def _read(source: str | os.PathLike) -> str: + """Read manifest text from a local path or an http(s)/ftp URL.""" + s = str(source) + if s.startswith(("http://", "https://", "ftp://")): + with urlopen(s) as resp: # noqa: S310 (trusted, user-supplied manifest source) + return resp.read().decode("utf-8") + return Path(s).read_text(encoding="utf-8") + + +def load_manifest(source: str | os.PathLike | None = None) -> dict: + """Read and validate a manifest from ``source`` (path/URL) or ``$RAVEN_PYTHON_MANIFEST``.""" + source = source or os.environ.get(ENV_MANIFEST) + if not source: + raise ValueError( + f"No manifest source: pass a path/URL or set ${ENV_MANIFEST}." + ) + manifest = json.loads(_read(source)) + version = manifest.get("manifest_version") + if version != SUPPORTED_VERSION: + raise ValueError( + f"Unsupported manifest_version {version!r} (this raven-toolbox understands " + f"{SUPPORTED_VERSION})." + ) + return manifest + + +def to_data_registry(manifest: dict) -> dict: + """Project ``manifest['data']`` onto the ``raven_toolbox.data._DATA_REGISTRY`` shape.""" + return { + dataset: { + "version": spec["version"], + "files": { + name: {"url": f["url"], "sha256": f["sha256"]} + for name, f in spec["files"].items() + }, + } + for dataset, spec in manifest.get("data", {}).items() + } + + +def to_binary_registry(manifest: dict) -> dict: + """Project ``manifest['binaries']`` onto the ``raven_toolbox.binaries._REGISTRY`` shape.""" + return { + bundle: { + "version": spec["version"], + "provides": list(spec["provides"]), + "platforms": { + key: {"url": f["url"], "sha256": f["sha256"]} + for key, f in spec["platforms"].items() + }, + } + for bundle, spec in manifest.get("binaries", {}).items() + } + + +def load_into_registries( + source: str | os.PathLike | None = None, *, replace: bool = False +) -> dict: + """Load a manifest and merge it into the live data/binary registries. + + Parameters + ---------- + source + Manifest path or URL; defaults to ``$RAVEN_PYTHON_MANIFEST``. + replace + If True, clear the existing registries first; otherwise merge (manifest wins). + + Returns + ------- + dict + The parsed manifest. + """ + manifest = load_manifest(source) + # Imported here (not at module top) so data/binaries can lazily call back + # into this module without an import cycle. + from raven_toolbox import binaries as _binaries + from raven_toolbox import data as _data + + if replace: + _data._DATA_REGISTRY.clear() + _binaries._REGISTRY.clear() + _data._DATA_REGISTRY.update(to_data_registry(manifest)) + _binaries._REGISTRY.update(to_binary_registry(manifest)) + return manifest diff --git a/src/raven_python/manipulation/__init__.py b/src/raven_toolbox/manipulation/__init__.py similarity index 90% rename from src/raven_python/manipulation/__init__.py rename to src/raven_toolbox/manipulation/__init__.py index 074c36fc..587643e8 100644 --- a/src/raven_python/manipulation/__init__.py +++ b/src/raven_toolbox/manipulation/__init__.py @@ -3,13 +3,14 @@ isozyme expansion, compartment merge / copy, and model merging by name.""" from .add import add_reactions_from_equations from .change import change_gene_reaction_rules, change_reaction_equations -from .expand import expand_model +from .expand import expand_model, gpr_to_dnf from .irreversible import convert_to_irreversible from .merge import merge_models from .parameters import set_variance_bounds from .remove import remove_genes, remove_metabolites from .simplify import ( constrain_reversible_reactions, + find_duplicate_reactions, group_linear_reactions, remove_dead_end_reactions, remove_duplicate_reactions, @@ -26,6 +27,8 @@ "constrain_reversible_reactions", "convert_to_irreversible", "expand_model", + "find_duplicate_reactions", + "gpr_to_dnf", "group_linear_reactions", "merge_models", "remove_dead_end_reactions", diff --git a/src/raven_python/manipulation/add.py b/src/raven_toolbox/manipulation/add.py similarity index 89% rename from src/raven_python/manipulation/add.py rename to src/raven_toolbox/manipulation/add.py index f5e6495d..ef8f7ed1 100644 --- a/src/raven_python/manipulation/add.py +++ b/src/raven_toolbox/manipulation/add.py @@ -35,7 +35,7 @@ from cobra import Metabolite, Reaction from cobra.core.gene import GPR -from raven_python.utils.parse import parse_name_comp +from raven_toolbox.utils.parse import parse_name_comp, subsystem_to_str # Reversibility arrows. ``<=>`` must be tried before ``=>`` (it contains it). _REVERSIBLE_ARROWS = ("<=>",) @@ -97,8 +97,24 @@ def _new_met_id(model: cobra.Model, prefix: str) -> str: return f"{prefix}{n}" +def _build_met_index(model: cobra.Model) -> dict[tuple[str, str | None], Metabolite]: + """Build a ``(name, compartment) -> metabolite`` index (first match wins, + mirroring the old linear scan). Lets name / name[comp] resolution be O(1) + instead of re-scanning ``model.metabolites`` per token; callers update it as + new mets are created so later tokens dedup against earlier ones.""" + index: dict[tuple[str, str | None], Metabolite] = {} + for met in model.metabolites: + index.setdefault((met.name, met.compartment), met) + return index + + def _try_existing( - model: cobra.Model, token: str, *, mets_by: str, compartment: str | None + model: cobra.Model, + token: str, + *, + mets_by: str, + compartment: str | None, + met_index: dict[tuple[str, str | None], Metabolite], ) -> Metabolite | None: """Look up ``token`` as an existing metabolite (no creation, no side effects). @@ -113,10 +129,7 @@ def _try_existing( target_comp = comp if comp is not None else compartment if target_comp is None: return None - for met in model.metabolites: - if met.name == name and met.compartment == target_comp: - return met - return None + return met_index.get((name, target_comp)) def _resolve_metabolite( @@ -127,6 +140,7 @@ def _resolve_metabolite( compartment: str | None, allow_new_mets: bool, new_met_prefix: str, + met_index: dict[tuple[str, str | None], Metabolite], ) -> Metabolite: """Resolve an equation token to an existing or newly created Metabolite.""" name, comp = parse_name_comp(token) @@ -146,6 +160,7 @@ def _resolve_metabolite( _warn_unknown_compartment(model, compartment, token) met = Metabolite(token, compartment=compartment) model.add_metabolites([met]) + met_index.setdefault((met.name, met.compartment), met) return met # name-based (mets_by="name") or explicit name[comp] @@ -158,13 +173,9 @@ def _resolve_metabolite( if comp is not None and target_comp not in model.compartments and not allow_new_mets: raise ValueError(f"Compartment {target_comp!r} is not in the model.") - matches = [ - met - for met in model.metabolites - if met.name == name and met.compartment == target_comp - ] - if matches: - return matches[0] + existing = met_index.get((name, target_comp)) + if existing is not None: + return existing if not allow_new_mets: raise ValueError( f"No metabolite named {name!r} in compartment {target_comp!r}; " @@ -173,6 +184,7 @@ def _resolve_metabolite( _warn_unknown_compartment(model, target_comp, name) met = Metabolite(_new_met_id(model, new_met_prefix), name=name, compartment=target_comp) model.add_metabolites([met]) + met_index.setdefault((met.name, met.compartment), met) return met @@ -202,6 +214,7 @@ def _stoichiometry( compartment: str | None, allow_new_mets: bool, new_met_prefix: str, + met_index: dict[tuple[str, str | None], Metabolite], ) -> tuple[dict[Metabolite, float], bool]: """Parse an equation into a {Metabolite: net coefficient} dict + reversibility.""" lhs, rhs, reversible = _split_equation(equation) @@ -217,7 +230,8 @@ def _stoichiometry( met = None if fallback is not None: met = _try_existing( - model, fallback, mets_by=mets_by, compartment=compartment + model, fallback, mets_by=mets_by, compartment=compartment, + met_index=met_index, ) if met is not None: coeff = 1.0 @@ -229,6 +243,7 @@ def _stoichiometry( compartment=compartment, allow_new_mets=allow_new_mets, new_met_prefix=new_met_prefix, + met_index=met_index, ) coeffs[met] = coeffs.get(met, 0.0) + sign * coeff # Drop metabolites that net to zero (present as both substrate and product). @@ -295,6 +310,7 @@ def add_reactions_from_equations( known_genes = {gene.id for gene in model.genes} added: list[Reaction] = [] + met_index = _build_met_index(model) for spec in reactions: if "id" not in spec: @@ -302,7 +318,9 @@ def add_reactions_from_equations( rxn_id = spec["id"] if rxn_id in model.reactions: raise ValueError( - f"Reaction {rxn_id!r} already exists; use changeRxns or remove it first." + f"Reaction {rxn_id!r} already exists. To change its stoichiometry use " + "change_reaction_equations; to replace it, remove it first with " + "model.remove_reactions([...])." ) if "equation" not in spec: raise ValueError(f"Reaction {rxn_id!r} spec missing required 'equation'.") @@ -314,6 +332,7 @@ def add_reactions_from_equations( compartment=compartment, allow_new_mets=allow_new_mets, new_met_prefix=new_met_prefix, + met_index=met_index, ) rxn = Reaction(rxn_id, name=spec.get("name", "")) @@ -324,7 +343,7 @@ def add_reactions_from_equations( lower = config.lower_bound if reversible else 0.0 rxn.bounds = (lower, config.upper_bound) if "subsystem" in spec: - rxn.subsystem = spec["subsystem"] + rxn.subsystem = subsystem_to_str(spec["subsystem"]) model.add_reactions([rxn]) rxn.add_metabolites(coeffs) diff --git a/src/raven_python/manipulation/change.py b/src/raven_toolbox/manipulation/change.py similarity index 92% rename from src/raven_python/manipulation/change.py rename to src/raven_toolbox/manipulation/change.py index 073fdfdf..01ab3f38 100644 --- a/src/raven_python/manipulation/change.py +++ b/src/raven_toolbox/manipulation/change.py @@ -4,7 +4,7 @@ bounds, GPR, subsystem, and position are preserved automatically by cobra. So this port simply re-parses the equation (reusing the same metabolite -matching as :func:`~raven_python.manipulation.add.add_reactions_from_equations`, +matching as :func:`~raven_toolbox.manipulation.add.add_reactions_from_equations`, including name and ``name[comp]`` modes that cobra lacks) and swaps the metabolites in place. @@ -18,7 +18,7 @@ import cobra from cobra import Reaction -from raven_python.manipulation.add import _stoichiometry +from raven_toolbox.manipulation.add import _build_met_index, _stoichiometry __all__ = ["change_reaction_equations", "change_gene_reaction_rules"] @@ -41,7 +41,7 @@ def change_reaction_equations( equations Mapping of ``reaction_id -> equation string``. Every ID must already exist in the model. Equation syntax is identical to - :func:`~raven_python.manipulation.add.add_reactions_from_equations`. + :func:`~raven_toolbox.manipulation.add.add_reactions_from_equations`. mets_by, compartment, allow_new_mets, new_met_prefix Metabolite-matching options, as in ``add_reactions_from_equations``. @@ -60,6 +60,7 @@ def change_reaction_equations( raise ValueError(f"mets_by must be 'id' or 'name', got {mets_by!r}") changed: list[Reaction] = [] + met_index = _build_met_index(model) for rxn_id, equation in equations.items(): if rxn_id not in model.reactions: raise ValueError(f"Reaction {rxn_id!r} not found in the model.") @@ -72,6 +73,7 @@ def change_reaction_equations( compartment=compartment, allow_new_mets=allow_new_mets, new_met_prefix=new_met_prefix, + met_index=met_index, ) rxn.subtract_metabolites(dict(rxn.metabolites), combine=True) diff --git a/src/raven_python/manipulation/compartments.py b/src/raven_toolbox/manipulation/compartments.py similarity index 97% rename from src/raven_python/manipulation/compartments.py rename to src/raven_toolbox/manipulation/compartments.py index 091d196d..2747ace3 100644 --- a/src/raven_python/manipulation/compartments.py +++ b/src/raven_toolbox/manipulation/compartments.py @@ -1,7 +1,7 @@ """Compartment manipulation — merge all compartments into one, or copy reactions to a new compartment (ports of RAVEN's ``mergeCompartments`` and ``copyToComps``). -Both functions are useful **independently of** :func:`raven_python.localization.predict_localization`: +Both functions are useful **independently of** :func:`raven_toolbox.localization.predict_localization`: ``merge_compartments`` flattens a multi-compartment model for a simplified analysis (e.g. checking whether the network can in principle make a metabolite, with no compartment topology in the way); ``copy_to_compartment`` is a building block for @@ -36,7 +36,7 @@ def merge_compartments( compartments. * As a pre-step for localisation when the user does want RAVEN's "start from scratch" workflow (call :func:`merge_compartments` then - :func:`raven_python.localization.predict_localization` with the full reaction list). + :func:`raven_toolbox.localization.predict_localization` with the full reaction list). Metabolites that already share a base id (e.g. ``glc__D_c`` and ``glc__D_e`` both map to ``glc__D``) collapse into one entity in the merged compartment; their diff --git a/src/raven_python/manipulation/expand.py b/src/raven_toolbox/manipulation/expand.py similarity index 94% rename from src/raven_python/manipulation/expand.py rename to src/raven_toolbox/manipulation/expand.py index 246f3b99..960516de 100644 --- a/src/raven_python/manipulation/expand.py +++ b/src/raven_toolbox/manipulation/expand.py @@ -4,11 +4,11 @@ Provenance: this implementation was first written for geckopy (``geckopy/ec_model/pipeline/expand.py``, where it backed makeEcModel stage 5) -and is adopted here as its canonical home; geckopy will import it from raven_python -once raven_python is published. +and is adopted here as its canonical home; geckopy will import it from raven_toolbox +once raven_toolbox is published. MATLAB-COMPAT: GECKO MATLAB and RAVEN ``expandModel.m`` use string manipulation -on grRules to detect and split isozymes. raven_python uses cobrapy's GPR AST +on grRules to detect and split isozymes. raven_toolbox uses cobrapy's GPR AST instead. Output should be equivalent for any well-formed GPR; cases that differ are likely malformed GPR strings that the AST flags as invalid. """ @@ -20,8 +20,10 @@ import cobra from cobra.core.gene import GPR +__all__ = ["expand_model", "gpr_to_dnf"] -def _gpr_to_dnf(gpr: GPR) -> list[list[str]]: + +def gpr_to_dnf(gpr: GPR) -> list[list[str]]: """Convert a GPR to disjunctive normal form (list of AND-clauses). An empty GPR yields an empty list. A single clause (no OR anywhere) @@ -88,7 +90,7 @@ def expand_model(model: cobra.Model) -> list[str]: for rxn in model.reactions: if not rxn.gene_reaction_rule: continue - clauses = _gpr_to_dnf(rxn.gpr) + clauses = gpr_to_dnf(rxn.gpr) if len(clauses) <= 1: continue expansions.append((rxn, clauses)) diff --git a/src/raven_python/manipulation/irreversible.py b/src/raven_toolbox/manipulation/irreversible.py similarity index 97% rename from src/raven_python/manipulation/irreversible.py rename to src/raven_toolbox/manipulation/irreversible.py index 3f64a68f..5c6c3dcf 100644 --- a/src/raven_python/manipulation/irreversible.py +++ b/src/raven_toolbox/manipulation/irreversible.py @@ -6,7 +6,7 @@ Provenance: first written for geckopy (``geckopy/ec_model/pipeline/preprocess.py``, makeEcModel stage 4, tagged "RAVENpy candidate") and adopted here as its canonical home; geckopy will -import it from raven_python once raven_python is published. +import it from raven_toolbox once raven_toolbox is published. """ from __future__ import annotations diff --git a/src/raven_python/manipulation/merge.py b/src/raven_toolbox/manipulation/merge.py similarity index 100% rename from src/raven_python/manipulation/merge.py rename to src/raven_toolbox/manipulation/merge.py diff --git a/src/raven_python/manipulation/parameters.py b/src/raven_toolbox/manipulation/parameters.py similarity index 100% rename from src/raven_python/manipulation/parameters.py rename to src/raven_toolbox/manipulation/parameters.py diff --git a/src/raven_python/manipulation/remove.py b/src/raven_toolbox/manipulation/remove.py similarity index 99% rename from src/raven_python/manipulation/remove.py rename to src/raven_toolbox/manipulation/remove.py index 492de36b..43f966fb 100644 --- a/src/raven_python/manipulation/remove.py +++ b/src/raven_toolbox/manipulation/remove.py @@ -99,7 +99,7 @@ def remove_genes( return [] # Reactions touched by these genes that currently have a GPR. - affected = set() + affected: set[str] = set() for gid in present: affected.update(r.id for r in model.genes.get_by_id(gid).reactions) had_gpr = {rid for rid in affected if model.reactions.get_by_id(rid).gene_reaction_rule} diff --git a/src/raven_python/manipulation/simplify.py b/src/raven_toolbox/manipulation/simplify.py similarity index 82% rename from src/raven_python/manipulation/simplify.py rename to src/raven_toolbox/manipulation/simplify.py index 2deaccdd..fe784b3b 100644 --- a/src/raven_python/manipulation/simplify.py +++ b/src/raven_toolbox/manipulation/simplify.py @@ -19,7 +19,7 @@ import cobra from cobra.flux_analysis import flux_variability_analysis -from raven_python.manipulation.irreversible import convert_to_irreversible +from raven_toolbox.manipulation.irreversible import convert_to_irreversible def _prune_orphan_metabolites(model: cobra.Model) -> list[str]: @@ -82,6 +82,54 @@ def _signature(rxn): return (mets, rxn.lower_bound, rxn.upper_bound, rxn.objective_coefficient) +def _stoich_signature(rxn, *, ignore_direction: bool) -> frozenset: + """Signature considering stoichiometry only (used by find_duplicate_reactions). + + When ``ignore_direction`` is True, ``A → B`` and ``B → A`` (the same + reaction with all coefficients negated) share a signature. Both + orientations are accumulated and the lexicographically smaller one + wins so the dict-key lookup is direction-symmetric. + """ + forward = frozenset((m.id, c) for m, c in rxn.metabolites.items()) + if not ignore_direction: + return forward + backward = frozenset((m.id, -c) for m, c in rxn.metabolites.items()) + return min(forward, backward, key=lambda s: sorted(s)) + + +def find_duplicate_reactions( + model: cobra.Model, + *, + ignore_direction: bool = True, +) -> list[list[cobra.Reaction]]: + """Return groups of reactions that share identical stoichiometry. + + Detection-only counterpart to :func:`remove_duplicate_reactions`. + Bounds, objective coefficients, GPRs and annotations are ignored — + only stoichiometry is compared, mirroring the legacy yeast-GEM + ``findDuplicatedRxns`` and matching the typical curation use case + (find reactions that *could* be merged). + + Parameters + ---------- + ignore_direction + When True (default), ``A → B`` and ``B → A`` are treated as + duplicates (yeast-GEM's convention). Set ``False`` to require + identical orientation. + + Returns + ------- + A list of duplicate groups. Each group is itself a list with + ≥ 2 reactions sharing the same stoichiometry. Reactions that have + no duplicate are omitted. + """ + groups: dict[frozenset, list[cobra.Reaction]] = {} + for rxn in model.reactions: + sig = _stoich_signature(rxn, ignore_direction=ignore_direction) + groups.setdefault(sig, []).append(rxn) + return [g for g in groups.values() if len(g) >= 2] + + def remove_duplicate_reactions( model: cobra.Model, *, reserved: Iterable[str] | None = None ) -> list[str]: diff --git a/src/raven_python/manipulation/transfer.py b/src/raven_toolbox/manipulation/transfer.py similarity index 98% rename from src/raven_python/manipulation/transfer.py rename to src/raven_toolbox/manipulation/transfer.py index 6534e385..733c2d95 100644 --- a/src/raven_python/manipulation/transfer.py +++ b/src/raven_toolbox/manipulation/transfer.py @@ -17,7 +17,7 @@ import cobra from cobra import Metabolite, Reaction -from raven_python.manipulation.add import _new_met_id +from raven_toolbox.manipulation.add import _new_met_id def _name_comp(met: Metabolite) -> str: diff --git a/src/raven_python/manipulation/transport.py b/src/raven_toolbox/manipulation/transport.py similarity index 91% rename from src/raven_python/manipulation/transport.py rename to src/raven_toolbox/manipulation/transport.py index d2223773..bde2bebb 100644 --- a/src/raven_python/manipulation/transport.py +++ b/src/raven_toolbox/manipulation/transport.py @@ -15,7 +15,7 @@ import cobra from cobra import Metabolite, Reaction -from raven_python.manipulation.add import _new_met_id +from raven_toolbox.manipulation.add import _new_met_id def _index_by_name(mets: Iterable[Metabolite], compartment: str) -> dict[str, Metabolite]: @@ -46,11 +46,15 @@ def _transport_id_factory(model: cobra.Model, prefix: str): def next_id() -> str: nonlocal counter - while f"{prefix}{counter:04d}" in model.reactions: + # A free id must appear within len(reactions)+1 tries (only that many are + # occupied); the bound turns any pathological case into a clear error + # instead of spinning forever. + for _ in range(len(model.reactions) + 2): + rid = f"{prefix}{counter:04d}" counter += 1 - rid = f"{prefix}{counter:04d}" - counter += 1 - return rid + if rid not in model.reactions: + return rid + raise RuntimeError(f"could not allocate a free reaction id with prefix {prefix!r}.") return next_id diff --git a/src/raven_python/omics/__init__.py b/src/raven_toolbox/omics/__init__.py similarity index 75% rename from src/raven_python/omics/__init__.py rename to src/raven_toolbox/omics/__init__.py index 166b0209..5b2d54f8 100644 --- a/src/raven_python/omics/__init__.py +++ b/src/raven_toolbox/omics/__init__.py @@ -1,8 +1,8 @@ """Omics integration — HPA proteomics + RNA-seq parsing and gene-scoring adapters. -Entry point for tissue-specific (f)tINIT runs. See :mod:`raven_python.omics.hpa`. +Entry point for tissue-specific (f)tINIT runs. See :mod:`raven_toolbox.omics.hpa`. """ -from raven_python.omics.hpa import ( +from raven_toolbox.omics.hpa import ( HPA_LEVEL_SCORES, HPAData, HPARnaData, diff --git a/src/raven_python/omics/hpa.py b/src/raven_toolbox/omics/hpa.py similarity index 96% rename from src/raven_python/omics/hpa.py rename to src/raven_toolbox/omics/hpa.py index 59caf933..afa10a31 100644 --- a/src/raven_python/omics/hpa.py +++ b/src/raven_toolbox/omics/hpa.py @@ -5,7 +5,7 @@ detected) plus reliability flags, and an **RNA-seq** table (``rna_tissue_consensus.tsv`` / ``rna_tissue_gtex.tsv``) with per-tissue *TPM* values. Both are returned as tidy :class:`pandas.DataFrame`\\ s; the scoring adapters delegate the GPR walk to -:func:`raven_python.init.score.score_reactions_from_genes` so there is one source of truth +:func:`raven_toolbox.init.score.score_reactions_from_genes` so there is one source of truth for reaction scoring. Pipeline (typical (f)tINIT entry): @@ -33,7 +33,7 @@ import pandas as pd -from raven_python.init.score import gene_scores_from_expression +from raven_toolbox.init.score import gene_scores_from_expression # RAVEN's hpaLevelScores defaults (scoreModel.m). HPA reports either antibody-staining # levels (Strong/Moderate/Weak/Negative) or "APE" classes (High/Medium/Low/Not detected / @@ -80,7 +80,7 @@ def tissues(self) -> list[str]: def expression(self, tissue: str) -> dict[str, float]: """{gene_id: TPM} for ``tissue``. Use this directly with - :func:`raven_python.init.score.gene_scores_from_expression`.""" + :func:`raven_toolbox.init.score.gene_scores_from_expression`.""" sub = self.df.loc[self.df["tissue"] == tissue, ["gene_id", "tpm"]] return dict(zip(sub["gene_id"], sub["tpm"], strict=True)) @@ -171,7 +171,7 @@ def rna_gene_scores( ) -> dict[str, float]: """Numeric gene scores from HPA RNA-seq TPM for one ``tissue``. - Thin wrapper over :func:`raven_python.init.score.gene_scores_from_expression` (the same + Thin wrapper over :func:`raven_toolbox.init.score.gene_scores_from_expression` (the same ``5·ln(TPM/reference)``-clamped scoring used elsewhere): selects the tissue, derives a reference if none is given (per-gene mean TPM across all tissues — RAVEN's default for ``arrayData.threshold``), and returns ``{gene_id: score}``. diff --git a/src/raven_toolbox/py.typed b/src/raven_toolbox/py.typed new file mode 100644 index 00000000..e69de29b diff --git a/src/raven_python/reconstruction/__init__.py b/src/raven_toolbox/reconstruction/__init__.py similarity index 100% rename from src/raven_python/reconstruction/__init__.py rename to src/raven_toolbox/reconstruction/__init__.py diff --git a/src/raven_python/reconstruction/homology/__init__.py b/src/raven_toolbox/reconstruction/homology/__init__.py similarity index 55% rename from src/raven_python/reconstruction/homology/__init__.py rename to src/raven_toolbox/reconstruction/homology/__init__.py index 6ed9748f..836403d6 100644 --- a/src/raven_python/reconstruction/homology/__init__.py +++ b/src/raven_toolbox/reconstruction/homology/__init__.py @@ -1,11 +1,15 @@ """Homology-based reconstruction from template models (getModelFromHomology, BLAST/DIAMOND).""" -from raven_python.reconstruction.homology.blast import ( +from raven_toolbox.reconstruction.homology.blast import ( blast_from_table, run_blast, run_diamond, ) -from raven_python.reconstruction.homology.hits import HIT_COLUMNS, make_ortholog_hits, validate_hits -from raven_python.reconstruction.homology.homology import HomologyResult, get_model_from_homology +from raven_toolbox.reconstruction.homology.hits import ( + HIT_COLUMNS, + make_ortholog_hits, + validate_hits, +) +from raven_toolbox.reconstruction.homology.homology import HomologyResult, get_model_from_homology __all__ = [ "HIT_COLUMNS", diff --git a/src/raven_python/reconstruction/homology/blast.py b/src/raven_toolbox/reconstruction/homology/blast.py similarity index 91% rename from src/raven_python/reconstruction/homology/blast.py rename to src/raven_toolbox/reconstruction/homology/blast.py index 246ddabf..e7432c01 100644 --- a/src/raven_python/reconstruction/homology/blast.py +++ b/src/raven_toolbox/reconstruction/homology/blast.py @@ -1,8 +1,8 @@ """Run BLAST+ / DIAMOND (or load precomputed hits) into a homology hits table. Each producer returns the bidirectional hits DataFrame (``HIT_COLUMNS``) consumed by -:func:`~raven_python.reconstruction.homology.get_model_from_homology`. Binaries are -located via :func:`raven_python.binaries.resolve_binary` (arg → env → PATH → bundled). +:func:`~raven_toolbox.reconstruction.homology.get_model_from_homology`. Binaries are +located via :func:`raven_toolbox.binaries.resolve_binary` (arg → env → PATH → bundled). """ from __future__ import annotations @@ -14,8 +14,8 @@ import pandas as pd -from raven_python.binaries import resolve_binary -from raven_python.reconstruction.homology.hits import HIT_COLUMNS, validate_hits +from raven_toolbox.binaries import resolve_binary +from raven_toolbox.reconstruction.homology.hits import HIT_COLUMNS, validate_hits # Tabular output columns requested from BLAST+/DIAMOND, in order. _OUTFMT_FIELDS = ["qseqid", "sseqid", "evalue", "pident", "length", "bitscore", "ppos"] @@ -72,8 +72,8 @@ def run_blast( outfmt = "10 " + " ".join(_OUTFMT_FIELDS) # 10 = CSV frames = [] - with tempfile.TemporaryDirectory() as tmp: - tmp = Path(tmp) + with tempfile.TemporaryDirectory() as tmp_dir: + tmp = Path(tmp_dir) def blastp_dir(query, subject_fasta, from_id, to_id): db = tmp / f"db_{from_id}_{to_id}" @@ -113,8 +113,8 @@ def run_diamond( diamond = resolve_binary("diamond", binary=diamond) frames = [] - with tempfile.TemporaryDirectory() as tmp: - tmp = Path(tmp) + with tempfile.TemporaryDirectory() as tmp_dir: + tmp = Path(tmp_dir) def diamond_dir(query, subject_fasta, from_id, to_id): db = tmp / f"db_{from_id}_{to_id}" diff --git a/src/raven_python/reconstruction/homology/hits.py b/src/raven_toolbox/reconstruction/homology/hits.py similarity index 100% rename from src/raven_python/reconstruction/homology/hits.py rename to src/raven_toolbox/reconstruction/homology/hits.py diff --git a/src/raven_python/reconstruction/homology/homology.py b/src/raven_toolbox/reconstruction/homology/homology.py similarity index 95% rename from src/raven_python/reconstruction/homology/homology.py rename to src/raven_toolbox/reconstruction/homology/homology.py index bc6fa412..ad1d017e 100644 --- a/src/raven_python/reconstruction/homology/homology.py +++ b/src/raven_toolbox/reconstruction/homology/homology.py @@ -19,8 +19,8 @@ import cobra import pandas as pd -from raven_python.manipulation.merge import merge_models -from raven_python.reconstruction.homology.hits import validate_hits +from raven_toolbox.manipulation.merge import merge_models +from raven_toolbox.reconstruction.homology.hits import validate_hits @dataclass @@ -153,8 +153,14 @@ def _ortholog_map( if pairs.empty: return {} - # Keep only template genes that actually exist in their model. - pairs = pairs[pairs.apply(lambda r: r.template_gene in model_genes.get(r.model_id, ()), axis=1)] + # Keep only template genes that actually exist in their model. A list + # comprehension over the columns avoids DataFrame.apply(axis=1)'s per-row + # Series construction (model_genes values are sets, so membership is O(1)). + keep = [ + tg in model_genes.get(mid, ()) + for mid, tg in zip(pairs.model_id, pairs.template_gene, strict=True) + ] + pairs = pairs[keep] ortho: dict = {} for model_id, template_gene, new_gene in zip(pairs.model_id, pairs.template_gene, pairs.new_gene, strict=True): diff --git a/src/raven_python/reconstruction/kegg/__init__.py b/src/raven_toolbox/reconstruction/kegg/__init__.py similarity index 75% rename from src/raven_python/reconstruction/kegg/__init__.py rename to src/raven_toolbox/reconstruction/kegg/__init__.py index 5d27602b..af7033df 100644 --- a/src/raven_python/reconstruction/kegg/__init__.py +++ b/src/raven_toolbox/reconstruction/kegg/__init__.py @@ -4,21 +4,21 @@ (:mod:`.parse`), 3b.3 HMM libraries (:mod:`.hmm`, :mod:`.taxonomy`). Runtime: 3b.4 model for a KEGG species (:mod:`.organism`). """ -from raven_python.reconstruction.kegg.download import ( +from raven_toolbox.reconstruction.kegg.download import ( download_kegg_dump, extract_kegg_dump, fetch_kegg_files, ) -from raven_python.reconstruction.kegg.hmm import ( +from raven_toolbox.reconstruction.kegg.hmm import ( build_hmm_library, build_ko_fastas, build_ko_hmm, ) -from raven_python.reconstruction.kegg.organism import ( +from raven_toolbox.reconstruction.kegg.organism import ( get_kegg_model_for_organism, get_kegg_model_for_organism_from_artefacts, ) -from raven_python.reconstruction.kegg.parse import ( +from raven_toolbox.reconstruction.kegg.parse import ( KeggCompound, KeggKO, KeggReaction, @@ -32,23 +32,27 @@ stream_organism_gene_ko, write_kegg_tables, ) -from raven_python.reconstruction.kegg.query import ( +from raven_toolbox.reconstruction.kegg.query import ( assign_kos, get_kegg_model_from_sequences, get_kegg_model_from_sequences_with_artefacts, - parse_hmmscan_tblout, - run_hmmscan, + parse_hmmsearch_tblout, + run_hmmsearch, ) -from raven_python.reconstruction.kegg.taxonomy import ( +from raven_toolbox.reconstruction.kegg.taxonomy import ( + PhylDist, organism_domains, organisms_in_domain, parse_taxonomy, + parse_taxonomy_records, + phyl_dist, ) __all__ = [ "KeggCompound", "KeggKO", "KeggReaction", + "PhylDist", "assign_kos", "build_hmm_library", "build_kegg_tables", @@ -64,14 +68,16 @@ "get_kegg_model_from_sequences_with_artefacts", "organism_domains", "organisms_in_domain", - "parse_hmmscan_tblout", + "parse_hmmsearch_tblout", "parse_kegg_compounds", "parse_kegg_dump", "parse_kegg_kos", "parse_kegg_reactions", "parse_taxonomy", + "parse_taxonomy_records", + "phyl_dist", "read_kegg_table", - "run_hmmscan", + "run_hmmsearch", "stream_organism_gene_ko", "write_kegg_tables", ] diff --git a/src/raven_python/reconstruction/kegg/assemble.py b/src/raven_toolbox/reconstruction/kegg/assemble.py similarity index 100% rename from src/raven_python/reconstruction/kegg/assemble.py rename to src/raven_toolbox/reconstruction/kegg/assemble.py diff --git a/src/raven_python/reconstruction/kegg/download.py b/src/raven_toolbox/reconstruction/kegg/download.py similarity index 90% rename from src/raven_python/reconstruction/kegg/download.py rename to src/raven_toolbox/reconstruction/kegg/download.py index 8bb1826a..55095501 100644 --- a/src/raven_python/reconstruction/kegg/download.py +++ b/src/raven_toolbox/reconstruction/kegg/download.py @@ -15,7 +15,7 @@ Typical use (run once per KEGG release):: - from raven_python.reconstruction.kegg import download_kegg_dump, parse_kegg_dump + from raven_toolbox.reconstruction.kegg import download_kegg_dump, parse_kegg_dump download_kegg_dump("keggdb") # -> keggdb/{reaction,compound,ko,...} parse_kegg_dump("keggdb", "artefacts") # -> reference model + gzipped TSVs @@ -27,12 +27,15 @@ from __future__ import annotations import gzip +import logging import netrc import shutil import tarfile import urllib.request from pathlib import Path +logger = logging.getLogger(__name__) + KEGG_HOST = "ftp.kegg.net" BASE_URL = "https://ftp.kegg.net" @@ -66,7 +69,14 @@ def _resolve_auth( f"No credentials given and {path} does not exist. Create it (chmod 600) " f"with a line:\n machine {host} login YOUR_USER password YOUR_PASS" ) - creds = netrc.netrc(str(path)).authenticators(host) + try: + creds = netrc.netrc(str(path)).authenticators(host) + except (netrc.NetrcParseError, OSError) as exc: + raise ValueError( + f"Could not read credentials from {path}: {exc}. Ensure it is a valid " + f".netrc (chmod 600) with a line:\n" + f" machine {host} login YOUR_USER password YOUR_PASS" + ) from exc if not creds: raise ValueError( f"No credentials for '{host}' in {path}. Add a line:\n" @@ -116,12 +126,12 @@ def fetch_kegg_files( target = dest / Path(path).name if target.exists() and not force: if verbose: - print(f" skip (exists): {target.name}") + logger.info("skip (exists): %s", target.name) out.append(target) continue url = f"{base_url.rstrip('/')}/{path.lstrip('/')}" if verbose: - print(f" fetching {path}") + logger.info("fetching %s", path) with opener.open(url) as resp, open(target, "wb") as handle: shutil.copyfileobj(resp, handle) out.append(target) @@ -192,6 +202,8 @@ def lift(rel: str, tmp: str) -> Path | None: f"KEGG archives did not yield required file(s): {missing}. " f"Check that the source .tar.gz archives are present in {dest}." ) + # Narrowing for the type checker: the `missing` check above raised if any were None. + assert reaction is not None and compound is not None and ko is not None shutil.move(str(reaction), str(dest / "reaction")) shutil.move(str(ko), str(dest / "ko")) @@ -240,7 +252,7 @@ def download_kegg_dump( Convenience wrapper chaining :func:`fetch_kegg_files` and :func:`extract_kegg_dump`. Returns the logical-name -> path mapping of the - arranged dump, ready for :func:`raven_python.reconstruction.kegg.parse_kegg_dump`. + arranged dump, ready for :func:`raven_toolbox.reconstruction.kegg.parse_kegg_dump`. """ fetch_kegg_files( dest, @@ -253,5 +265,5 @@ def download_kegg_dump( verbose=verbose, ) if verbose: - print(">>> Extracting and arranging KEGG dump...") + logger.info("Extracting and arranging KEGG dump...") return extract_kegg_dump(dest) diff --git a/src/raven_python/reconstruction/kegg/hmm.py b/src/raven_toolbox/reconstruction/kegg/hmm.py similarity index 95% rename from src/raven_python/reconstruction/kegg/hmm.py rename to src/raven_toolbox/reconstruction/kegg/hmm.py index 0e210b6e..824b4168 100644 --- a/src/raven_python/reconstruction/kegg/hmm.py +++ b/src/raven_toolbox/reconstruction/kegg/hmm.py @@ -12,13 +12,13 @@ 3. **MAFFT** — multiple-sequence alignment (``--auto --anysymbol``). 4. **hmmbuild** — train the profile HMM. -Finally the per-KO HMMs are concatenated and ``hmmpress``-ed into a single searchable -library: a single ``hmmscan`` against the pressed database replaces a per-KO sweep with -``hmmsearch``. +Finally the per-KO HMMs are concatenated into a single library: end users search it +with a single ``hmmsearch`` over the multi-profile query (the fast direction), replacing +RAVEN's per-KO ``hmmsearch`` sweep — one invocation, no ``hmmpress``. The pure parts (FASTA indexing/grouping, command construction, CD-HIT ``-n`` choice) are unit-tested; running the binaries needs HMMER/MAFFT/CD-HIT, located -via :func:`raven_python.binaries.resolve_binary`. +via :func:`raven_toolbox.binaries.resolve_binary`. """ from __future__ import annotations @@ -33,8 +33,8 @@ import pandas as pd -from raven_python.binaries import resolve_binary -from raven_python.reconstruction.kegg.taxonomy import organisms_in_domain +from raven_toolbox.binaries import resolve_binary +from raven_toolbox.reconstruction.kegg.taxonomy import organisms_in_domain logger = logging.getLogger(__name__) @@ -245,7 +245,7 @@ def _hmmbuild_cmd( hmmbuild: str, out_hmm: Path, aligned: Path, threads: int, name: str | None = None ) -> list[str]: cmd = [hmmbuild, "--cpu", str(threads)] - if name: # name the profile after its KO so hmmscan targets are KO ids + if name: # name the profile after its KO so it is the query name in hmmsearch output cmd += ["-n", name] cmd += [str(out_hmm), str(aligned)] return cmd @@ -329,8 +329,8 @@ def build_ko_hmm( logger.info("[%s] start: %d sequences", label, n) hmmbuild = resolve_binary("hmmbuild", binary=hmmbuild) - with tempfile.TemporaryDirectory() as tmp: - tmp = Path(tmp) + with tempfile.TemporaryDirectory() as tmp_dir: + tmp = Path(tmp_dir) if n == 1: if verbose: logger.info("[%s] single sequence: skipping CD-HIT/MAFFT", label) @@ -403,18 +403,17 @@ def build_hmm_library( threads: int = 1, fast: bool = True, verbose: bool = False, - press: bool = True, + concatenate: bool = True, cdhit: str | Path | None = None, mafft: str | Path | None = None, hmmbuild: str | Path | None = None, - hmmpress: str | Path | None = None, -) -> dict[str, Path | list[Path]]: +) -> dict[str, Path | list[Path] | None]: """Build a domain (``"prokaryotes"``/``"eukaryotes"``) HMM library. Restricts genes to the domain's organisms (from ``taxonomy``), builds a - multi-FASTA and a profile HMM per KO under ``out_dir``, and (if ``press``) - concatenates them into ``out_dir/library.hmm`` and ``hmmpress``-es it for fast - ``hmmscan`` querying. Returns ``{"hmms": [...], "library": path | None}``. + multi-FASTA and a profile HMM per KO under ``out_dir``, and (if ``concatenate``) + concatenates them into ``out_dir/library.hmm`` — searched directly with + ``hmmsearch`` (no ``hmmpress`` needed). Returns ``{"hmms": [...], "library": path | None}``. Heavy and binary-dependent — intended for the maintainer, run once per KEGG release. Skips KOs that already have an ``.hmm`` (resumable). @@ -442,12 +441,11 @@ def build_hmm_library( hmms.append(out_hmm) library: Path | None = None - if press and hmms: + if concatenate and hmms: library = out_dir / "library.hmm" with open(library, "wb") as out: for hmm in sorted(hmms): with open(hmm, "rb") as fh: shutil.copyfileobj(fh, out) - _run([resolve_binary("hmmpress", binary=hmmpress), "-f", str(library)]) return {"hmms": hmms, "library": library} diff --git a/src/raven_python/reconstruction/kegg/organism.py b/src/raven_toolbox/reconstruction/kegg/organism.py similarity index 78% rename from src/raven_python/reconstruction/kegg/organism.py rename to src/raven_toolbox/reconstruction/kegg/organism.py index 9f305755..b079ad6c 100644 --- a/src/raven_python/reconstruction/kegg/organism.py +++ b/src/raven_toolbox/reconstruction/kegg/organism.py @@ -26,10 +26,10 @@ import cobra import pandas as pd -from raven_python.io.yaml import read_yaml_model -from raven_python.reconstruction.kegg.assemble import _DOMAINS, assemble_model_from_ko_genes -from raven_python.reconstruction.kegg.parse import read_kegg_table -from raven_python.reconstruction.kegg.taxonomy import organisms_in_domain +from raven_toolbox.io.yaml import read_yaml_model +from raven_toolbox.reconstruction.kegg.assemble import _DOMAINS, assemble_model_from_ko_genes +from raven_toolbox.reconstruction.kegg.parse import _resolve_artefact, read_kegg_table +from raven_toolbox.reconstruction.kegg.taxonomy import organisms_in_domain _NOTE = "Included by get_kegg_model_for_organism (no HMMs)" @@ -125,6 +125,7 @@ def get_kegg_model_for_organism_from_artefacts( artefact_dir: str | Path | None = None, *, version: str | None = None, + taxonomy: str | Path | None = None, **kwargs, ) -> cobra.Model: """Load the published 3b.2 artefacts from ``artefact_dir`` and build the model. @@ -132,22 +133,36 @@ def get_kegg_model_for_organism_from_artefacts( Reads ``reference_model.yml.gz`` and the ``ko_reaction``/``organism_gene_ko``/ ``rxn_flags`` gzipped-TSV tables, then calls :func:`get_kegg_model_for_organism`. If ``artefact_dir`` is ``None`` the published artefacts are fetched/cached via - :func:`raven_python.data.ensure_kegg_data` (``version`` selects the release). + :func:`raven_toolbox.data.ensure_kegg_data` (``version`` selects the release). + + For a whole-domain model (``organism_id`` = ``"prokaryotes"``/``"eukaryotes"``) + the KEGG ``taxonomy`` artefact is resolved automatically — from ``artefact_dir`` + if present, otherwise fetched via :func:`raven_toolbox.data.ensure_kegg_taxonomy` + (it is a separate artefact, not part of the core set) — unless an explicit + ``taxonomy`` path is given. """ if artefact_dir is None: - from raven_python.data import ensure_kegg_data + from raven_toolbox.data import ensure_kegg_data artefact_dir = ensure_kegg_data(version=version) artefact_dir = Path(artefact_dir) - reference_model = read_yaml_model(artefact_dir / "reference_model.yml.gz") - ko_reaction = read_kegg_table(artefact_dir / "ko_reaction.tsv.gz") - organism_gene_ko = read_kegg_table(artefact_dir / "organism_gene_ko.tsv.xz") - rxn_flags = read_kegg_table(artefact_dir / "rxn_flags.tsv.gz") + if taxonomy is None and organism_id.lower() in _DOMAINS: + try: + taxonomy = _resolve_artefact(artefact_dir, "taxonomy.gz") + except FileNotFoundError: + from raven_toolbox.data import ensure_kegg_taxonomy + + taxonomy = ensure_kegg_taxonomy(version=version) + reference_model = read_yaml_model(_resolve_artefact(artefact_dir, "reference_model.yml.gz")) + ko_reaction = read_kegg_table(_resolve_artefact(artefact_dir, "ko_reaction.tsv.gz")) + organism_gene_ko = read_kegg_table(_resolve_artefact(artefact_dir, "organism_gene_ko.tsv.gz")) + rxn_flags = read_kegg_table(_resolve_artefact(artefact_dir, "rxn_flags.tsv.gz")) return get_kegg_model_for_organism( organism_id, reference_model, ko_reaction, organism_gene_ko, rxn_flags=rxn_flags, + taxonomy=taxonomy, **kwargs, ) diff --git a/src/raven_python/reconstruction/kegg/parse.py b/src/raven_toolbox/reconstruction/kegg/parse.py similarity index 88% rename from src/raven_python/reconstruction/kegg/parse.py rename to src/raven_toolbox/reconstruction/kegg/parse.py index 3ecd6f4f..b3485c64 100644 --- a/src/raven_python/reconstruction/kegg/parse.py +++ b/src/raven_toolbox/reconstruction/kegg/parse.py @@ -1,6 +1,6 @@ """Parse a local KEGG flat-file dump into a reference model + relational tables. -Maintainer-side, build-time tooling. Produces the published raven_python KEGG artefacts: +Maintainer-side, build-time tooling. Produces the published raven_toolbox KEGG artefacts: * a **gene-free reference GEM** (reactions + metabolites only) as a ``cobra.Model``; * minimal **relational tables** (``pandas.DataFrame``) written as gzipped TSV — @@ -28,7 +28,6 @@ import gzip import heapq -import lzma import re import tempfile from collections.abc import Iterator @@ -38,7 +37,7 @@ import cobra import pandas as pd -from raven_python.io.yaml import write_yaml_model +from raven_toolbox.io.yaml import write_yaml_model # A KEGG entry id is the first token after the 12-char ENTRY label (6 chars: # R00010, C00001, K01194, ...). @@ -444,17 +443,21 @@ def build_kegg_tables( } -def write_kegg_tables(tables: dict[str, pd.DataFrame], out_dir: str | Path) -> list[Path]: - """Write each table as a gzipped TSV (``.tsv.gz``) into ``out_dir``. +def write_kegg_tables( + tables: dict[str, pd.DataFrame], out_dir: str | Path, *, prefix: str = "" +) -> list[Path]: + """Write each table as a gzipped TSV (``.tsv.gz``) into ``out_dir``. Gzipped TSV is the dependency-free cross-language format shared with MATLAB - RAVEN (see docs/kegg_data_format.md). Returns the written paths. + RAVEN (see docs/kegg_data_format.md) — readable by MATLAB's built-in ``gunzip`` + with no external tool. ``prefix`` version-tags the filenames (e.g. + ``kegg116_``). Returns the written paths. """ out_dir = Path(out_dir) out_dir.mkdir(parents=True, exist_ok=True) written = [] for name, frame in tables.items(): - path = out_dir / f"{name}.tsv.gz" + path = out_dir / f"{prefix}{name}.tsv.gz" with gzip.open(path, "wt", encoding="utf-8", newline="") as handle: frame.to_csv(handle, sep="\t", index=False) written.append(path) @@ -465,13 +468,35 @@ def read_kegg_table(path: str | Path) -> pd.DataFrame: """Read a KEGG table written by :func:`write_kegg_tables` or :func:`stream_organism_gene_ko`. - Compression is inferred from the suffix, so both the gzipped small tables - (``.tsv.gz``) and the xz-compressed ``organism_gene_ko.tsv.xz`` are read - transparently. + Compression is inferred from the suffix; all published tables are gzipped TSV + (``.tsv.gz``), and a version-prefixed name (``kegg116_.tsv.gz``) reads + just the same. """ return pd.read_csv(path, sep="\t", dtype=str, keep_default_na=False) +def _resolve_artefact(directory: str | Path, base: str) -> Path: + """Locate an artefact file by its base name, tolerating a version prefix. + + Returns ``directory/base`` if it exists, else the single version-prefixed + match ``directory/_`` (the published asset name, e.g. + ``kegg116_ko_reaction.tsv.gz``). This lets one reader consume both a user's own + unprefixed build directory and the version-pinned download cache. + """ + directory = Path(directory) + exact = directory / base + if exact.exists(): + return exact + matches = sorted(directory.glob(f"*_{base}")) + if len(matches) == 1: + return matches[0] + if not matches: + raise FileNotFoundError( + f"{base!r} (or a _ prefixed form) not found in {directory}" + ) + raise ValueError(f"Ambiguous {base!r} in {directory}: {sorted(p.name for p in matches)}") + + def _flush_sorted_run(rows: list[str], tmp_dir: Path, run_no: int) -> Path: """Sort a buffer of ``organism\\tgene\\tko\\n`` lines and write one gzipped run.""" rows.sort(key=_ogk_sort_key) @@ -490,14 +515,15 @@ def _ogk_sort_key(line: str) -> tuple[str, str]: def stream_organism_gene_ko( kegg_dir: str | Path, keep: set[str], ogk_path: str | Path, *, chunk_rows: int = 1_000_000 ) -> pd.DataFrame: - """Stream the ``ko`` file to a sorted, xz-compressed ``organism_gene_ko.tsv.xz``. + """Stream the ``ko`` file to a sorted, gzipped ``organism_gene_ko.tsv.gz``. Real KEGG has ~9M gene↔KO associations — far too many to hold in memory as a DataFrame. Rows are sorted by ``(organism, gene)`` before writing: gene IDs from one organism share long common prefixes (locus tags, numeric runs), so - sorting makes them adjacent and lets the compressor shrink the table ~2.9x - versus the unsorted gzip form. The order also matches the by-organism query - pattern in :func:`get_kegg_model_for_organism`. + sorting makes them adjacent and helps the compressor; the order also matches + the by-organism query pattern in :func:`get_kegg_model_for_organism`. Gzip + (not xz) keeps the table readable by MATLAB's built-in ``gunzip`` with no + external tool, at a modestly larger size. The sort is an **external merge sort** bounded to ``chunk_rows`` rows in memory at a time (sorted runs spooled to gzipped temp files, then merged with @@ -526,7 +552,7 @@ def stream_organism_gene_ko( handles = [gzip.open(r, "rt", encoding="utf-8") for r in runs] try: - with lzma.open(ogk_path, "wt", encoding="utf-8", newline="") as out: + with gzip.open(ogk_path, "wt", encoding="utf-8", newline="") as out: out.write("organism\tgene\tko\n") out.writelines(heapq.merge(*handles, key=_ogk_sort_key)) finally: @@ -535,7 +561,9 @@ def stream_organism_gene_ko( return pd.DataFrame(names, columns=["ko", "name"]) -def parse_kegg_dump(kegg_dir: str | Path, out_dir: str | Path) -> dict[str, Path]: +def parse_kegg_dump( + kegg_dir: str | Path, out_dir: str | Path, *, version: str | None = None +) -> dict[str, Path]: """Parse a full KEGG dump into the reference model + tables and write them out. Writes ``reference_model.yml.gz`` (gzipped RAVEN/cobra YAML) plus the @@ -544,9 +572,14 @@ def parse_kegg_dump(kegg_dir: str | Path, out_dir: str | Path) -> dict[str, Path ``organism_gene_ko`` table is streamed to disk (see :func:`stream_organism_gene_ko`) rather than built in memory, so this scales to the full KEGG database; the small derived tables are built in memory. + + When ``version`` is given (e.g. ``"kegg116"``) the output filenames are + version-prefixed (``_``), matching the published release + assets; the returned dict keys stay the logical table names. """ out_dir = Path(out_dir) out_dir.mkdir(parents=True, exist_ok=True) + prefix = f"{version}_" if version else "" reactions = parse_kegg_reactions(kegg_dir) compounds = parse_kegg_compounds(kegg_dir) @@ -563,16 +596,23 @@ def parse_kegg_dump(kegg_dir: str | Path, out_dir: str | Path) -> dict[str, Path columns=["reaction", "spontaneous", "undefined_stoich", "incomplete", "general"], ), } - paths = {name: p for name, p in zip(small, write_kegg_tables(small, out_dir), strict=True)} + paths = { + name: p + for name, p in zip(small, write_kegg_tables(small, out_dir, prefix=prefix), strict=True) + } - ogk_path = out_dir / "organism_gene_ko.tsv.xz" + ogk_path = out_dir / f"{prefix}organism_gene_ko.tsv.gz" ko_names = stream_organism_gene_ko(kegg_dir, linked_kos, ogk_path) paths["organism_gene_ko"] = ogk_path paths.update( - zip(["ko_names"], write_kegg_tables({"ko_names": ko_names}, out_dir), strict=True) + zip( + ["ko_names"], + write_kegg_tables({"ko_names": ko_names}, out_dir, prefix=prefix), + strict=True, + ) ) - ref_path = out_dir / "reference_model.yml.gz" + ref_path = out_dir / f"{prefix}reference_model.yml.gz" write_yaml_model(model, ref_path) paths["reference_model"] = ref_path return paths diff --git a/src/raven_python/reconstruction/kegg/query.py b/src/raven_toolbox/reconstruction/kegg/query.py similarity index 70% rename from src/raven_python/reconstruction/kegg/query.py rename to src/raven_toolbox/reconstruction/kegg/query.py index 2df3f781..f7163781 100644 --- a/src/raven_python/reconstruction/kegg/query.py +++ b/src/raven_toolbox/reconstruction/kegg/query.py @@ -5,14 +5,15 @@ score cut-off and the two score-ratio filters, then build the draft model with the shared assembler. For organisms not in KEGG. -Improvement over RAVEN: one ``hmmscan`` against the single ``hmmpress``-ed library -(K7) replaces RAVEN's per-KO ``hmmsearch`` loop. Phylogenetic-distance subsampling -is **not** used — our prebuilt prok90/euk90 libraries already fix the sequence set, -so picking the right domain library (not per-organism distance weighting) is the -relevant choice. - -The scoring/assignment logic (:func:`assign_kos`, :func:`parse_hmmscan_tblout`) is -pure and unit-tested; running the search needs HMMER (``hmmscan``). +Improvement over RAVEN: one ``hmmsearch`` of the whole concatenated KO library (K7) +replaces RAVEN's per-KO ``hmmsearch`` loop — the same fast search direction, but one +invocation instead of thousands, and no ``hmmpress``/``hmmscan`` needed. +Phylogenetic-distance subsampling is **not** used — our prebuilt prok90/euk90 +libraries already fix the sequence set, so picking the right domain library (not +per-organism distance weighting) is the relevant choice. + +The scoring/assignment logic (:func:`assign_kos`, :func:`parse_hmmsearch_tblout`) is +pure and unit-tested; running the search needs HMMER (``hmmsearch``). """ from __future__ import annotations @@ -24,39 +25,57 @@ import cobra import pandas as pd -from raven_python.binaries import resolve_binary -from raven_python.io.yaml import read_yaml_model -from raven_python.reconstruction.kegg.assemble import assemble_model_from_ko_genes -from raven_python.reconstruction.kegg.parse import read_kegg_table +from raven_toolbox.binaries import resolve_binary +from raven_toolbox.io.yaml import read_yaml_model +from raven_toolbox.reconstruction.kegg.assemble import assemble_model_from_ko_genes +from raven_toolbox.reconstruction.kegg.parse import _resolve_artefact, read_kegg_table _NOTE = "Included by get_kegg_model_from_sequences (using HMMs)" _MIN_EVALUE = 1e-250 # floor for a reported E-value of 0, to keep logs finite -def run_hmmscan( +def _count_profiles(library: str | Path) -> int: + """Number of profiles in an HMMER flatfile (one ``HMMER3/`` record header each).""" + n = 0 + with open(library, encoding="utf-8", errors="ignore") as handle: + for line in handle: + if line.startswith("HMMER3/"): + n += 1 + return n + + +def run_hmmsearch( fasta: str | Path, library: str | Path, *, threads: int = 1, - hmmscan: str | Path | None = None, + hmmsearch: str | Path | None = None, ) -> str: - """Run ``hmmscan`` of ``fasta`` against the pressed ``library``; return tblout text.""" - exe = resolve_binary("hmmscan", binary=hmmscan) + """Search the profile ``library`` against proteome ``fasta``; return tblout text. + + One ``hmmsearch`` of the whole concatenated multi-profile ``library`` (query) + against ``fasta`` (target) — the fast search direction, parallelised with + ``--cpu``, and no ``hmmpress`` needed. ``-Z`` is fixed to the profile count so the + per-hit E-values match the convention :func:`assign_kos` is calibrated against + (identical to a ``hmmscan`` against the same library). + """ + exe = resolve_binary("hmmsearch", binary=hmmsearch) + nprofiles = _count_profiles(library) + z_opt = ["-Z", str(nprofiles)] if nprofiles > 0 else [] with tempfile.TemporaryDirectory() as tmp: tbl = Path(tmp) / "hits.tbl" - cmd = [exe, "--cpu", str(threads), "--tblout", str(tbl), str(library), str(fasta)] + cmd = [exe, "--cpu", str(threads), *z_opt, "--tblout", str(tbl), str(library), str(fasta)] proc = subprocess.run(cmd, capture_output=True, text=True) if proc.returncode != 0: - raise RuntimeError(f"hmmscan failed:\n{(proc.stderr or '').strip()}") + raise RuntimeError(f"hmmsearch failed:\n{(proc.stderr or '').strip()}") return tbl.read_text() -def parse_hmmscan_tblout(text: str) -> pd.DataFrame: - """Parse ``hmmscan --tblout`` text into a ``[ko, gene, evalue]`` table. +def parse_hmmsearch_tblout(text: str) -> pd.DataFrame: + """Parse ``hmmsearch --tblout`` text into a ``[ko, gene, evalue]`` table. - In ``hmmscan`` the HMM database is the *target*, so column 1 (target name) is - the KO, column 3 (query name) is the proteome gene, and column 5 is the - full-sequence E-value. + With the profile library as the *query*, column 1 (target name) is the proteome + gene, column 3 (query name) is the KO, and column 5 is the full-sequence E-value. """ rows = [] for line in text.splitlines(): @@ -65,7 +84,7 @@ def parse_hmmscan_tblout(text: str) -> pd.DataFrame: fields = line.split() if len(fields) < 5: continue - rows.append((fields[0], fields[2], float(fields[4]))) + rows.append((fields[2], fields[0], float(fields[4]))) return pd.DataFrame(rows, columns=["ko", "gene", "evalue"]) @@ -170,15 +189,15 @@ def get_kegg_model_from_sequences( keep_incomplete: bool = True, keep_general: bool = False, threads: int = 1, - hmmscan: str | Path | None = None, + hmmsearch: str | Path | None = None, ) -> cobra.Model: """Reconstruct a draft model for a proteome by HMM-searching the KO library. - Searches ``fasta`` against the pressed ``library`` (3b.3), assigns KOs + Searches the ``library`` against ``fasta`` (3b.3), assigns KOs (:func:`assign_kos`), and assembles the model against ``reference_model`` / ``ko_reaction``. Genes are the query proteome's identifiers. """ - hits = parse_hmmscan_tblout(run_hmmscan(fasta, library, threads=threads, hmmscan=hmmscan)) + hits = parse_hmmsearch_tblout(run_hmmsearch(fasta, library, threads=threads, hmmsearch=hmmsearch)) ko_to_genes = assign_kos( hits, cutoff=cutoff, @@ -212,20 +231,20 @@ def get_kegg_model_from_sequences_with_artefacts( """Load reference model + tables from ``artefact_dir`` and run the HMM query. If ``artefact_dir`` / ``library`` are ``None`` they are fetched/cached via - :func:`raven_python.data.ensure_kegg_data` / :func:`raven_python.data.ensure_kegg_hmm_library` + :func:`raven_toolbox.data.ensure_kegg_data` / :func:`raven_toolbox.data.ensure_kegg_hmm_library` (``domain`` selects the prok/euk library; ``version`` the release). """ if artefact_dir is None or library is None: - from raven_python.data import ensure_kegg_data, ensure_kegg_hmm_library + from raven_toolbox.data import ensure_kegg_data, ensure_kegg_hmm_library if artefact_dir is None: artefact_dir = ensure_kegg_data(version=version) if library is None: library = ensure_kegg_hmm_library(domain, version=version) artefact_dir = Path(artefact_dir) - reference_model = read_yaml_model(artefact_dir / "reference_model.yml.gz") - ko_reaction = read_kegg_table(artefact_dir / "ko_reaction.tsv.gz") - rxn_flags = read_kegg_table(artefact_dir / "rxn_flags.tsv.gz") + reference_model = read_yaml_model(_resolve_artefact(artefact_dir, "reference_model.yml.gz")) + ko_reaction = read_kegg_table(_resolve_artefact(artefact_dir, "ko_reaction.tsv.gz")) + rxn_flags = read_kegg_table(_resolve_artefact(artefact_dir, "rxn_flags.tsv.gz")) return get_kegg_model_from_sequences( fasta, reference_model, ko_reaction, library, rxn_flags=rxn_flags, **kwargs ) diff --git a/src/raven_toolbox/reconstruction/kegg/taxonomy.py b/src/raven_toolbox/reconstruction/kegg/taxonomy.py new file mode 100644 index 00000000..4b8dbb20 --- /dev/null +++ b/src/raven_toolbox/reconstruction/kegg/taxonomy.py @@ -0,0 +1,176 @@ +"""Parse the KEGG ``taxonomy`` file into lineages, domains, and phylogenetic distances. + +Ports RAVEN ``getPhylDist``: the file-reading half (per-organism category lineages and +the domain split) **and** the distance-matrix half (:func:`phyl_dist`), which earlier +ports deferred. The ``taxonomy`` file is an indented tree: ``#``-prefixed lines name a +category, the number of leading ``#`` giving its depth; organism lines are tab-separated +``T-numberorg_idT-numbername``. Each organism inherits the stack of +categories above it, the first of which is its domain (``Prokaryotes`` / ``Eukaryotes``). + +Used by 3b.3 to split genes into the prok/euk HMM libraries, by 3b.4 domain mode, and by +:func:`phyl_dist` to regenerate the KEGG phylogenetic distance matrix (RAVEN's +``keggPhylDist``) that GECKO uses to pick the closest organism for kcat assignment — so +that capability needs no MATLAB ``.mat`` file, only the published ``taxonomy`` artefact. +""" +from __future__ import annotations + +import gzip +import warnings +from dataclasses import dataclass, field +from pathlib import Path + +import numpy as np + + +def _open_text(path: str | Path): + """Open ``path`` as UTF-8 text, transparently decompressing a ``.gz`` artefact.""" + path = Path(path) + if path.suffix == ".gz": + return gzip.open(path, "rt", encoding="utf-8") + return open(path, encoding="utf-8") + + +def parse_taxonomy_records(path: str | Path) -> list[tuple[str, str, list[str]]]: + """Return ``[(organism_id, name, [category, ...]), ...]`` in file order. + + ``name`` is the organism's scientific name with any trailing KEGG parenthetical + kept (e.g. ``"Homo sapiens (human)"``), matching RAVEN ``getPhylDist``; the category + list is the lineage from outermost (domain) to innermost. Order is preserved so it + aligns with the rows/columns of :func:`phyl_dist`. + """ + records: list[tuple[str, str, list[str]]] = [] + stack: list[str] = [] + skipped_level_warned = False + with _open_text(path) as handle: + for line_no, raw in enumerate(handle, start=1): + line = raw.rstrip("\n") + if not line.strip(): + continue + if line.startswith("#"): + depth = len(line) - len(line.lstrip("#")) + name = line[depth:].strip() + if depth - 1 > len(stack): + # Depth-skip (e.g. ## then ####): pad the missing levels with blanks + # so downstream slices stay aligned; warn once. + if not skipped_level_warned: + warnings.warn( + f"{path}: taxonomy depth skips a level near line {line_no} " + f"({'#' * depth} {name!r} appeared with stack {stack!r}); " + "padding the missing levels with '' (later occurrences silenced).", + stacklevel=2, + ) + skipped_level_warned = True + stack = stack + [""] * (depth - 1 - len(stack)) + else: + stack = stack[: depth - 1] + stack.append(name) + else: + # Organism line. RAVEN reads the id between the 1st/2nd whitespace and the + # name after the 3rd; with the tab-delimited file that is fields[1]/fields[3]. + if "\t" in line: + fields = line.split("\t") + org_id = fields[1].strip() if len(fields) > 1 else "" + org_name = fields[3].strip() if len(fields) > 3 else "" + else: + fields = line.split() + org_id = fields[1] if len(fields) > 1 else "" + org_name = " ".join(fields[3:]) if len(fields) > 3 else "" + if not org_id: + continue + records.append((org_id, org_name, list(stack))) + return records + + +def parse_taxonomy(path: str | Path) -> dict[str, list[str]]: + """Return ``{organism_id: [category, ...]}`` from outermost to innermost.""" + return {org_id: lineage for org_id, _name, lineage in parse_taxonomy_records(path)} + + +def organism_domains(path: str | Path) -> dict[str, str]: + """Return ``{organism_id: domain}`` (the outermost category).""" + return {org: cats[0] for org, cats in parse_taxonomy(path).items() if cats} + + +def organisms_in_domain(path: str | Path, domain: str) -> set[str]: + """Organism ids whose outermost category matches ``domain`` (case-insensitive). + + Accepts a prefix, so ``"prok"`` matches ``"Prokaryotes"``. + """ + needle = domain.lower() + return { + org + for org, dom in organism_domains(path).items() + if dom.lower().startswith(needle) or needle.startswith(dom.lower()) + } + + +@dataclass +class PhylDist: + """KEGG phylogenetic distance structure — RAVEN ``getPhylDist`` / ``keggPhylDist``. + + ``ids``/``names`` are the KEGG organism ids and scientific names (names keep RAVEN's + trailing parenthetical; consumers that need them cleaned, e.g. geckopy, strip it). + ``dist_matrix[i, j]`` is RAVEN's distance from ``ids[i]`` to ``ids[j]``. It is a + faithful reproduction of RAVEN's metric, which is **asymmetric and may be negative**; + GECKO consumes it only by taking the closest (minimum-distance) organism, so the exact + values matter for parity with MATLAB but the sign/asymmetry are harmless there. + """ + + ids: list[str] = field(default_factory=list) + names: list[str] = field(default_factory=list) + dist_matrix: np.ndarray = field(default_factory=lambda: np.empty((0, 0), dtype=np.int64)) + + +def phyl_dist(path: str | Path, *, only_in_kingdom: bool = False) -> PhylDist: + """Compute the KEGG phylogenetic distance matrix from the ``taxonomy`` file. + + Faithful port of RAVEN ``getPhylDist``. For organisms ``i`` and ``j`` with category + lineages of lengths ``Li, Lj``, the distance is ``(Li - Lj) + min(Li, Lj) - k`` where + ``k`` is the deepest level (within the shorter lineage, comparing position by position + from the root) at which the two categories coincide — ``1`` if none coincide. This + reproduces RAVEN's exact values (including their asymmetry and occasional negatives) + so the output matches MATLAB ``keggPhylDist`` that GECKO was built against. + + ``only_in_kingdom`` sets the distance to ``+inf`` between organisms in different + domains (RAVEN's ``onlyInKingdom`` variant); the matrix is then returned as ``float``. + + Cost is ``O(N²)`` in time and memory (``N`` = number of KEGG organisms, ~10⁴), so this + is a maintainer/GECKO-side generation step; persist the result rather than rebuilding. + """ + records = parse_taxonomy_records(path) + ids = [r[0] for r in records] + names = [r[1] for r in records] + lineages = [r[2] for r in records] + n = len(records) + if n == 0: + return PhylDist(ids, names, np.empty((0, 0), dtype=np.int64)) + + lengths = np.array([len(lin) for lin in lineages], dtype=np.int32) + max_depth = int(lengths.max()) + + # deepest[i, j] = deepest level d (1-based) at which lineage_i[d-1] == lineage_j[d-1], + # counting only levels present in both (d <= min(Li, Lj)); 0 if none coincide. + deepest = np.zeros((n, n), dtype=np.int32) + for d in range(1, max_depth + 1): + lookup: dict[str, int] = {} + lab = np.full(n, -1, dtype=np.int64) + for i, lin in enumerate(lineages): + if len(lin) >= d: + lab[i] = lookup.setdefault(lin[d - 1], len(lookup)) + eq = (lab[:, None] == lab[None, :]) & (lab[:, None] != -1) + # Ascending d, so the last (deepest) coinciding level wins -> max coinciding depth. + deepest[eq] = d + k = np.where(deepest == 0, 1, deepest).astype(np.int32) + + # Values are tiny (|dist| <= max lineage depth); int32 throughout keeps the N x N + # matrix (and its intermediates) half the size of the int64 default. + li = lengths[:, None] + lj = lengths[None, :] + dist = (li - lj) + np.minimum(li, lj) - k + + if only_in_kingdom: + domains = np.array([lin[0] if lin else "" for lin in lineages]) + dist = dist.astype(float) + dist[domains[:, None] != domains[None, :]] = np.inf + + return PhylDist(ids=ids, names=names, dist_matrix=dist) diff --git a/src/raven_python/tasks/__init__.py b/src/raven_toolbox/tasks/__init__.py similarity index 86% rename from src/raven_python/tasks/__init__.py rename to src/raven_toolbox/tasks/__init__.py index d232c164..f96c18bf 100644 --- a/src/raven_python/tasks/__init__.py +++ b/src/raven_toolbox/tasks/__init__.py @@ -5,13 +5,13 @@ * :func:`find_task_essential_reactions` + :class:`EssentialReactionsResult` — reactions a model must use to satisfy a task list (the input for (f)tINIT's task layer). """ -from raven_python.tasks.check import ( +from raven_toolbox.tasks.check import ( EssentialReactionsResult, TaskResult, check_tasks, find_task_essential_reactions, ) -from raven_python.tasks.tasklist import Task, parse_task_list +from raven_toolbox.tasks.tasklist import Task, parse_task_list __all__ = [ "EssentialReactionsResult", diff --git a/src/raven_python/tasks/check.py b/src/raven_toolbox/tasks/check.py similarity index 96% rename from src/raven_python/tasks/check.py rename to src/raven_toolbox/tasks/check.py index 817bae5f..58da5c47 100644 --- a/src/raven_python/tasks/check.py +++ b/src/raven_toolbox/tasks/check.py @@ -17,14 +17,15 @@ from collections.abc import Iterable from dataclasses import dataclass from pathlib import Path +from typing import cast import cobra from cobra.exceptions import OptimizationError from cobra.flux_analysis import flux_variability_analysis, pfba from optlang.symbolics import Zero -from raven_python.manipulation.add import add_reactions_from_equations -from raven_python.tasks.tasklist import Task, parse_task_list +from raven_toolbox.manipulation.add import add_reactions_from_equations +from raven_toolbox.tasks.tasklist import Task, parse_task_list _ALLMETS = "ALLMETS" _ALLMETSIN = "ALLMETSIN" @@ -51,11 +52,14 @@ def _set_constraint_bounds(constraint, lb: float, ub: float) -> None: constraint.ub = ub -def _classify(token: str) -> tuple[str, str | None]: - """Return ``("all", None)``, ``("comp", COMP)``, or ``("met", token_upper)``.""" +def _classify(token: str) -> tuple[str, str]: + """Return ``("all", "")``, ``("comp", COMP)``, or ``("met", token_upper)``. + + The arg is empty only for ``"all"``, which never reads it. + """ upper = token.upper() if upper == _ALLMETS: - return "all", None + return "all", "" if upper.startswith(_ALLMETSIN + "[") and upper.endswith("]"): return "comp", upper[len(_ALLMETSIN) + 1: -1] return "met", upper # incl. malformed ALLMETSIN[... → treated as a (missing) metabolite @@ -210,7 +214,7 @@ def check_tasks( def _as_tasks(tasks: str | Iterable[Task]) -> list[Task]: if isinstance(tasks, (str, bytes)) or hasattr(tasks, "__fspath__"): - return parse_task_list(tasks) + return parse_task_list(cast("str | Path", tasks)) # guard above ⇒ path-like return list(tasks) diff --git a/src/raven_python/tasks/tasklist.py b/src/raven_toolbox/tasks/tasklist.py similarity index 99% rename from src/raven_python/tasks/tasklist.py rename to src/raven_toolbox/tasks/tasklist.py index 5bdbcb0f..d2588d94 100644 --- a/src/raven_python/tasks/tasklist.py +++ b/src/raven_toolbox/tasks/tasklist.py @@ -2,7 +2,7 @@ A task list defines, per task, allowed inputs/outputs, optional extra reactions (equations), reaction-bound changes, and whether the task *should fail*. Tasks -are checked with :func:`raven_python.tasks.check_tasks`. +are checked with :func:`raven_toolbox.tasks.check_tasks`. The file is tab-delimited (``.txt``/``.tsv``) or Excel (``.xlsx``, sheet ``TASKS``; needs the ``[excel]`` extra). Recognised columns (the only required one is ``ID``): diff --git a/src/raven_python/utils/__init__.py b/src/raven_toolbox/utils/__init__.py similarity index 50% rename from src/raven_python/utils/__init__.py rename to src/raven_toolbox/utils/__init__.py index 7127bdd1..affca62d 100644 --- a/src/raven_python/utils/__init__.py +++ b/src/raven_toolbox/utils/__init__.py @@ -1,8 +1,8 @@ """Shared helpers — GPR linting, elemental balance, model curation checks, id sorting.""" -from raven_python.utils.balance import ElementalBalance, get_elemental_balance -from raven_python.utils.gpr import GPRIssue, find_non_dnf_grrules, is_dnf -from raven_python.utils.sort import sort_identifiers -from raven_python.utils.validate import ModelIssue, check_model +from raven_toolbox.utils.balance import ElementalBalance, get_elemental_balance +from raven_toolbox.utils.gpr import GPRIssue, find_non_dnf_grrules, is_dnf +from raven_toolbox.utils.sort import sort_identifiers +from raven_toolbox.utils.validate import ModelIssue, check_model __all__ = [ "ElementalBalance", diff --git a/src/raven_python/utils/balance.py b/src/raven_toolbox/utils/balance.py similarity index 100% rename from src/raven_python/utils/balance.py rename to src/raven_toolbox/utils/balance.py diff --git a/src/raven_python/utils/gpr.py b/src/raven_toolbox/utils/gpr.py similarity index 73% rename from src/raven_python/utils/gpr.py rename to src/raven_toolbox/utils/gpr.py index 2e2122d3..69d73e08 100644 --- a/src/raven_python/utils/gpr.py +++ b/src/raven_toolbox/utils/gpr.py @@ -10,18 +10,48 @@ ``(G1 and G2) or G3``. Rules where an AND contains an OR — e.g. ``(G1 or G2) and (G3 or G4)`` — are *valid* for cobra but ambiguous for the isoenzyme/complex reasoning used across RAVEN/GECKO, and ``expand_model`` -(see :mod:`raven_python.manipulation.expand`) only does something for DNF rules. +(see :mod:`raven_toolbox.manipulation.expand`) only does something for DNF rules. :func:`find_non_dnf_grrules` surfaces them as structured data rather than, as RAVEN did, only printing a warning. """ from __future__ import annotations import ast +import statistics +from collections.abc import Callable from dataclasses import dataclass import cobra from cobra.core.gene import GPR +#: Score-aggregation functions for combining gene scores across a GPR: isozymes +#: (genes joined by OR) and complex subunits (joined by AND). Shared by the +#: gene→reaction scoring (:mod:`raven_toolbox.init.score`) and low-score-gene +#: pruning (:mod:`raven_toolbox.init.genes`) so both validate the same way. +AGGREGATORS: dict[str, Callable] = { + "min": min, + "max": max, + "median": statistics.median, + "average": statistics.fmean, +} + + +def resolve_aggregators( + isozyme_scoring: str, complex_scoring: str +) -> tuple[Callable, Callable]: + """Validate the scoring-mode names and return ``(isozyme_fn, complex_fn)``. + + Raises ``ValueError`` naming the offending argument if either mode is not one + of :data:`AGGREGATORS`. + """ + for name, value in ( + ("isozyme_scoring", isozyme_scoring), + ("complex_scoring", complex_scoring), + ): + if value not in AGGREGATORS: + raise ValueError(f"{name} must be one of {sorted(AGGREGATORS)}; got {value!r}.") + return AGGREGATORS[isozyme_scoring], AGGREGATORS[complex_scoring] + def _contains_or(node: ast.AST | None) -> bool: """True if ``node``'s subtree contains an OR operator anywhere.""" @@ -65,11 +95,10 @@ def is_dnf(gpr: GPR | str | None) -> bool: >>> is_dnf("(G1 or G2) and G3") False """ - if isinstance(gpr, str): - gpr = GPR.from_string(gpr) - if gpr is None: + parsed = GPR.from_string(gpr) if isinstance(gpr, str) else gpr + if parsed is None: return True - return _is_dnf_node(gpr.body) + return _is_dnf_node(parsed.body) @dataclass(frozen=True) diff --git a/src/raven_python/utils/parse.py b/src/raven_toolbox/utils/parse.py similarity index 52% rename from src/raven_python/utils/parse.py rename to src/raven_toolbox/utils/parse.py index 8068f6c2..c0ae2c7f 100644 --- a/src/raven_python/utils/parse.py +++ b/src/raven_toolbox/utils/parse.py @@ -1,4 +1,4 @@ -"""Small parsing helpers shared across raven_python.""" +"""Small parsing helpers shared across raven_toolbox.""" from __future__ import annotations import re @@ -31,3 +31,29 @@ def parse_name_comp(token: str) -> tuple[str, str | None]: if match: return match.group("name").strip(), match.group("comp").strip() return token.strip(), None + + +def subsystem_to_str(value) -> str: + """Coerce a reaction ``subsystem`` to cobra's canonical ``str`` form. + + cobra defines ``Reaction.subsystem`` as a plain string, but RAVEN/MATLAB and + "messy" YAML sometimes deliver a list/tuple of subsystem names (a reaction can + belong to several). This collapses any such value to a single ``;``-joined + string of its parts — losing no data, unlike taking only the first — and returns + ``""`` for an empty/None subsystem. Used wherever a subsystem is rendered or + compared so the handling is identical across modules. + + Examples + -------- + >>> subsystem_to_str("glycolysis") + 'glycolysis' + >>> subsystem_to_str(["glycolysis", "TCA cycle"]) + 'glycolysis;TCA cycle' + >>> subsystem_to_str(None) + '' + """ + if value is None: + return "" + if isinstance(value, (list, tuple)): + return ";".join(str(s) for s in value if str(s).strip()) + return str(value) diff --git a/src/raven_python/utils/sort.py b/src/raven_toolbox/utils/sort.py similarity index 100% rename from src/raven_python/utils/sort.py rename to src/raven_toolbox/utils/sort.py diff --git a/src/raven_python/utils/validate.py b/src/raven_toolbox/utils/validate.py similarity index 100% rename from src/raven_python/utils/validate.py rename to src/raven_toolbox/utils/validate.py diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 00000000..d4837351 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,210 @@ +"""Shared pytest fixtures for the raven_toolbox test-suite. + +Currently the linear-chain INIT model used by the tINIT/ftINIT scoring tests, +which several modules previously built independently and identically. +""" +from pathlib import Path + +import cobra +import pytest + + +def _linear_chain_model(*, with_genes: bool = False) -> cobra.Model: + """EX_A -> A -(r1)-> B -(r2)-> C -(r3)-> D. + + A is taken up via the reversible ``EX_A``; ``r1``/``r2`` are the productive path + and ``r3`` the dead-end branch the INIT/scoring tests penalise. With ``with_genes`` + the three internal reactions get gene rules ``g1``/``g2``/``g3``. + """ + m = cobra.Model("net") + A, B, C, D = ( + cobra.Metabolite(x, name=x[:-2], compartment="c") + for x in ("A_c", "B_c", "C_c", "D_c") + ) + m.add_metabolites([A, B, C, D]) + exa = cobra.Reaction("EX_A", lower_bound=-1000, upper_bound=1000) + exa.add_metabolites({A: -1}) # negative flux = uptake of A + r1 = cobra.Reaction("r1", lower_bound=0, upper_bound=1000) + r1.add_metabolites({A: -1, B: 1}) + r2 = cobra.Reaction("r2", lower_bound=0, upper_bound=1000) + r2.add_metabolites({B: -1, C: 1}) + r3 = cobra.Reaction("r3", lower_bound=0, upper_bound=1000) + r3.add_metabolites({C: -1, D: 1}) + m.add_reactions([exa, r1, r2, r3]) + if with_genes: + for rid, rule in (("r1", "g1"), ("r2", "g2"), ("r3", "g3")): + m.reactions.get_by_id(rid).gene_reaction_rule = rule + return m + + +@pytest.fixture +def linear_chain_model() -> cobra.Model: + """A fresh linear-chain INIT model (no gene rules).""" + return _linear_chain_model() + + +@pytest.fixture +def linear_chain_model_with_genes() -> cobra.Model: + """A fresh linear-chain INIT model with gene rules g1/g2/g3 on r1/r2/r3.""" + return _linear_chain_model(with_genes=True) + + +# --------------------------------------------------------------------------- # +# Synthetic KEGG dump +# --------------------------------------------------------------------------- # +# A small, entirely fictional KEGG-format dump used by the reconstruction/kegg +# tests. It mimics the flat-file *format* (so it exercises the parser's flag +# detection, overview-map skipping, InChI/formula handling, mapformula +# irreversibility, KO/gene grouping and taxonomy lineages) but contains no real +# KEGG content — identifiers, names, sequences and cross-references are all +# made up, since the project is not licensed to redistribute KEGG data. The org +# codes (aaa/bbb/ccc/zzz), gene ids (GENE0x), KO/reaction/compound numbers +# (K9xxxx/R9xxxx/C9xxxx) and protein sequences are invented; only the structural +# domain/phylum words and the comment keywords the parser keys off are generic. + +_REACTION = """\ +ENTRY R90010 Reaction +NAME fictional glucohydrolase analogue +DEFINITION Trehalike + Aqualike <=> 2 Glucolike +EQUATION C91083 + C90001 <=> 2 C90031 +ENZYME 9.9.9.99 +PATHWAY rn09500 Fictional sugar metabolism + rn01199 Fictional global overview +MODULE M90599 example module +ORTHOLOGY K90001 fictional trehalase analogue [EC:9.9.9.99] +DBLINKS RHEA: 99999 +/// +ENTRY R90100 Reaction +NAME spontaneous example +COMMENT This reaction is spontaneous. +EQUATION C90002 <=> C90003 +ORTHOLOGY K90002 fictional dehydrogenase analogue +/// +ENTRY R90200 Reaction +NAME undefined stoich example +EQUATION C90001 + n C90002 <=> C90003 +/// +ENTRY R90300 Reaction +NAME general example +COMMENT General reaction. +EQUATION C90031 <=> C90006 +ORTHOLOGY K90009 lumped ortholog +/// +ENTRY R90400 Reaction +NAME empty after cancellation +EQUATION C90007 <=> C90007 +/// +""" + +_COMPOUND = """\ +ENTRY C90001 Compound +NAME Aqualike; + Aquatwo +FORMULA A2O +DBLINKS PubChem: 9303 + ChEBI: 95377 +/// +ENTRY C90002 Compound +NAME Atriplike +FORMULA A10 +/// +ENTRY C90003 Compound +NAME Endione; + Endi +FORMULA N21 +/// +ENTRY C90006 Compound +NAME Endionep +FORMULA N22 +/// +ENTRY C90031 Compound +NAME Glucolike; + Sweetgrain +FORMULA G6 +DBLINKS ChEBI: 94167 97634 +/// +ENTRY C91083 Compound +NAME Trehalike +FORMULA T12 +/// +ENTRY C90007 Compound +NAME Oxylike +FORMULA O2 +/// +""" + +# InChI for C90031 is a made-up string; only its "InChI=" prefix is asserted, and +# its presence makes the parser prefer it over the FORMULA (clearing the latter). +_COMPOUND_INCHI = "C90031\tInChI=1S/FAKE6/c1-2-3\n" + +_KO = """\ +ENTRY K90001 KO +NAME treF, FCT +DEFINITION fictional trehalase analogue [EC:9.9.9.99] +GENES AAA: GENE01(alias1) GENE02 + CCC: GENE04 GENE05(alias5) +/// +ENTRY K90002 KO +DEFINITION fictional dehydrogenase analogue +GENES BBB: GENE03 +/// +ENTRY K90099 KO +DEFINITION unlinked ortholog +GENES BBB: GENE98 +/// +""" + +# Invented (non-biological) protein sequences; only bbb:GENE03's is asserted. +_GENES_PEP = """\ +>aaa:GENE01 alias1; fictional protein one +MKKLLAVTACGGVTAGGVTAGGVTAGGVTAGGVTAGGVTAGGVTAGG +>aaa:GENE02 hypothetical protein +MKKLLAVTACGGVTAGGVTAGGVTAGGVTAGGVTAGGVTAGGVTAGX +>bbb:GENE03 fictional aspartokinase analogue +MQFKTLVIDEGHKLPSTWYNACRMQFKTLVIDEGHKLPSTWYNACR +>ccc:GENE04 fictional dehydrogenase a +MSSTTGGKKVVIICCKAAALLWWEELLKKPPFFSSIIEEVVEEVV +>ccc:GENE05 fictional dehydrogenase b +MSSTTGGKKVVIICCKAAALLWWEEVVKKPPFFSSIIEEDDVVEEVV +>zzz:GENE99 some other gene not in any KO +MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +""" + +_TAXONOMY = ( + "# Prokaryotes\n" + "## Bacteria\n" + "### Firmicutes\n" + "T90010\taaa\tFictobacter examplensis\tFictobacter\n" + "### Gammaproteobacteria\n" + "T90007\tbbb\tMockella testium\tMockella\n" + "# Eukaryotes\n" + "## Animals\n" + "### Vertebrates\n" + "T90001\tccc\tImaginaria animalis\tImaginaria\n" +) + +_MAPFORMULA = ( + "R90010: 09500: C91083 => C90031\n" # one direction in this map ... + "R90010: 09010: C90031 => C91083\n" # ... the opposite in another -> stays reversible + "R90100: 09010: C90002 => C90003\n" # only ever one direction -> irreversible +) + + +@pytest.fixture(scope="session") +def kegg_dump(tmp_path_factory) -> Path: + """A synthetic, fully fictional KEGG-format dump (no real KEGG content). + + Exercises the KEGG dump parser end-to-end — reaction flags, overview-map + skipping, InChI/formula handling, mapformula irreversibility, KO/gene + grouping and taxonomy lineages — without redistributing any KEGG-derived + data, which the project is not licensed to ship. + """ + d = tmp_path_factory.mktemp("kegg_dump") + (d / "reaction").write_text(_REACTION, encoding="utf-8") + (d / "compound").write_text(_COMPOUND, encoding="utf-8") + (d / "compound.inchi").write_text(_COMPOUND_INCHI, encoding="utf-8") + (d / "ko").write_text(_KO, encoding="utf-8") + (d / "genes.pep").write_text(_GENES_PEP, encoding="utf-8") + (d / "reaction_mapformula.lst").write_text(_MAPFORMULA, encoding="utf-8") + (d / "taxonomy").write_text(_TAXONOMY, encoding="utf-8") + return d diff --git a/tests/data/kegg_dump/compound b/tests/data/kegg_dump/compound deleted file mode 100644 index a78d1767..00000000 --- a/tests/data/kegg_dump/compound +++ /dev/null @@ -1,34 +0,0 @@ -ENTRY C00001 Compound -NAME H2O; - Water -FORMULA H2O -DBLINKS PubChem: 3303 - ChEBI: 15377 -/// -ENTRY C00002 Compound -NAME ATP -FORMULA C10H16N5O13P3 -/// -ENTRY C00003 Compound -NAME NAD+; - NAD -FORMULA C21H28N7O14P2 -/// -ENTRY C00006 Compound -NAME NADP+ -FORMULA C21H29N7O17P3 -/// -ENTRY C00031 Compound -NAME D-Glucose; - Grape sugar -FORMULA C6H12O6 -DBLINKS ChEBI: 4167 17634 -/// -ENTRY C01083 Compound -NAME alpha,alpha-Trehalose -FORMULA C12H22O11 -/// -ENTRY C00007 Compound -NAME Oxygen -FORMULA O2 -/// diff --git a/tests/data/kegg_dump/compound.inchi b/tests/data/kegg_dump/compound.inchi deleted file mode 100644 index 448312f8..00000000 --- a/tests/data/kegg_dump/compound.inchi +++ /dev/null @@ -1 +0,0 @@ -C00031 InChI=1S/C6H12O6/c7-1-2-3(8)4(9)5(10)6(11)12-2/h2-11H,1H2 diff --git a/tests/data/kegg_dump/genes.pep b/tests/data/kegg_dump/genes.pep deleted file mode 100644 index f30073df..00000000 --- a/tests/data/kegg_dump/genes.pep +++ /dev/null @@ -1,12 +0,0 @@ ->bsu:BSU31050 gbsB; choline dehydrogenase -MKVLAAGGTGYIGSHTVVELLEAGYDVVVLDNLSNGHREAVPKGVPFveqIDLRDREALDR ->bsu:BSU31060 hypothetical protein -MKVLAAGGTGYIGSHTVVELLEAGYDVVVLDNLSNGHREAVPKGVPFveqIDLRDREALDX ->eco:b0001 thrA; aspartokinase -MRVLKFGGTSVANAERFLRVADILESNARQGQVATVLSAPAKITNHLVAMIEKTISGQDA ->hsa:124 ADH1A; alcohol dehydrogenase 1A -MSTAGKVIKCKAAVLWELKKPFSIEEVEVAPPKAHEVRIKMVATGICRSDDHVVSGTLVT ->hsa:125 ADH1B; alcohol dehydrogenase 1B -MSTAGKVIKCKAAVLWEVKKPFSIEDVEVAPPKAHEVRIKMVATGICRSDDHVVSGTLVT ->xxx:unused some other gene not in any KO -MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA diff --git a/tests/data/kegg_dump/ko b/tests/data/kegg_dump/ko deleted file mode 100644 index f6ae027e..00000000 --- a/tests/data/kegg_dump/ko +++ /dev/null @@ -1,14 +0,0 @@ -ENTRY K01194 KO -NAME treA, TREH -DEFINITION alpha,alpha-trehalase [EC:3.2.1.28] -GENES BSU: BSU31050(gbsB) BSU31060 - HSA: 124 125(ADH) -/// -ENTRY K00002 KO -DEFINITION AKR1A1; alcohol dehydrogenase (NADP+) [EC:1.1.1.2] -GENES ECO: b0001 -/// -ENTRY K99999 KO -DEFINITION unlinked ortholog -GENES ECO: b9999 -/// diff --git a/tests/data/kegg_dump/reaction b/tests/data/kegg_dump/reaction deleted file mode 100644 index b0e75c56..00000000 --- a/tests/data/kegg_dump/reaction +++ /dev/null @@ -1,31 +0,0 @@ -ENTRY R00010 Reaction -NAME alpha,alpha-trehalose glucohydrolase -DEFINITION alpha,alpha-Trehalose + H2O <=> 2 D-Glucose -EQUATION C01083 + C00001 <=> 2 C00031 -ENZYME 3.2.1.28 -PATHWAY rn00500 Starch and sucrose metabolism - rn01100 Metabolic pathways -MODULE M00599 example module -ORTHOLOGY K01194 alpha,alpha-trehalase [EC:3.2.1.28] -DBLINKS RHEA: 32678 -/// -ENTRY R00100 Reaction -NAME spontaneous example -COMMENT This reaction is spontaneous. -EQUATION C00002 <=> C00003 -ORTHOLOGY K00002 some enzyme -/// -ENTRY R00200 Reaction -NAME undefined stoich example -EQUATION C00001 + n C00002 <=> C00003 -/// -ENTRY R00300 Reaction -NAME general example -COMMENT General reaction. -EQUATION C00031 <=> C00006 -ORTHOLOGY K09999 lumped ortholog -/// -ENTRY R00400 Reaction -NAME empty after cancellation -EQUATION C00007 <=> C00007 -/// diff --git a/tests/data/kegg_dump/reaction_mapformula.lst b/tests/data/kegg_dump/reaction_mapformula.lst deleted file mode 100644 index 0adb8f0f..00000000 --- a/tests/data/kegg_dump/reaction_mapformula.lst +++ /dev/null @@ -1,3 +0,0 @@ -R00010: 00500: C01083 => C00031 -R00010: 00010: C00031 => C01083 -R00100: 00010: C00002 => C00003 diff --git a/tests/data/kegg_dump/taxonomy b/tests/data/kegg_dump/taxonomy deleted file mode 100644 index f0447e62..00000000 --- a/tests/data/kegg_dump/taxonomy +++ /dev/null @@ -1,10 +0,0 @@ -# Prokaryotes -## Bacteria -### Firmicutes -T00010 bsu Bacillus subtilis 168 Bacillus -### Gammaproteobacteria - Enterobacteria -T00007 eco Escherichia coli K-12 MG1655 Escherichia -# Eukaryotes -## Animals -### Vertebrates - Mammals -T01001 hsa Homo sapiens (human) Homo diff --git a/tests/test_analysis_fseof.py b/tests/test_analysis_fseof.py index 5f23f3f1..e75d814c 100644 --- a/tests/test_analysis_fseof.py +++ b/tests/test_analysis_fseof.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.analysis import FSEOFResult, fseof +from raven_toolbox.analysis import FSEOFResult, fseof @pytest.fixture @@ -97,7 +97,7 @@ def test_amplify_label_uses_abs_slope_not_endpoint_difference(): import numpy as np import pandas as pd - from raven_python.analysis.fseof import _classify + from raven_toolbox.analysis.fseof import _classify # Endpoints equal (0), but the |flux| regression slope is clearly positive # over the scan — the new classifier picks amplify; the old endpoint code diff --git a/tests/test_analysis_reporter.py b/tests/test_analysis_reporter.py index 918f15df..775f8f17 100644 --- a/tests/test_analysis_reporter.py +++ b/tests/test_analysis_reporter.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.analysis import ReporterResult, reporter_metabolites +from raven_toolbox.analysis import ReporterResult, reporter_metabolites def _met(mid): diff --git a/tests/test_analysis_sampling.py b/tests/test_analysis_sampling.py index 155500b4..21e7488a 100644 --- a/tests/test_analysis_sampling.py +++ b/tests/test_analysis_sampling.py @@ -3,7 +3,7 @@ import numpy as np import pytest -from raven_python.analysis import ( +from raven_toolbox.analysis import ( RandomSamplingResult, find_good_reactions, random_sampling, diff --git a/tests/test_annotation.py b/tests/test_annotation.py new file mode 100644 index 00000000..d61a6511 --- /dev/null +++ b/tests/test_annotation.py @@ -0,0 +1,204 @@ +"""Tests for raven_toolbox.annotation (SBO terms + ΔG CSV persistence).""" +from __future__ import annotations + +import math + +import cobra +import pandas as pd + +from raven_toolbox.annotation import ( + DEFAULT_BIOMASS_MET_NAMES, + add_sbo_terms, + load_delta_g_csv, + save_delta_g_csv, +) +from raven_toolbox.annotation.sbo import _default_transport_detector + +# --- shared tiny model ------------------------------------------------- + +def _toy_model() -> cobra.Model: + """Toy yeast-flavoured model with: extracellular exchange, transport, + metabolic, biomass, NGAM, generic pseudo, lipid-backbone biomass met.""" + m = cobra.Model("toy") + m.compartments = {"c": "cytoplasm", "e": "extracellular", "x": "peroxisome"} + + mets = { + "atp_c": cobra.Metabolite("atp_c", name="ATP", compartment="c", charge=-4, formula="C10H12N5O13P3"), + "atp_x": cobra.Metabolite("atp_x", name="ATP", compartment="x", charge=-4, formula="C10H12N5O13P3"), + "glc_e": cobra.Metabolite("glc_e", name="glucose", compartment="e", charge=0, formula="C6H12O6"), + "biomass": cobra.Metabolite("biomass_c", name="biomass", compartment="c"), + "lipid_bb": cobra.Metabolite("lbb_c", name="lipid backbone", compartment="c"), + } + m.add_metabolites(list(mets.values())) + + # Exchange (single met in extracellular) + ex = cobra.Reaction("EX_glc", lower_bound=-10, upper_bound=1000) + ex.add_metabolites({mets["glc_e"]: -1}) + # Transport (ATP_c ↔ ATP_x — same met name, two compartments) + tr = cobra.Reaction("ATP_tx", lower_bound=-1000, upper_bound=1000) + tr.add_metabolites({mets["atp_c"]: -1, mets["atp_x"]: 1}) + # Normal metabolic reaction + met = cobra.Reaction("metab1", lower_bound=0, upper_bound=1000) + met.add_metabolites({mets["glc_e"]: -1, mets["atp_c"]: 1}) + # Biomass pseudoreaction (must be last for the bug-compat test) + bio = cobra.Reaction("biomass_rxn", lower_bound=0, upper_bound=1000) + bio.name = "biomass pseudoreaction" + bio.add_metabolites({mets["atp_c"]: -1, mets["biomass"]: 1}) + m.add_reactions([ex, tr, met, bio]) + return m + + +# --- add_sbo_terms ----------------------------------------------------- + +def test_default_biomass_names_set_is_reasonable(): + assert {"biomass", "DNA", "RNA", "protein", "lipid"} <= DEFAULT_BIOMASS_MET_NAMES + + +def test_simple_chemical_gets_simple_sbo(): + m = _toy_model() + add_sbo_terms(m) + assert m.metabolites.get_by_id("glc_e").annotation["sbo"] == "SBO:0000247" + + +def test_biomass_pseudo_metabolite_gets_biomass_sbo(): + m = _toy_model() + add_sbo_terms(m) + assert m.metabolites.get_by_id("biomass_c").annotation["sbo"] == "SBO:0000649" + + +def test_lipid_backbone_suffix_match(): + m = _toy_model() + add_sbo_terms(m) + assert m.metabolites.get_by_id("lbb_c").annotation["sbo"] == "SBO:0000649" + + +def test_exchange_gets_exchange_sbo(): + m = _toy_model() + add_sbo_terms(m) + assert m.reactions.get_by_id("EX_glc").annotation["sbo"] == "SBO:0000627" + + +def test_transport_gets_transport_sbo(): + m = _toy_model() + add_sbo_terms(m) + assert m.reactions.get_by_id("ATP_tx").annotation["sbo"] == "SBO:0000655" + + +def test_biomass_pseudoreaction_gets_pseudo_sbo_by_default(): + m = _toy_model() + add_sbo_terms(m) + assert m.reactions.get_by_id("biomass_rxn").annotation["sbo"] == "SBO:0000629" + + +def test_legacy_bug_flag_scopes_pseudo_to_last_reaction(): + """The yeast-GEM compat flag must restrict the pseudo override to + the last reaction; if biomass_rxn is last, it still wins.""" + m = _toy_model() + add_sbo_terms(m, only_last_reaction_for_pseudo=True) + # biomass_rxn was added last → still gets the pseudo SBO. + assert m.reactions.get_by_id("biomass_rxn").annotation["sbo"] == "SBO:0000629" + + +def test_legacy_bug_flag_skips_pseudo_when_not_last(): + """Reorder: a non-pseudo reaction last; the bug flag should leave + the pseudoreaction with whatever non-pseudo SBO it already had.""" + m = _toy_model() + # Swap last two reactions so the pseudo is NOT last. + rxns = list(m.reactions) + new_order = rxns[:-2] + rxns[-1:] + rxns[-2:-1] + m.reactions._generate_index() # noqa: SLF001 (rebuild before reorder) + # Easier: copy bounds, set order via remove+re-add + m.remove_reactions([rxns[-2], rxns[-1]]) + m.add_reactions(new_order[-2:]) + add_sbo_terms(m, only_last_reaction_for_pseudo=True) + bio = m.reactions.get_by_id("biomass_rxn") + # The biomass reaction has a single bounded reactant set (atp_c -1, + # biomass +1) → 2 mets → falls to default SBO:0000176. + assert bio.annotation["sbo"] == "SBO:0000176" + + +def test_fill_semantic_preserves_existing(): + m = _toy_model() + rxn = m.reactions.get_by_id("metab1") + rxn.annotation["sbo"] = "SBO:0009999" + add_sbo_terms(m) + assert rxn.annotation["sbo"] == "SBO:0009999" + + +def test_custom_transport_detector_overrides_default(): + m = _toy_model() + add_sbo_terms(m, transport_detector=lambda _m: set()) + # ATP_tx is no longer flagged transport → falls through to default + # (it has 2 metabolites so it is not exchange/sink/demand). + assert m.reactions.get_by_id("ATP_tx").annotation["sbo"] == "SBO:0000176" + + +def test_default_transport_detector_finds_transport(): + m = _toy_model() + transports = _default_transport_detector(m) + assert "ATP_tx" in transports + assert "EX_glc" not in transports + + +# --- ΔG CSV round-trip ------------------------------------------------ + +def test_save_then_load_round_trip(tmp_path): + m = _toy_model() + m.metabolites.get_by_id("atp_c").notes["deltaG"] = "-12.34" + m.metabolites.get_by_id("glc_e").notes["deltaG"] = "0.5" + + csv = tmp_path / "met_dg.csv" + written = save_delta_g_csv(m.metabolites, csv) + assert written == len(m.metabolites) + df = pd.read_csv(csv) + assert list(df.columns) == ["Var1", "Var2"] + + # Reload on a fresh model + fresh = _toy_model() + stamped = load_delta_g_csv(fresh.metabolites, csv) + assert stamped == 2 + assert fresh.metabolites.get_by_id("atp_c").notes["deltaG"] == "-12.34" + assert fresh.metabolites.get_by_id("glc_e").notes["deltaG"] == "0.5" + + +def test_save_writes_nan_for_missing_notes(tmp_path): + m = _toy_model() # no notes set + csv = tmp_path / "met_dg.csv" + save_delta_g_csv(m.metabolites, csv) + df = pd.read_csv(csv) + assert len(df) == len(m.metabolites) + assert df["Var2"].apply(lambda v: math.isnan(v)).all() + + +def test_load_skips_nan_rows(tmp_path): + """A NaN in the CSV must NOT clobber the entity's existing note.""" + m = _toy_model() + m.metabolites.get_by_id("atp_c").notes["deltaG"] = "preserved" + + csv = tmp_path / "met_dg.csv" + pd.DataFrame( + {"Var1": ["atp_c", "glc_e"], "Var2": [math.nan, 1.0]} + ).to_csv(csv, index=False) + + load_delta_g_csv(m.metabolites, csv) + assert m.metabolites.get_by_id("atp_c").notes["deltaG"] == "preserved" + assert m.metabolites.get_by_id("glc_e").notes["deltaG"] == "1.0" + + +def test_custom_columns_and_note_key(tmp_path): + m = _toy_model() + m.metabolites.get_by_id("atp_c").notes["dG_kJ"] = "-30.5" + csv = tmp_path / "out.csv" + save_delta_g_csv( + m.metabolites, csv, + id_column="id", value_column="dG", note_key="dG_kJ", + ) + df = pd.read_csv(csv) + assert list(df.columns) == ["id", "dG"] + + fresh = _toy_model() + load_delta_g_csv( + fresh.metabolites, csv, + id_column="id", value_column="dG", note_key="dG_kJ", + ) + assert fresh.metabolites.get_by_id("atp_c").notes["dG_kJ"] == "-30.5" diff --git a/tests/test_binaries.py b/tests/test_binaries.py index d74ce0b5..152e20da 100644 --- a/tests/test_binaries.py +++ b/tests/test_binaries.py @@ -1,12 +1,13 @@ -"""Tests for raven_python.binaries (binary resolution + bundled-ZIP provisioning).""" +"""Tests for raven_toolbox.binaries (binary resolution + bundled-ZIP provisioning).""" import hashlib import shutil +import stat import zipfile from pathlib import Path import pytest -from raven_python import binaries +from raven_toolbox import binaries def test_resolve_explicit_path(): @@ -78,3 +79,77 @@ def test_ensure_binary_unhosted_platform_raises(tmp_path): registry = {"footool": {"version": "1", "provides": ["footool"], "platforms": {}}} with pytest.raises(FileNotFoundError, match="No bundled"): binaries.ensure_binary("footool", registry=registry) + + +def test_ensure_binary_rejects_path_traversal_zip(tmp_path, monkeypatch): + # A ZIP whose member escapes the extraction dir must be refused, not written + # outside the cache (ZipFile.extractall has no traversal guard of its own). + archive = tmp_path / "evil.zip" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("footool", "ok") + zf.writestr("../escape.txt", "pwned") + sha = hashlib.sha256(archive.read_bytes()).hexdigest() + registry = { + "footool": { + "version": "1.0", + "provides": ["footool"], + "platforms": {binaries.platform_key(): {"url": archive.as_uri(), "sha256": sha}}, + } + } + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache")) + with pytest.raises(ValueError, match="path traversal"): + binaries.ensure_binary("footool", registry=registry) + assert not (tmp_path / "escape.txt").exists() + + +def test_ensure_binary_rejects_symlink_zip(tmp_path, monkeypatch): + # A ZIP member flagged as a symlink must be refused: extractall would + # recreate it as a real symlink whose target the path guard never validates, + # so it could point outside the cache dir. + archive = tmp_path / "evil.zip" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("footool", "ok") + link = zipfile.ZipInfo("escape") + link.external_attr = (stat.S_IFLNK | 0o777) << 16 + zf.writestr(link, "/etc/passwd") + sha = hashlib.sha256(archive.read_bytes()).hexdigest() + registry = { + "footool": { + "version": "1.0", + "provides": ["footool"], + "platforms": {binaries.platform_key(): {"url": archive.as_uri(), "sha256": sha}}, + } + } + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache")) + with pytest.raises(ValueError, match="symlink"): + binaries.ensure_binary("footool", registry=registry) + + +def test_ensure_binary_passes_download_timeout(tmp_path, monkeypatch): + # The download must use a socket timeout so a stalled server can't hang the + # process forever. Spy on urlopen to confirm a positive timeout is forwarded. + exe = tmp_path / "footool" + exe.write_text("#!/bin/sh\necho hi\n") + archive = tmp_path / "footool.zip" + with zipfile.ZipFile(archive, "w") as zf: + zf.write(exe, "footool") + sha = hashlib.sha256(archive.read_bytes()).hexdigest() + registry = { + "footool": { + "version": "1.0", + "provides": ["footool"], + "platforms": {binaries.platform_key(): {"url": archive.as_uri(), "sha256": sha}}, + } + } + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache")) + + seen = {} + real_urlopen = binaries.urlopen + + def spy(url, *args, **kwargs): + seen["timeout"] = kwargs.get("timeout") + return real_urlopen(url, *args, **kwargs) + + monkeypatch.setattr(binaries, "urlopen", spy) + binaries.ensure_binary("footool", registry=registry) + assert isinstance(seen["timeout"], (int, float)) and seen["timeout"] > 0 diff --git a/tests/test_biomass.py b/tests/test_biomass.py new file mode 100644 index 00000000..da3e31a5 --- /dev/null +++ b/tests/test_biomass.py @@ -0,0 +1,246 @@ +"""Tests for raven_toolbox.biomass.""" +from __future__ import annotations + +import cobra +import pytest + +from raven_toolbox.biomass import ( + BiomassComponent, + BiomassConfig, + rescale_pseudoreaction, + scale_biomass, + set_gam, + sum_biomass, +) +from raven_toolbox.biomass.scale import _formula_mw + +# --- formula MW helper ---------------------------------------------- + +@pytest.mark.parametrize( + "formula, expected", + [ + ("H2O", 18.015), # 2*1.008 + 15.999 + ("ATP", 0.0), # invalid: no element 'ATP'... we raise + ("CO2", 44.008), # 12.01 + 2*15.999 + ("C6H12O6", 180.156), # 6*12.01 + 12*1.008 + 6*15.999 + ], +) +def test_formula_mw(formula, expected): + if formula == "ATP": + with pytest.raises(ValueError, match="Unknown element"): + _formula_mw(formula) + else: + assert _formula_mw(formula) == pytest.approx(expected, rel=1e-4) + + +# --- toy biomass model ---------------------------------------------- + +def _toy_biomass_model() -> tuple[cobra.Model, BiomassConfig]: + """Tiny but realistic biomass layout: + + - protein pseudoreaction: 2 amino-acid tRNAs → protein (mass_strategy mw_minus_2h) + - carbohydrate pseudoreaction: glucose → carbohydrate (mass_strategy mw) + - biomass pseudoreaction: protein + carbohydrate + GAM cofactors → biomass + """ + m = cobra.Model("toy") + m.compartments = {"c": "cytoplasm"} + + aa1 = cobra.Metabolite("aa1_c", name="alanine-tRNA", compartment="c", + charge=0, formula="C3H6N1O2") # MW ≈ 88 + aa2 = cobra.Metabolite("aa2_c", name="glycine-tRNA", compartment="c", + charge=0, formula="C2H4N1O2") # MW ≈ 74 + glc = cobra.Metabolite("glc_c", name="glucose", compartment="c", + charge=0, formula="C6H12O6") # MW = 180.156 + protein = cobra.Metabolite("protein_c", name="protein", compartment="c") + carb = cobra.Metabolite("carb_c", name="carbohydrate", compartment="c") + biomass = cobra.Metabolite("biomass_c", name="biomass", compartment="c") + h = cobra.Metabolite("h_c", name="H+", compartment="c", charge=1) + atp = cobra.Metabolite("atp_c", name="ATP", compartment="c", charge=-4) + adp = cobra.Metabolite("adp_c", name="ADP", compartment="c", charge=-3) + h2o = cobra.Metabolite("h2o_c", name="H2O", compartment="c", charge=0) + pi = cobra.Metabolite("pi_c", name="phosphate", compartment="c", charge=-2) + m.add_metabolites([aa1, aa2, glc, protein, carb, biomass, h, atp, adp, h2o, pi]) + + prot_rxn = cobra.Reaction("PROT_pseudo") + prot_rxn.name = "protein pseudoreaction" + prot_rxn.add_metabolites({aa1: -0.5, aa2: -0.5, protein: 1, h: 1}) + carb_rxn = cobra.Reaction("CARB_pseudo") + carb_rxn.name = "carbohydrate pseudoreaction" + carb_rxn.add_metabolites({glc: -0.001, carb: 1}) # 1 mmol → 180 mg + bio_rxn = cobra.Reaction("BIO_pseudo") + bio_rxn.name = "biomass pseudoreaction" + bio_rxn.add_metabolites({ + protein: -1, carb: -1, biomass: 1, + atp: -50, h2o: -50, adp: 50, h: 50, pi: 50, # GAM = 50 + }) + m.add_reactions([prot_rxn, carb_rxn, bio_rxn]) + + cfg = BiomassConfig( + biomass_rxn="BIO_pseudo", + proton_met="h_c", + components=( + BiomassComponent("protein", "protein pseudoreaction", + mass_strategy="mw_minus_2h"), + BiomassComponent("carbohydrate", "carbohydrate pseudoreaction", + mass_strategy="mw"), + ), + ) + return m, cfg + + +# --- sum_biomass ----------------------------------------------------- + +def test_sum_biomass_keys_match_components_plus_total(): + m, cfg = _toy_biomass_model() + out = sum_biomass(m, cfg) + assert set(out) == {"protein", "carbohydrate", "total"} + + +def test_sum_biomass_protein_uses_minus_2h_offset(): + m, cfg = _toy_biomass_model() + out = sum_biomass(m, cfg) + # aa1 (alanine-tRNA, C3H6N1O2, MW ≈ 88.09): MW - 2.016 + # aa2 (glycine-tRNA, C2H4N1O2, MW ≈ 74.07): MW - 2.016 + # contribs: 0.5*(88.09 - 2.016) + 0.5*(74.07 - 2.016) ≈ 79.066, /1000 + assert out["protein"] == pytest.approx(0.079066, rel=1e-3) + + +def test_sum_biomass_carbohydrate_uses_plain_mw(): + m, cfg = _toy_biomass_model() + out = sum_biomass(m, cfg) + # 0.001 mmol glucose * 180.156 g/mol / 1000 = 0.000180156 + assert out["carbohydrate"] == pytest.approx(0.000180156, rel=1e-3) + + +def test_sum_biomass_total_is_components_sum(): + m, cfg = _toy_biomass_model() + out = sum_biomass(m, cfg) + assert out["total"] == pytest.approx(out["protein"] + out["carbohydrate"]) + + +def test_sum_biomass_grams_strategy(): + m, cfg = _toy_biomass_model() + # Add a lipid component whose stoichiometry is already in grams. + lipid = cobra.Metabolite("lipid_c", name="lipid backbone", compartment="c") + m.add_metabolites([lipid]) + rxn = cobra.Reaction("LIPID_pseudo") + rxn.name = "lipid backbone pseudoreaction" + rxn.add_metabolites({lipid: 1, m.metabolites.get_by_id("glc_c"): -0.05}) + m.add_reactions([rxn]) + cfg2 = BiomassConfig.from_components( + cfg.biomass_rxn, cfg.proton_met, + [*cfg.components, + BiomassComponent("lipid_backbone", "lipid backbone pseudoreaction", + mass_strategy="grams")], + ) + out = sum_biomass(m, cfg2) + assert out["lipid_backbone"] == pytest.approx(0.05) + + +def test_sum_biomass_missing_pseudoreaction_contributes_zero(): + m, cfg = _toy_biomass_model() + cfg2 = BiomassConfig.from_components( + cfg.biomass_rxn, cfg.proton_met, + [*cfg.components, BiomassComponent("DNA", "DNA pseudoreaction")], + ) + out = sum_biomass(m, cfg2) + assert out["DNA"] == 0 + + +# --- rescale_pseudoreaction ----------------------------------------- + +def test_rescale_pseudoreaction_doubles_substrate_coefs(): + m, cfg = _toy_biomass_model() + rxn = m.reactions.get_by_id("PROT_pseudo") + aa1 = m.metabolites.get_by_id("aa1_c") + before = rxn.metabolites[aa1] + rescale_pseudoreaction(m, cfg, "protein", 2.0) + assert rxn.metabolites[aa1] == pytest.approx(2 * before) + + +def test_rescale_pseudoreaction_leaves_product_alone(): + m, cfg = _toy_biomass_model() + rxn = m.reactions.get_by_id("PROT_pseudo") + protein = m.metabolites.get_by_id("protein_c") + before = rxn.metabolites[protein] + rescale_pseudoreaction(m, cfg, "protein", 3.0) + assert rxn.metabolites[protein] == before + + +def test_rescale_pseudoreaction_rebalances_h_for_charge(): + m, cfg = _toy_biomass_model() + # Add a non-zero-charge substrate so H+ balance is meaningful. + rxn = m.reactions.get_by_id("PROT_pseudo") + aa1 = m.metabolites.get_by_id("aa1_c") + aa1.charge = -1 # negatively charged + rescale_pseudoreaction(m, cfg, "protein", 2.0) + total = sum((m_.charge or 0) * c for m_, c in rxn.metabolites.items()) + assert total == pytest.approx(0) + + +# --- scale_biomass -------------------------------------------------- + +def test_scale_biomass_lands_on_new_value(): + m, cfg = _toy_biomass_model() + target = 0.40 + scale_biomass(m, cfg, "protein", target) + out = sum_biomass(m, cfg) + assert out["protein"] == pytest.approx(target, rel=1e-6) + + +def test_scale_biomass_with_balance_keeps_total_at_one(): + m, cfg = _toy_biomass_model() + # Force a known starting total ≠ 1 by tweaking the toy model. + scale_biomass(m, cfg, "protein", 0.6, balance_out="carbohydrate") + out = sum_biomass(m, cfg) + assert out["total"] == pytest.approx(1.0, rel=1e-6) + + +def test_scale_biomass_zero_current_raises(): + m, cfg = _toy_biomass_model() + # Build a config that references a missing component → 0 current + cfg2 = BiomassConfig.from_components( + cfg.biomass_rxn, cfg.proton_met, + [*cfg.components, BiomassComponent("DNA", "DNA pseudoreaction")], + ) + with pytest.raises(ValueError, match="current"): + scale_biomass(m, cfg2, "DNA", 0.1) + + +# --- set_gam -------------------------------------------------------- + +def test_set_gam_scales_cofactor_coefficients(): + m, cfg = _toy_biomass_model() + set_gam(m, 80, biomass_rxn=cfg.biomass_rxn, + cofactor_met_names=("ATP", "ADP", "H2O", "H+", "phosphate")) + bio = m.reactions.get_by_id(cfg.biomass_rxn) + atp = m.metabolites.get_by_id("atp_c") + adp = m.metabolites.get_by_id("adp_c") + h2o = m.metabolites.get_by_id("h2o_c") + pi = m.metabolites.get_by_id("pi_c") + assert bio.metabolites[atp] == -80 + assert bio.metabolites[adp] == 80 + assert bio.metabolites[h2o] == -80 + assert bio.metabolites[pi] == 80 + + +def test_set_gam_with_ngam_sets_equality_bounds(): + m, _ = _toy_biomass_model() + ngam = cobra.Reaction("NGAM", lower_bound=0, upper_bound=1000) + atp = m.metabolites.get_by_id("atp_c") + ngam.add_metabolites({atp: -1}) + m.add_reactions([ngam]) + set_gam(m, 80, biomass_rxn="BIO_pseudo", + cofactor_met_names=("ATP",), + ngam_rxn="NGAM", ngam_value=1.5) + assert m.reactions.get_by_id("NGAM").bounds == (1.5, 1.5) + + +def test_set_gam_ignores_non_cofactor_mets(): + m, cfg = _toy_biomass_model() + bio = m.reactions.get_by_id(cfg.biomass_rxn) + protein = m.metabolites.get_by_id("protein_c") + before = bio.metabolites[protein] + set_gam(m, 80, biomass_rxn=cfg.biomass_rxn, + cofactor_met_names=("ATP",)) + assert bio.metabolites[protein] == before diff --git a/tests/test_change_grrules.py b/tests/test_change_grrules.py index d33f723f..4d22255b 100644 --- a/tests/test_change_grrules.py +++ b/tests/test_change_grrules.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.manipulation import add_reactions_from_equations, change_gene_reaction_rules +from raven_toolbox.manipulation import add_reactions_from_equations, change_gene_reaction_rules @pytest.fixture diff --git a/tests/test_comparison.py b/tests/test_comparison.py index 26a20f0b..b26c503a 100644 --- a/tests/test_comparison.py +++ b/tests/test_comparison.py @@ -4,8 +4,8 @@ import cobra import pytest -from raven_python.comparison import ModelComparison, compare_models -from raven_python.tasks import Task +from raven_toolbox.comparison import ModelComparison, compare_models +from raven_toolbox.tasks import Task def _mk(model_id: str, reactions: list[tuple[str, dict[str, int], str | None]], diff --git a/tests/test_comparison_diff.py b/tests/test_comparison_diff.py new file mode 100644 index 00000000..91ea5f76 --- /dev/null +++ b/tests/test_comparison_diff.py @@ -0,0 +1,146 @@ +"""Tests for raven_toolbox.comparison.diff — strict 2-model equality.""" +from __future__ import annotations + +import cobra +import pytest + +from raven_toolbox.comparison import DiffReport, diff_models +from raven_toolbox.comparison.diff import _normalise_gpr + + +def _mini_model(model_id: str = "m") -> cobra.Model: + """Tiny but realistic model: one reaction, one extracellular met.""" + m = cobra.Model(model_id) + a = cobra.Metabolite("a_c", name="A", compartment="c", charge=0, formula="C1") + b = cobra.Metabolite("b_e", name="B", compartment="e", charge=0, formula="C2") + m.add_metabolites([a, b]) + r = cobra.Reaction("r1", lower_bound=-1000, upper_bound=1000) + r.add_metabolites({a: -1, b: 1}) + r.gene_reaction_rule = "g1 AND g2" + r.annotation["sbo"] = "SBO:0000176" + m.add_reactions([r]) + return m + + +def test_model_equal_to_itself(): + m = _mini_model() + report = diff_models(m, m) + assert isinstance(report, DiffReport) + assert report.equal + + +def test_independent_copies_are_equal(): + a = _mini_model() + b = _mini_model() + assert diff_models(a, b).equal + + +def test_dropped_reaction_is_detected(): + a = _mini_model() + b = _mini_model() + b.remove_reactions([b.reactions[0]]) + report = diff_models(a, b) + assert not report.equal + assert any("reactions only in A" in d for d in report.differences) + + +def test_bound_diff_is_detected(): + a = _mini_model() + b = _mini_model() + b.reactions[0].lower_bound = -42 + report = diff_models(a, b) + assert not report.equal + assert any("bounds" in d for d in report.differences) + + +def test_stoich_within_tolerance_is_equal(): + a = _mini_model() + b = _mini_model() + met = next(iter(b.reactions[0].metabolites)) + b.reactions[0].add_metabolites({met: 1e-12}, combine=True) + assert diff_models(a, b, stoichiometry_tol=1e-9).equal + + +def test_stoich_outside_tolerance_is_not_equal(): + a = _mini_model() + b = _mini_model() + met = next(iter(b.reactions[0].metabolites)) + b.reactions[0].add_metabolites({met: 5.0}, combine=False) + report = diff_models(a, b, stoichiometry_tol=1e-9) + assert not report.equal + assert any("coef[" in d for d in report.differences) + + +def test_annotation_diff_detected(): + a = _mini_model() + b = _mini_model() + b.reactions[0].annotation["sbo"] = "SBO:0009999" + report = diff_models(a, b) + assert not report.equal + assert any("annotation['sbo']" in d for d in report.differences) + + +def test_ignore_annotations_suppresses_diff(): + a = _mini_model() + b = _mini_model() + b.reactions[0].annotation["sbo"] = "SBO:0009999" + assert diff_models(a, b, ignore_annotations={"sbo"}).equal + + +def test_extra_annotations_picked_up(): + a = _mini_model() + b = _mini_model() + a.reactions[0].annotation["custom-key"] = "v1" + b.reactions[0].annotation["custom-key"] = "v2" + # Default key set ignores custom-key → equal. + assert diff_models(a, b).equal + # Pull it in → diff. + assert not diff_models(a, b, extra_annotations={"custom-key"}).equal + + +@pytest.mark.parametrize( + "ga, gb", + [ + ("A and B", "a AND b"), + ("A and B", "A and B"), + ("(A or B) and C", "(a OR b) AND c"), + ], +) +def test_gpr_normalisation(ga, gb): + assert _normalise_gpr(ga) == _normalise_gpr(gb) + + +def test_max_per_category_truncates(): + """The per-category cap kicks in on per-reaction (and per-met) diffs, + not on the id-set summary line. Build a scenario with many bound + diffs across shared reactions.""" + a = cobra.Model("a") + b = cobra.Model("b") + mets_a = [cobra.Metabolite(f"m{i}_c", compartment="c") for i in range(60)] + mets_b = [cobra.Metabolite(f"m{i}_c", compartment="c") for i in range(60)] + a.add_metabolites(mets_a) + b.add_metabolites(mets_b) + for i in range(60): + ra = cobra.Reaction(f"r{i}", lower_bound=0, upper_bound=1000) + ra.add_metabolites({mets_a[i]: -1}) + a.add_reactions([ra]) + rb = cobra.Reaction(f"r{i}", lower_bound=-7, upper_bound=1000) # differs + rb.add_metabolites({mets_b[i]: -1}) + b.add_reactions([rb]) + report = diff_models(a, b, max_per_category=5) + assert not report.equal + assert any("truncated at 5" in d for d in report.differences) + + +def test_report_str_lists_differences(): + a = _mini_model() + b = _mini_model() + b.remove_reactions([b.reactions[0]]) + text = str(diff_models(a, b)) + assert "Models differ" in text + assert "reactions only in A" in text + + +def test_report_str_when_equal(): + m = _mini_model() + assert str(diff_models(m, m)) == "Models are semantically equal." diff --git a/tests/test_conditions.py b/tests/test_conditions.py new file mode 100644 index 00000000..6619fce9 --- /dev/null +++ b/tests/test_conditions.py @@ -0,0 +1,211 @@ +"""Tests for raven_toolbox.conditions.apply (generic condition mechanism).""" +from __future__ import annotations + +import io + +import cobra +import pytest +from ruamel.yaml import YAML + +from raven_toolbox.conditions import ( + apply_condition, + load_condition, + set_reaction_bounds, +) + +_SAFE_YAML = YAML(typ="safe") + + +def _yaml_dump(cfg: dict) -> str: + """Helper: dump a dict to YAML via ruamel.yaml (PyYAML.safe_dump + equivalent). Used here because PyYAML isn't a project dependency.""" + buf = io.StringIO() + _SAFE_YAML.dump(cfg, buf) + return buf.getvalue() + + +def _toy_model() -> cobra.Model: + m = cobra.Model("toy") + m.compartments = {"c": "cytoplasm", "e": "extracellular"} + + mets = { + "atp_c": cobra.Metabolite("atp_c", name="ATP", compartment="c", charge=-4), + "glc_e": cobra.Metabolite("glc_e", name="glucose", compartment="e", charge=0), + "h_c": cobra.Metabolite("h_c", name="H+", compartment="c", charge=1), + "fad": cobra.Metabolite("fad", name="FAD", compartment="c", charge=-2), + "fadh2": cobra.Metabolite("fadh2", name="FADH2", compartment="c", charge=0), + "heme": cobra.Metabolite("heme", name="heme a", compartment="c", charge=-2), + } + m.add_metabolites(list(mets.values())) + + ex = cobra.Reaction("EX_glc", lower_bound=-10, upper_bound=1000) + ex.add_metabolites({mets["glc_e"]: -1}) + cofactor = cobra.Reaction("cofac", lower_bound=0, upper_bound=1000) + cofactor.add_metabolites({mets["heme"]: -1, mets["fad"]: -1, mets["h_c"]: 3}) + biomass = cobra.Reaction("bio", lower_bound=0, upper_bound=1000) + biomass.add_metabolites({mets["atp_c"]: -1}) + blocked = cobra.Reaction("blocked", lower_bound=-1000, upper_bound=1000) + blocked.add_metabolites({mets["atp_c"]: -1, mets["glc_e"]: 1}) + m.add_reactions([ex, cofactor, biomass, blocked]) + return m + + +# --- prelude ---------------------------------------------------------- + +def test_prelude_reset_exchanges_zeroes_lb_and_caps_ub(): + m = _toy_model() + apply_condition(m, {"prelude": {"reset_exchanges": "out"}}) + ex = m.reactions.get_by_id("EX_glc") + assert ex.lower_bound == 0 + assert ex.upper_bound == 1000 + + +# --- bounds ----------------------------------------------------------- + +def test_bounds_apply_lb_only(): + m = _toy_model() + apply_condition(m, {"bounds": [{"rxn": "EX_glc", "lb": -1000}]}) + assert m.reactions.get_by_id("EX_glc").lower_bound == -1000 + # ub preserved + assert m.reactions.get_by_id("EX_glc").upper_bound == 1000 + + +def test_bounds_apply_both(): + m = _toy_model() + apply_condition(m, {"bounds": [{"rxn": "blocked", "lb": 0, "ub": 0}]}) + assert m.reactions.get_by_id("blocked").bounds == (0, 0) + + +def test_bounds_lb_gt_ub_bypasses_cobra_validator(): + m = _toy_model() + apply_condition(m, {"bounds": [{"rxn": "blocked", "lb": 1000, "ub": 0}]}) + rxn = m.reactions.get_by_id("blocked") + assert rxn.lower_bound == 1000 + assert rxn.upper_bound == 0 + + +def test_bounds_missing_rxn_warns_and_continues(): + m = _toy_model() + with pytest.warns(UserWarning, match="not_a_real_rxn"): + apply_condition(m, {"bounds": [{"rxn": "not_a_real_rxn", "lb": 0}]}) + + +# --- cofactor pseudoreaction ----------------------------------------- + +def test_remove_met_zeroes_coefficient(): + m = _toy_model() + apply_condition( + m, + { + "cofactor_pseudoreaction": { + "rxn_id": "cofac", + "remove_mets": [{"met": "heme"}], + } + }, + ) + rxn = m.reactions.get_by_id("cofac") + heme = m.metabolites.get_by_id("heme") + assert rxn.metabolites.get(heme, 0) == 0 + + +def test_charge_balance_recomputed_after_removal(): + m = _toy_model() + apply_condition( + m, + { + "cofactor_pseudoreaction": { + "rxn_id": "cofac", + "remove_mets": [{"met": "heme"}], + "charge_balance_met": "h_c", + } + }, + ) + rxn = m.reactions.get_by_id("cofac") + # heme (-1·-2) and fad (-1·-2) contributed +4 charge. + # After heme removal: only fad (-1·-2) contributes +2. + # H+ (charge +1) coef should be -2 to zero the net. + h_c = m.metabolites.get_by_id("h_c") + assert rxn.metabolites[h_c] == -2 + + +# --- biomass stoichiometry delta ------------------------------------- + +def test_biomass_stoichiometry_delta_combines_with_existing(): + m = _toy_model() + bio = m.reactions.get_by_id("bio") + atp = m.metabolites.get_by_id("atp_c") + before = bio.metabolites[atp] + apply_condition( + m, + { + "biomass_stoichiometry_delta": { + "rxn_id": "bio", + "add": [{"met": "atp_c", "coef": 0.5}], + } + }, + ) + assert bio.metabolites[atp] == pytest.approx(before + 0.5) + + +# --- expected_uptake_count ------------------------------------------- + +def test_expected_uptake_count_mismatch_warns(): + m = _toy_model() + cfg = { + "bounds": [{"rxn": "EX_glc", "lb": -1000}], + "expected_uptake_count": 5, + } + with pytest.warns(UserWarning, match="Expected 5 uptake reactions, applied 1"): + apply_condition(m, cfg) + + +def test_expected_uptake_count_match_silent(recwarn): + m = _toy_model() + apply_condition( + m, + { + "bounds": [{"rxn": "EX_glc", "lb": -1000}], + "expected_uptake_count": 1, + }, + ) + assert not any("Expected" in str(w.message) for w in recwarn) + + +# --- yaml path entrypoint -------------------------------------------- + +def test_apply_condition_accepts_yaml_path(tmp_path): + cfg = {"bounds": [{"rxn": "EX_glc", "lb": -42}]} + path = tmp_path / "cond.yml" + path.write_text(_yaml_dump(cfg)) + m = _toy_model() + apply_condition(m, path) + assert m.reactions.get_by_id("EX_glc").lower_bound == -42 + + +def test_load_condition_round_trip(tmp_path): + cfg = {"name": "x", "bounds": [{"rxn": "r1", "lb": 0}]} + path = tmp_path / "cond.yml" + path.write_text(_yaml_dump(cfg)) + assert load_condition(path) == cfg + + +def test_load_condition_missing_path_raises(tmp_path): + with pytest.raises(FileNotFoundError): + load_condition(tmp_path / "nope.yml") + + +# --- set_reaction_bounds public helper ------------------------------- + +def test_set_reaction_bounds_normal_case(): + m = _toy_model() + rxn = m.reactions.get_by_id("EX_glc") + set_reaction_bounds(rxn, -5, 5) + assert rxn.bounds == (-5, 5) + + +def test_set_reaction_bounds_infeasible_case(): + m = _toy_model() + rxn = m.reactions.get_by_id("EX_glc") + set_reaction_bounds(rxn, 100, 0) + assert rxn.lower_bound == 100 + assert rxn.upper_bound == 0 diff --git a/tests/test_curation.py b/tests/test_curation.py new file mode 100644 index 00000000..e7f9bd0b --- /dev/null +++ b/tests/test_curation.py @@ -0,0 +1,270 @@ +"""Tests for raven_toolbox.curation.batch.""" +from __future__ import annotations + +import cobra +import pandas as pd +import pytest + +from raven_toolbox.curation import ( + CurationResult, + batch_curate, + batch_curate_from_tsv, +) + + +def _base_model() -> cobra.Model: + """Small yeast-flavoured model with a couple of mets/genes/rxns.""" + m = cobra.Model("base") + m.compartments = {"c": "cytoplasm", "e": "extracellular"} + atp = cobra.Metabolite("s_0001", name="ATP", compartment="c", + formula="C10H12N5O13P3", charge=-4) + adp = cobra.Metabolite("s_0002", name="ADP", compartment="c", + formula="C10H12N5O10P2", charge=-3) + glc_e = cobra.Metabolite("s_0003", name="glucose", compartment="e", + formula="C6H12O6", charge=0) + m.add_metabolites([atp, adp, glc_e]) + from cobra.core.gene import Gene + m.genes.append(Gene("YAL001C", name="A1")) + r = cobra.Reaction("r_0001", lower_bound=0, upper_bound=1000) + r.name = "ATP hydrolysis" + r.add_metabolites({atp: -1, adp: 1}) + r.gene_reaction_rule = "YAL001C" + m.add_reactions([r]) + return m + + +# --- metabolites ------------------------------------------------------ + +def test_add_new_metabolite(): + m = _base_model() + df = pd.DataFrame([ + {"metNames": "NADH", "comps": "c", "formula": "C21H27N7O14P2", + "charge": -2, "inchi": "", "metNotes": "added by curation"}, + ]) + result = batch_curate(m, mets_df=df, met_id_prefix="s_") + assert isinstance(result, CurationResult) + assert result.added_metabolites == ["s_0004"] + assert len(result.updated_metabolites) == 0 + new = m.metabolites.get_by_id("s_0004") + assert new.name == "NADH" + assert new.compartment == "c" + assert new.formula == "C21H27N7O14P2" + assert new.charge == -2 + assert new.notes.get("metNotes") == "added by curation" + + +def test_update_existing_metabolite(): + m = _base_model() + df = pd.DataFrame([ + {"metNames": "ATP", "comps": "c", "formula": "C10H16N5O13P3", + "charge": -3, "inchi": "InChI=ATP", "metNotes": ""}, + ]) + with pytest.warns(UserWarning, match="overwritten"): + result = batch_curate(m, mets_df=df, met_id_prefix="s_") + assert result.added_metabolites == [] + assert result.updated_metabolites == ["s_0001"] + atp = m.metabolites.get_by_id("s_0001") + assert atp.formula == "C10H16N5O13P3" # overwritten + assert atp.charge == -3 + assert atp.annotation["inchi"] == "InChI=ATP" + + +def test_miriam_columns_auto_detected(): + m = _base_model() + df = pd.DataFrame([ + {"metNames": "NADH", "comps": "c", "formula": "C21H27N7O14P2", + "charge": -2, "inchi": "", "metNotes": "", + "kegg.compound": "C00004", "chebi": "CHEBI:16908"}, + ]) + batch_curate(m, mets_df=df, met_id_prefix="s_") + new = m.metabolites.get_by_id("s_0004") + assert new.annotation["kegg.compound"] == "C00004" + assert new.annotation["chebi"] == "CHEBI:16908" + + +def test_new_metabolite_id_increment_preserves_width(): + """If existing ids are s_0001, s_0002, s_0003 (width 4), new ids + should be s_0004, s_0005, … not s_4 / s_5.""" + m = _base_model() + df = pd.DataFrame([ + {"metNames": "X", "comps": "c"}, + {"metNames": "Y", "comps": "c"}, + ]) + result = batch_curate(m, mets_df=df, met_id_prefix="s_") + assert result.added_metabolites == ["s_0004", "s_0005"] + + +# --- genes ------------------------------------------------------------ + +def test_add_new_gene(): + m = _base_model() + df = pd.DataFrame([ + {"genes": "YBR123C", "geneShortNames": "B2", "uniprot": "P12345"}, + ]) + result = batch_curate(m, genes_df=df) + assert result.added_genes == ["YBR123C"] + g = m.genes.get_by_id("YBR123C") + assert g.name == "B2" + assert g.annotation["uniprot"] == "P12345" + + +def test_update_existing_gene(): + m = _base_model() + df = pd.DataFrame([ + {"genes": "YAL001C", "geneShortNames": "A1_NEW", "uniprot": "P99999"}, + ]) + with pytest.warns(UserWarning, match="overwritten"): + batch_curate(m, genes_df=df) + g = m.genes.get_by_id("YAL001C") + assert g.name == "A1_NEW" + assert g.annotation["uniprot"] == "P99999" + + +# --- reactions -------------------------------------------------------- + +def test_add_new_reaction_with_existing_mets(): + m = _base_model() + rxns_df = pd.DataFrame([ + {"rxnNames": "ADP phosphorylation", "grRules": "YBR456W", + "lb": -1000, "ub": 1000, "rev": 1, "subSystems": "energy", + "eccodes": "2.7.4.6", "rxnNotes": "", "rxnReferences": "", + "rxnConfidenceScores": 3, "kegg.reaction": "R00187"}, + ]) + coeffs_df = pd.DataFrame([ + {"rxnNames": "ADP phosphorylation", "metNames": "ADP", "comps": "c", + "coefficient": -1.0}, + {"rxnNames": "ADP phosphorylation", "metNames": "ATP", "comps": "c", + "coefficient": 1.0}, + ]) + result = batch_curate(m, rxns_df=rxns_df, rxns_coeffs_df=coeffs_df, + rxn_id_prefix="r_") + assert result.added_reactions == ["r_0002"] + new = m.reactions.get_by_id("r_0002") + assert new.name == "ADP phosphorylation" + assert new.gene_reaction_rule == "YBR456W" + assert new.annotation["ec-code"] == "2.7.4.6" + assert new.annotation["kegg.reaction"] == "R00187" + assert new.notes.get("rxnConfidenceScores") == "3" + + +def test_add_new_reaction_joins_list_subsystems(): + # RAVEN allows a reaction in several subsystems (a list). The new-reaction + # path must ``;``-join it like the update path does, not emit a str(list) repr. + m = _base_model() + rxns_df = pd.DataFrame([ + {"rxnNames": "ADP phosphorylation", "grRules": "YBR456W", + "lb": -1000, "ub": 1000, "rev": 1, + "subSystems": ["glycolysis", "energy"], + "eccodes": "2.7.4.6", "rxnNotes": "", "rxnReferences": "", + "rxnConfidenceScores": 3, "kegg.reaction": "R00187"}, + ]) + coeffs_df = pd.DataFrame([ + {"rxnNames": "ADP phosphorylation", "metNames": "ADP", "comps": "c", + "coefficient": -1.0}, + {"rxnNames": "ADP phosphorylation", "metNames": "ATP", "comps": "c", + "coefficient": 1.0}, + ]) + result = batch_curate(m, rxns_df=rxns_df, rxns_coeffs_df=coeffs_df, + rxn_id_prefix="r_") + assert result.added_reactions == ["r_0002"] + assert m.reactions.get_by_id("r_0002").subsystem == "glycolysis;energy" + + +def test_update_existing_reaction_by_stoichiometry(): + m = _base_model() + rxns_df = pd.DataFrame([ + {"rxnNames": "ATP hydrolysis renamed", "grRules": "YAL999W", + "lb": -100, "ub": 100, "rev": 1, + "subSystems": "", "eccodes": "3.6.1.3", "rxnNotes": "updated", + "rxnReferences": "", "rxnConfidenceScores": 2}, + ]) + coeffs_df = pd.DataFrame([ + {"rxnNames": "ATP hydrolysis renamed", "metNames": "ATP", + "comps": "c", "coefficient": -1.0}, + {"rxnNames": "ATP hydrolysis renamed", "metNames": "ADP", + "comps": "c", "coefficient": 1.0}, + ]) + with pytest.warns(UserWarning, match="overwritten"): + result = batch_curate(m, rxns_df=rxns_df, rxns_coeffs_df=coeffs_df, + rxn_id_prefix="r_") + assert result.updated_reactions == ["r_0001"] + rxn = m.reactions.get_by_id("r_0001") + assert rxn.name == "ATP hydrolysis renamed" + assert rxn.gene_reaction_rule == "YAL999W" + assert rxn.lower_bound == -100 + assert rxn.notes.get("rxnNotes") == "updated" + + +def test_rxn_with_unknown_met_raises(): + m = _base_model() + rxns_df = pd.DataFrame([ + {"rxnNames": "uses unknown", "grRules": "", "lb": 0, "ub": 1000, + "rev": 0, "subSystems": "", "eccodes": "", "rxnNotes": "", + "rxnReferences": "", "rxnConfidenceScores": ""}, + ]) + coeffs_df = pd.DataFrame([ + {"rxnNames": "uses unknown", "metNames": "mystery", "comps": "c", + "coefficient": -1.0}, + ]) + with pytest.raises(ValueError, match="mystery"): + batch_curate(m, rxns_df=rxns_df, rxns_coeffs_df=coeffs_df, + rxn_id_prefix="r_") + + +def test_must_provide_both_rxn_tables(): + m = _base_model() + rxns_df = pd.DataFrame([{"rxnNames": "X"}]) + with pytest.raises(ValueError, match="must be provided together"): + batch_curate(m, rxns_df=rxns_df) + + +def test_add_new_met_and_use_in_new_rxn(): + """Common curation pattern: introduce a new metabolite + a new + reaction that uses it, in a single call.""" + m = _base_model() + mets_df = pd.DataFrame([ + {"metNames": "NADH", "comps": "c", "formula": "C21H27N7O14P2", + "charge": -2, "inchi": "", "metNotes": ""}, + ]) + rxns_df = pd.DataFrame([ + {"rxnNames": "made up", "grRules": "", "lb": 0, "ub": 1000, + "rev": 0, "subSystems": "", "eccodes": "", "rxnNotes": "", + "rxnReferences": "", "rxnConfidenceScores": ""}, + ]) + coeffs_df = pd.DataFrame([ + {"rxnNames": "made up", "metNames": "ATP", "comps": "c", + "coefficient": -1.0}, + {"rxnNames": "made up", "metNames": "NADH", "comps": "c", + "coefficient": 1.0}, + ]) + result = batch_curate( + m, mets_df=mets_df, rxns_df=rxns_df, rxns_coeffs_df=coeffs_df, + met_id_prefix="s_", rxn_id_prefix="r_", + ) + assert result.added_metabolites == ["s_0004"] + assert result.added_reactions == ["r_0002"] + new_rxn = m.reactions.get_by_id("r_0002") + new_met = m.metabolites.get_by_id("s_0004") + assert new_met in new_rxn.metabolites + + +# --- from_tsv --------------------------------------------------------- + +def test_from_tsv_round_trip(tmp_path): + m = _base_model() + mets_path = tmp_path / "mets.tsv" + mets_path.write_text( + "metNames\tcomps\tformula\tcharge\tinchi\tmetNotes\tkegg.compound\n" + "NADH\tc\tC21H27N7O14P2\t-2\t\t\tC00004\n" + ) + result = batch_curate_from_tsv(m, mets_tsv=mets_path, met_id_prefix="s_") + assert result.added_metabolites == ["s_0004"] + new = m.metabolites.get_by_id("s_0004") + assert new.annotation["kegg.compound"] == "C00004" + + +def test_empty_call_no_op(): + m = _base_model() + result = batch_curate(m) + assert not result + assert result.added_metabolites == [] diff --git a/tests/test_data.py b/tests/test_data.py index 714c3a91..d3eb013b 100644 --- a/tests/test_data.py +++ b/tests/test_data.py @@ -1,66 +1,85 @@ """Tests for ensure_data (data.py). Uses file:// URLs to avoid the network.""" +import gzip import hashlib +import io +import tarfile import pytest -from raven_python.data import ( - CORE_KEGG_FILES, +from raven_toolbox import data +from raven_toolbox.data import ( ensure_data_file, ensure_kegg_data, + ensure_kegg_hmm_library, + ensure_kegg_taxonomy, ) -def _sha256(data: bytes) -> str: - return hashlib.sha256(data).hexdigest() +def _sha256(b: bytes) -> str: + return hashlib.sha256(b).hexdigest() @pytest.fixture def served(tmp_path, monkeypatch): - """A fake registry served from local files, with the cache pointed at tmp.""" + """A fake registry served from local files, with the cache pointed at tmp. + + The core artefacts are delivered as a single version-prefixed bundle + (``v1_core.tar.gz``) that ``ensure_kegg_data`` extracts; taxonomy is a separate + file. ``core`` maps each bundled member name -> its raw payload. + """ src = tmp_path / "src" src.mkdir() - payloads = { - "reference_model.yml.gz": b"!!omap model bytes", - "ko_reaction.tsv.gz": b"ko\treaction\n", - "ko_names.tsv.gz": b"ko\tname\n", - "organism_gene_ko.tsv.xz": b"organism\tgene\tko\n", - "rxn_flags.tsv.gz": b"reaction\tspontaneous\n", + ver = "v1" + core = { + f"{ver}_reference_model.yml.gz": b"!!omap model bytes", + f"{ver}_ko_reaction.tsv.gz": b"ko\treaction\n", + f"{ver}_ko_names.tsv.gz": b"ko\tname\n", + f"{ver}_organism_gene_ko.tsv.gz": b"organism\tgene\tko\n", + f"{ver}_rxn_flags.tsv.gz": b"reaction\tspontaneous\n", + } + bundle = src / f"{ver}_core.tar.gz" + with tarfile.open(bundle, "w:gz") as tar: + for name, payload in core.items(): + info = tarfile.TarInfo(name=name) + info.size = len(payload) + tar.addfile(info, io.BytesIO(payload)) + taxonomy = src / f"{ver}_taxonomy.gz" + taxonomy.write_bytes(b"# Prokaryotes\n") + + files = { + f"{ver}_core.tar.gz": {"url": bundle.as_uri(), "sha256": _sha256(bundle.read_bytes())}, + f"{ver}_taxonomy.gz": {"url": taxonomy.as_uri(), "sha256": _sha256(taxonomy.read_bytes())}, } - files = {} - for name, data in payloads.items(): - path = src / name - path.write_bytes(data) - files[name] = {"url": path.as_uri(), "sha256": _sha256(data)} - registry = {"kegg": {"version": "v1", "files": files}} + registry = {"kegg": {"version": ver, "files": files}} cache = tmp_path / "cache" monkeypatch.setenv("XDG_CACHE_HOME", str(cache)) - return registry, cache, payloads + return registry, cache, core def test_ensure_data_file_downloads_and_caches(served): - registry, cache, payloads = served - path = ensure_data_file("kegg", "ko_reaction.tsv.gz", registry=registry) - assert path == cache / "raven_python" / "data" / "kegg-v1" / "ko_reaction.tsv.gz" - assert path.read_bytes() == payloads["ko_reaction.tsv.gz"] + registry, cache, _ = served + path = ensure_data_file("kegg", "v1_taxonomy.gz", registry=registry) + assert path == cache / "raven_toolbox" / "data" / "kegg-v1" / "v1_taxonomy.gz" + assert path.read_bytes() == b"# Prokaryotes\n" -def test_ensure_data_file_reuses_cache(served, monkeypatch): +def test_ensure_data_file_reuses_cache(served): registry, _, _ = served - first = ensure_data_file("kegg", "ko_names.tsv.gz", registry=registry) + first = ensure_data_file("kegg", "v1_taxonomy.gz", registry=registry) # Break the URL: a second call must hit the cache, not re-download. - registry["kegg"]["files"]["ko_names.tsv.gz"]["url"] = "file:///nonexistent" - second = ensure_data_file("kegg", "ko_names.tsv.gz", registry=registry) + registry["kegg"]["files"]["v1_taxonomy.gz"]["url"] = "file:///nonexistent" + second = ensure_data_file("kegg", "v1_taxonomy.gz", registry=registry) assert first == second and second.exists() def test_sha256_mismatch_rejected(served): registry, cache, _ = served - registry["kegg"]["files"]["rxn_flags.tsv.gz"]["sha256"] = "0" * 64 + registry["kegg"]["files"]["v1_taxonomy.gz"]["sha256"] = "0" * 64 with pytest.raises(ValueError, match="SHA256 mismatch"): - ensure_data_file("kegg", "rxn_flags.tsv.gz", registry=registry) + ensure_data_file("kegg", "v1_taxonomy.gz", registry=registry) # The corrupt partial download must not be left behind. - assert not (cache / "raven_python" / "data" / "kegg-v1" / "rxn_flags.tsv.gz").exists() + assert not (cache / "raven_toolbox" / "data" / "kegg-v1" / "v1_taxonomy.gz").exists() def test_unknown_dataset_actionable_error(served): @@ -75,15 +94,52 @@ def test_unknown_file_lists_available(served): ensure_data_file("kegg", "missing.tsv.gz", registry=registry) -def test_ensure_kegg_data_fetches_core_set(served): - registry, cache, _ = served +def test_ensure_kegg_data_extracts_core_bundle(served): + registry, cache, core = served out = ensure_kegg_data(registry=registry) - assert out == cache / "raven_python" / "data" / "kegg-v1" - for name in CORE_KEGG_FILES: - assert (out / name).is_file() + assert out == cache / "raven_toolbox" / "data" / "kegg-v1" + # The single bundle is fetched and its members extracted into the cache dir. + for name, payload in core.items(): + member = out / name + assert member.is_file() and member.read_bytes() == payload + + +def test_ensure_kegg_taxonomy(served): + registry, cache, _ = served + path = ensure_kegg_taxonomy(registry=registry) + assert path == cache / "raven_toolbox" / "data" / "kegg-v1" / "v1_taxonomy.gz" + assert path.is_file() -def test_empty_registry_raises(): - # The shipped registry is empty until artefacts are published. +def test_ensure_kegg_hmm_library_decompresses(served, tmp_path): + registry, cache, _ = served + raw = b"HMMER3/f [3.4]\nNAME K00001\n//\n" + blob = gzip.compress(raw, mtime=0) + gz = tmp_path / "src" / "v1_prokaryotes.hmm.gz" + gz.write_bytes(blob) + registry["kegg"]["files"]["v1_prokaryotes.hmm.gz"] = { + "url": gz.as_uri(), + "sha256": _sha256(blob), + } + library = ensure_kegg_hmm_library("prokaryotes", registry=registry) + assert library.name == "v1_prokaryotes.hmm" + assert library.read_bytes() == raw # decompressed flatfile, no hmmpress + + # Second call: already decompressed, returns the same cached library. + assert ensure_kegg_hmm_library("prokaryotes", registry=registry) == library + + +def test_unregistered_dataset_raises(): + # An unpublished dataset still raises an actionable error against the shipped registry. with pytest.raises(FileNotFoundError, match="No data artefacts registered"): - ensure_data_file("kegg", "ko_reaction.tsv.gz") + ensure_data_file("metacyc", "x") + + +def test_shipped_registry_has_expected_assets(): + # The published registry holds the core bundle, both HMM libraries, and taxonomy. + kegg = data._DATA_REGISTRY["kegg"] + ver = kegg["version"] + names = set(kegg["files"]) + assert f"{ver}_core.tar.gz" in names + assert {f"{ver}_{d}.hmm.gz" for d in ("prokaryotes", "eukaryotes")} <= names + assert f"{ver}_taxonomy.gz" in names diff --git a/tests/test_gapfilling.py b/tests/test_gapfilling.py index b92e9829..3c37c576 100644 --- a/tests/test_gapfilling.py +++ b/tests/test_gapfilling.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.gapfilling import GapFillResult, connect_blocked_reactions +from raven_toolbox.gapfilling import GapFillResult, connect_blocked_reactions def _met(mid): @@ -37,6 +37,12 @@ def draft_and_template(): # --------------------------------------------------------------------------- # # Connectivity gap-fill # --------------------------------------------------------------------------- # +def test_non_positive_penalty_rejected(draft_and_template): + draft, template = draft_and_template + with pytest.raises(ValueError, match="penalty must be"): + connect_blocked_reactions(draft, template, penalty=0) + + def test_fill_gaps_connects_blocked_reaction(draft_and_template): draft, template = draft_and_template assert "r1" in cobra.flux_analysis.find_blocked_reactions(draft) # precondition @@ -48,6 +54,16 @@ def test_fill_gaps_connects_blocked_reaction(draft_and_template): assert "r_unneeded" not in res.added_reactions # irrelevant template rxn not added +def test_fill_gaps_handles_infinite_template_bound(draft_and_template): + # A template reaction with an unbounded (inf) bound must not put an infinite + # coefficient into the big-M MILP; it is clamped to a finite surrogate. + draft, template = draft_and_template + template.reactions.get_by_id("r2").upper_bound = float("inf") + res = connect_blocked_reactions(draft, template) + assert "r1" in res.newly_connected + assert "r2" in res.added_reactions + + def test_fill_gaps_returns_working_model_that_unblocks(draft_and_template): draft, template = draft_and_template res = connect_blocked_reactions(draft, template) diff --git a/tests/test_init.py b/tests/test_init.py index a61ee193..d9d46f30 100644 --- a/tests/test_init.py +++ b/tests/test_init.py @@ -1,8 +1,9 @@ """Tests for the INIT MILP (init/init.py, Phase 4c).""" import cobra import pytest +from cobra.exceptions import OptimizationError -from raven_python.init import InitResult, run_init +from raven_toolbox.init import InitResult, run_init def _met(mid): @@ -10,24 +11,10 @@ def _met(mid): @pytest.fixture -def model(): - """EX_A -> A -(r1)-> B -(r2)-> C -(r3)-> D, with A uptake and excretion allowed. - - r1, r2 are good (positive score); r3 is bad (negative score). - """ - m = cobra.Model("net") - A, B, C, D = _met("A_c"), _met("B_c"), _met("C_c"), _met("D_c") - m.add_metabolites([A, B, C, D]) - exa = cobra.Reaction("EX_A", lower_bound=-1000, upper_bound=1000) - exa.add_metabolites({A: -1}) # negative flux = uptake of A - r1 = cobra.Reaction("r1", lower_bound=0, upper_bound=1000) - r1.add_metabolites({A: -1, B: 1}) - r2 = cobra.Reaction("r2", lower_bound=0, upper_bound=1000) - r2.add_metabolites({B: -1, C: 1}) - r3 = cobra.Reaction("r3", lower_bound=0, upper_bound=1000) - r3.add_metabolites({C: -1, D: 1}) - m.add_reactions([exa, r1, r2, r3]) - return m +def model(linear_chain_model): + # The linear-chain INIT model now lives in tests/conftest.py (it was built + # identically here, in test_init_build.py and test_init_solvers.py). + return linear_chain_model def test_keeps_positive_drops_negative(model): @@ -40,6 +27,21 @@ def test_keeps_positive_drops_negative(model): assert "r3" not in kept +def test_infeasible_milp_raises_optimization_error(): + # A lone irreversible reaction that drains A (no producer) forced essential cannot + # satisfy steady state without excretion -> the solver is infeasible. raven aligns + # with cobra by raising cobra.exceptions.OptimizationError (not bare RuntimeError). + m = cobra.Model("infeasible") + A = _met("A_c") + m.add_metabolites([A]) + drain = cobra.Reaction("drain", lower_bound=0, upper_bound=1000) + drain.add_metabolites({A: -1}) + m.add_reactions([drain]) + with pytest.raises(OptimizationError): + run_init(m, {"drain": 1.0}, essential_rxns=["drain"], + prod_weight=0.0, allow_excretion=False) + + def test_negative_scores_emptied_when_no_reward(model): # All reactions negative and no production reward -> keep nothing (empty optimum). scores = {r.id: -1.0 for r in model.reactions} diff --git a/tests/test_init_build.py b/tests/test_init_build.py index cbc566f0..be0588de 100644 --- a/tests/test_init_build.py +++ b/tests/test_init_build.py @@ -4,7 +4,7 @@ import cobra import pytest -from raven_python.init import ( +from raven_toolbox.init import ( InitModelResult, gene_scores_from_expression, get_init_model, @@ -78,22 +78,9 @@ def test_expression_per_gene_reference(): # get_init_model pipeline # --------------------------------------------------------------------------- # @pytest.fixture -def model(): - m = cobra.Model("net") - A, B, C, D = (cobra.Metabolite(x, name=x[:-2], compartment="c") for x in ("A_c", "B_c", "C_c", "D_c")) - m.add_metabolites([A, B, C, D]) - exa = cobra.Reaction("EX_A", lower_bound=-1000, upper_bound=1000) - exa.add_metabolites({A: -1}) - r1 = cobra.Reaction("r1", lower_bound=0, upper_bound=1000) - r1.add_metabolites({A: -1, B: 1}) - r2 = cobra.Reaction("r2", lower_bound=0, upper_bound=1000) - r2.add_metabolites({B: -1, C: 1}) - r3 = cobra.Reaction("r3", lower_bound=0, upper_bound=1000) - r3.add_metabolites({C: -1, D: 1}) - m.add_reactions([exa, r1, r2, r3]) - for r, rule in (("r1", "g1"), ("r2", "g2"), ("r3", "g3")): - m.reactions.get_by_id(r).gene_reaction_rule = rule - return m +def model(linear_chain_model_with_genes): + # Shared linear-chain INIT model (with gene rules g1/g2/g3) — see tests/conftest.py. + return linear_chain_model_with_genes def test_get_init_model_from_gene_scores(model): diff --git a/tests/test_init_ftinit.py b/tests/test_init_ftinit.py index 58f0542e..2a58c436 100644 --- a/tests/test_init_ftinit.py +++ b/tests/test_init_ftinit.py @@ -19,8 +19,8 @@ import pytest from tinit_oracles import TEST_MODEL_SCORES, expr_for_rxn_score, make_test_model -from raven_python.init import FtInitResult, run_ftinit, run_init -from raven_python.init.score import gene_scores_from_expression, score_reactions_from_genes +from raven_toolbox.init import FtInitResult, run_ftinit, run_init +from raven_toolbox.init.score import gene_scores_from_expression, score_reactions_from_genes _LOOP = {"R4", "R6", "R9", "R10"} # the score-optimal subnetwork (8.0) diff --git a/tests/test_init_genes.py b/tests/test_init_genes.py index 862ca17b..d59085b6 100644 --- a/tests/test_init_genes.py +++ b/tests/test_init_genes.py @@ -5,7 +5,7 @@ """ import cobra -from raven_python.init import remove_low_score_genes +from raven_toolbox.init import remove_low_score_genes def _model(rule: str) -> cobra.Model: diff --git a/tests/test_init_merge.py b/tests/test_init_merge.py index f6fea8a8..c526b145 100644 --- a/tests/test_init_merge.py +++ b/tests/test_init_merge.py @@ -17,7 +17,7 @@ make_test_model4, ) -from raven_python.init import group_rxn_scores, merge_linear +from raven_toolbox.init import group_rxn_scores, merge_linear def test_test_model_group_ids(): diff --git a/tests/test_init_oracles.py b/tests/test_init_oracles.py index 3f3e52d4..d8c1654a 100644 --- a/tests/test_init_oracles.py +++ b/tests/test_init_oracles.py @@ -17,7 +17,7 @@ make_test_model5, ) -from raven_python.init.score import gene_scores_from_expression, score_reactions_from_genes +from raven_toolbox.init.score import gene_scores_from_expression, score_reactions_from_genes @pytest.mark.parametrize( diff --git a/tests/test_init_pipeline.py b/tests/test_init_pipeline.py index bf2a2ac0..910d727f 100644 --- a/tests/test_init_pipeline.py +++ b/tests/test_init_pipeline.py @@ -14,14 +14,14 @@ make_test_task, ) -from raven_python.init import ( +from raven_toolbox.init import ( classify_reactions, ftinit, get_init_steps, prep_init_model, score_reactions_from_genes, ) -from raven_python.init.score import gene_scores_from_expression +from raven_toolbox.init.score import gene_scores_from_expression def _scores(model): @@ -112,7 +112,7 @@ def test_orient_forward_reverses_a_reversible_reaction(): """_orient_forward(rxn, -1) flips stoichiometry and makes it irreversible forward.""" import cobra - from raven_python.init.prep import _orient_forward + from raven_toolbox.init.prep import _orient_forward m = cobra.Model("o") a, b = (cobra.Metabolite(x, compartment="s") for x in "ab") @@ -140,7 +140,7 @@ def test_essential_merged_away_is_skipped(): """ import cobra - from raven_python.tasks import Task + from raven_toolbox.tasks import Task m = cobra.Model("collapse") a, b = (cobra.Metabolite(x, name=x, compartment="s") for x in "ab") diff --git a/tests/test_init_solvers.py b/tests/test_init_solvers.py index 514c408b..46d59063 100644 --- a/tests/test_init_solvers.py +++ b/tests/test_init_solvers.py @@ -19,8 +19,8 @@ import cobra import pytest -from raven_python.init import ftinit, prep_init_model, run_ftinit, run_init -from raven_python.tasks import Task, check_tasks +from raven_toolbox.init import ftinit, prep_init_model, run_ftinit, run_init +from raven_toolbox.tasks import Task, check_tasks # Detect which MILP-capable optlang interfaces actually work; skip the rest. # We do a real import (not just find_spec) because optlang ships every backend's @@ -66,22 +66,6 @@ def _met(mid, comp="c"): return cobra.Metabolite(mid, name=mid.split("_")[0], compartment=comp) -def _toy_init_model() -> cobra.Model: - """EX_A → A → B → C → D (r1, r2 good; r3 bad). Same network as test_init.py.""" - def rxn(rid, lb, ub, mets): - r = cobra.Reaction(rid, lower_bound=lb, upper_bound=ub) - r.add_metabolites(mets) - return r - m = cobra.Model("toy") - A, B, C, D = (_met(x) for x in ("A_c", "B_c", "C_c", "D_c")) - m.add_metabolites([A, B, C, D]) - m.add_reactions([rxn("EX_A", -1000, 1000, {A: -1}), - rxn("r1", 0, 1000, {A: -1, B: 1}), - rxn("r2", 0, 1000, {B: -1, C: 1}), - rxn("r3", 0, 1000, {C: -1, D: 1})]) - return m - - def _toy_ftinit_model() -> cobra.Model: """Small flux-consistent network for ftINIT: A→B, B→C, parallel A→C (negative-score).""" def rxn(rid, lb, ub, mets): @@ -101,9 +85,9 @@ def rxn(rid, lb, ub, mets): # --------------------------------------------------------------------- tests -def test_run_init_same_verdict(solver): +def test_run_init_same_verdict(solver, linear_chain_model): """tINIT MILP on a small network drops the negative-score reaction with any solver.""" - m = _toy_init_model() + m = linear_chain_model m.solver = solver res = run_init(m, {"r1": 1.0, "r2": 1.0, "r3": -1.0}, prod_weight=0.0, allow_excretion=True) assert "r3" in res.deleted_reactions diff --git a/tests/test_init_taskfill.py b/tests/test_init_taskfill.py index c975f41a..a876382c 100644 --- a/tests/test_init_taskfill.py +++ b/tests/test_init_taskfill.py @@ -6,7 +6,7 @@ """ from tinit_oracles import make_test_model, make_test_task -from raven_python.init import TaskFillResult, fill_tasks +from raven_toolbox.init import TaskFillResult, fill_tasks def _reference_without_exchanges(): @@ -35,7 +35,7 @@ def test_no_fill_when_already_feasible(): def test_should_fail_tasks_ignored(): - from raven_python.tasks import Task + from raven_toolbox.tasks import Task ref = _reference_without_exchanges() gapped = ref.copy() diff --git a/tests/test_io_excel.py b/tests/test_io_excel.py index 12434efe..f7cd11bc 100644 --- a/tests/test_io_excel.py +++ b/tests/test_io_excel.py @@ -1,11 +1,11 @@ -"""Tests for raven_python.io.excel (exportToExcelFormat port, export only).""" +"""Tests for raven_toolbox.io.excel (exportToExcelFormat port, export only).""" import cobra import pytest openpyxl = pytest.importorskip("openpyxl") -from raven_python.io import export_to_excel -from raven_python.manipulation import add_reactions_from_equations +from raven_toolbox.io import export_to_excel +from raven_toolbox.manipulation import add_reactions_from_equations @pytest.fixture @@ -76,7 +76,7 @@ def test_mets_sheet(model, tmp_path): assert atp["ID"] == "ATP[c]" assert atp["NAME"] == "ATP" assert atp["InChI"] == "InChI=1S/X" - assert atp["COMPOSITION"] is None # suppressed when InChI present + assert atp["COMPOSITION"] == "C10H16N5O13P3" # formula kept even when InChI present assert atp["CHARGE"] == -4 assert atp["MIRIAM"] == "kegg.compound/C00002" # smiles excluded diff --git a/tests/test_io_git.py b/tests/test_io_git.py index 28881dcb..8405df2d 100644 --- a/tests/test_io_git.py +++ b/tests/test_io_git.py @@ -1,9 +1,9 @@ -"""Tests for raven_python.io.git (exportForGit port).""" +"""Tests for raven_toolbox.io.git (exportForGit port).""" import cobra import pytest -from raven_python.io import export_for_git -from raven_python.manipulation import add_reactions_from_equations +from raven_toolbox.io import export_for_git +from raven_toolbox.manipulation import add_reactions_from_equations @pytest.fixture @@ -34,7 +34,7 @@ def test_dependencies_file(model, tmp_path): deps = (root / "dependencies.txt").read_text() assert "python\t" in deps assert "cobra\t" in deps - assert "raven_python\t" in deps + assert "raven_toolbox\t" in deps def test_flat_layout(model, tmp_path): diff --git a/tests/test_io_sif.py b/tests/test_io_sif.py index d50ad984..abc41a34 100644 --- a/tests/test_io_sif.py +++ b/tests/test_io_sif.py @@ -1,9 +1,9 @@ -"""Tests for raven_python.io.sif (exportModelToSIF port).""" +"""Tests for raven_toolbox.io.sif (exportModelToSIF port).""" import cobra import pytest -from raven_python.io import export_model_to_sif -from raven_python.manipulation import add_reactions_from_equations +from raven_toolbox.io import export_model_to_sif +from raven_toolbox.manipulation import add_reactions_from_equations @pytest.fixture diff --git a/tests/test_io_yaml.py b/tests/test_io_yaml.py index c406857d..eaaf450e 100644 --- a/tests/test_io_yaml.py +++ b/tests/test_io_yaml.py @@ -1,11 +1,11 @@ -"""Tests for raven_python.io.yaml against the RAVEN fa281a1 (cobra-native !!omap) schema.""" +"""Tests for raven_toolbox.io.yaml against the RAVEN fa281a1 (cobra-native !!omap) schema.""" from pathlib import Path import cobra import pytest from cobra.io.yaml import yaml as cobra_yaml -from raven_python.io import read_yaml_model, write_yaml_model +from raven_toolbox.io import read_yaml_model, write_yaml_model # A model laid out exactly as RAVEN writeYAMLmodel (fa281a1) emits: cobra-native # structure, RAVEN-only fields as top-level per-entry keys, smiles/ec-code inside @@ -165,12 +165,16 @@ def test_gzipped_round_trip(yaml_file, tmp_path): def test_output_is_cobra_readable(yaml_file, tmp_path): - # The written file must load with stock cobra (it's cobra's native format). + """Cobrapy must be able to parse the file (no syntax error) and + recover the metabolites, reactions, and the cobrapy-canonical + annotation block. Model-level id / name / version live inside the + metaData section (RAVEN convention) — cobrapy doesn't know about + metaData, so cobra_model.id is None here. raven_toolbox recovers + them; cobrapy ignores them gracefully.""" model = read_yaml_model(yaml_file) out = tmp_path / "out.yml" write_yaml_model(model, out) cobra_model = cobra.io.load_yaml_model(str(out)) - assert cobra_model.id == "testModel" assert {m.id for m in cobra_model.metabolites} == {"s_0001", "s_0002"} # RAVEN-only fields land in cobra notes; smiles in annotation assert cobra_model.metabolites.get_by_id("s_0001").annotation["smiles"] == ["C1=NC2"] @@ -215,3 +219,42 @@ def test_real_yeast_gem_loads(): assert len(model.reactions) > 1000 # legacy file: identity comes from metaData assert model.id + + +def test_code_built_model_round_trips(tmp_path): + """A model built from cobra objects (not parsed from a doc) survives a + write->read cycle intact — including the objective, which no other I/O + round-trip test asserts is preserved.""" + m = cobra.Model("built") + m.compartments = {"c": "cytoplasm"} + a = cobra.Metabolite("a_c", name="A", compartment="c", formula="C6H12O6", charge=0) + b = cobra.Metabolite("b_c", name="B", compartment="c", formula="C3H4O3", charge=-1) + a.annotation = {"kegg.compound": ["C00031"]} + m.add_metabolites([a, b]) + r = cobra.Reaction("R1", lower_bound=0, upper_bound=1000) + r.add_metabolites({a: -1, b: 2}) + r.gene_reaction_rule = "G1" + r.subsystem = "glycolysis" + bio = cobra.Reaction("BIOMASS", lower_bound=0, upper_bound=1000) + bio.add_metabolites({b: -1}) + m.add_reactions([r, bio]) + m.objective = "BIOMASS" + + out = tmp_path / "built.yml" + write_yaml_model(m, out) + back = read_yaml_model(out) + + assert {x.id for x in back.metabolites} == {"a_c", "b_c"} + assert {x.id for x in back.reactions} == {"R1", "BIOMASS"} + rr = back.reactions.get_by_id("R1") + assert rr.bounds == (0.0, 1000.0) + assert rr.get_coefficient("a_c") == -1 + assert rr.get_coefficient("b_c") == 2 + assert rr.gene_reaction_rule == "G1" + assert rr.subsystem == "glycolysis" + ma = back.metabolites.get_by_id("a_c") + assert ma.formula == "C6H12O6" + assert ma.annotation["kegg.compound"] == ["C00031"] + # The objective survives the round-trip (no other I/O test checks this). + assert {x.id for x in back.reactions if x.objective_coefficient != 0} == {"BIOMASS"} + assert back.reactions.get_by_id("BIOMASS").objective_coefficient == 1.0 diff --git a/tests/test_io_yaml_ec_data.py b/tests/test_io_yaml_ec_data.py index 94c2ebbf..f236b851 100644 --- a/tests/test_io_yaml_ec_data.py +++ b/tests/test_io_yaml_ec_data.py @@ -1,4 +1,4 @@ -"""Tests for raven_python.io.yaml's GECKO ec-model support. +"""Tests for raven_toolbox.io.yaml's GECKO ec-model support. Covers: - model.ec populated from `ec-rxns` / `ec-enzymes` / `gecko_light` sections; @@ -20,8 +20,9 @@ from cobra.io.yaml import yaml as cobra_yaml from scipy import sparse -from raven_python.io import EcData, read_yaml_model, write_yaml_model -from raven_python.io.yaml import model_from_yaml_data +from raven_toolbox.io import EcData, read_yaml_model, write_yaml_model +from raven_toolbox.io.ec_data import _eccodes_to_yaml +from raven_toolbox.io.yaml import model_from_yaml_data # --------------------------------------------------------------------------- # # Helpers @@ -133,6 +134,16 @@ def test_load_eccodes_scalar_or_list_both_round_trip(tmp_path): assert m2.ec.eccodes == ["1.1.1.1;1.1.99.40"] +def test_eccodes_to_yaml_strips_trailing_separator(): + # The write-path helper must clean the internal `;`-joined form: a single EC + # round-trips to a bare scalar, and stray separators never leak into the YAML. + assert _eccodes_to_yaml("1.1.1.1") == "1.1.1.1" + assert _eccodes_to_yaml("1.1.1.1;") == "1.1.1.1" + assert _eccodes_to_yaml(";") == "" + assert _eccodes_to_yaml("") == "" + assert _eccodes_to_yaml("1.1.1.1;1.1.99.40") == ["1.1.1.1", "1.1.99.40"] + + # --------------------------------------------------------------------------- # # Save: model.ec serialised back to top-level sections # --------------------------------------------------------------------------- # @@ -349,6 +360,16 @@ def test_load_dangling_enzyme_reference_raises(tmp_path): read_yaml_model(_write_yaml(doc, tmp_path / "m.yml")) +def test_load_all_zero_coupling_matrix_warns(tmp_path): + """ec-rxns present but none reference an enzyme -> the coupling matrix is + all-zero (likely malformed). Warn rather than build a silently empty model.""" + doc = _minimal_model_doc() + doc["ec-rxns"] = [{"id": "R1", "kcat": 1.0, "enzymes": {}}] + doc["ec-enzymes"] = [{"genes": "G1", "enzymes": "P1", "mw": 1.0}] + with pytest.warns(UserWarning, match="all-zero"): + read_yaml_model(_write_yaml(doc, tmp_path / "m.yml")) + + # --------------------------------------------------------------------------- # # In-memory model_from_yaml_data (no file I/O) # --------------------------------------------------------------------------- # diff --git a/tests/test_io_yaml_parity.py b/tests/test_io_yaml_parity.py new file mode 100644 index 00000000..6fff479d --- /dev/null +++ b/tests/test_io_yaml_parity.py @@ -0,0 +1,364 @@ +"""Parity gate: round-tripping a YAML model through raven_toolbox.io.yaml +must produce a file that: + + * cobra.io.load_yaml_model can read (the cobrapy-canonical core); + * keeps every RAVEN-only field (inchis / deltaG / rxnFrom / + metFrom / references / confidence_score / rxnNotes / protein / + metMiriams / rxnMiriams / annotation-side SMILES and EC codes); + * emits ``!!omap`` tags on each per-entry mapping (so RAVEN MATLAB's + line-based reader can ingest it); + * places the ``metaData`` block first, matching RAVEN MATLAB's layout. + +The fixture below is the smallest model that exercises every RAVEN +extension, plus a legacy ``rxnNotes`` key (read-time alias the writer +must normalise to ``notes``) and a metabolite with a SMILES value +that would parse as a flow sequence if emitted unquoted. +""" +from __future__ import annotations + +from pathlib import Path + +import cobra +import cobra.io +import pytest +from cobra.io.yaml import yaml as cobra_yaml + +from raven_toolbox.io import read_yaml_model, write_yaml_model + +SAMPLE = { + "metabolites": [ + { + "id": "s_0001", + "name": "ATP", + "compartment": "c", + "charge": -4, + "formula": "C10H16N5O13P3", + "inchis": "InChI=1S/CH4", + "deltaG": 12.5, + "metFrom": "KEGG", + "notes": "metabolite note", + "annotation": { + "kegg.compound": ["C00002"], + "smiles": ["C1=NC2=C(N=CN2)N(C1=O)C"], # YAML-ambiguous chars + }, + }, + {"id": "s_0002", "name": "ADP", "compartment": "c"}, + ], + "reactions": [ + { + "id": "R1", + "name": "rxn one", + "metabolites": {"s_0001": -1.0, "s_0002": 1.0}, + "lower_bound": -1000.0, + "upper_bound": 1000.0, + "gene_reaction_rule": "G1", + "objective_coefficient": 0, + "subsystem": "glycolysis", + "confidence_score": 2, + "references": "PMID:123", + "rxnFrom": "manual", + "eccodes": "1.1.1.1", + "rxnNotes": "legacy reaction note key", # read-time alias + "deltaG": -5.0, + "annotation": {"ec-code": ["1.1.1.1"]}, + } + ], + "genes": [ + {"id": "G1", "name": "geneOne", "protein": "P12345", + "annotation": {"uniprot": ["P12345"]}} + ], + "compartments": {"c": "cytoplasm"}, + "metaData": { + "id": "testModel", + "name": "Test", + "version": "1.0", + "date": "2026-05-23", + "taxonomy": "taxonomy/559292", + }, +} + + +@pytest.fixture +def src(tmp_path) -> Path: + p = tmp_path / "source.yml" + with p.open("w", encoding="utf-8") as fh: + cobra_yaml.dump(SAMPLE, fh) + return p + + +def test_round_trip_preserves_every_raven_field(src, tmp_path): + model = read_yaml_model(src) + out = tmp_path / "out.yml" + write_yaml_model(model, out) + reloaded = read_yaml_model(out) + + # Core (cobra-known) content stayed. + assert reloaded.id == "testModel" + assert {m.id for m in reloaded.metabolites} == {"s_0001", "s_0002"} + r = reloaded.reactions.get_by_id("R1") + assert r.bounds == (-1000.0, 1000.0) + assert r.gene_reaction_rule == "G1" + assert r.subsystem == "glycolysis" + + # Metabolite RAVEN extras. + a = reloaded.metabolites.get_by_id("s_0001") + assert a.notes["inchis"] == "InChI=1S/CH4" + assert a.notes["deltaG"] == 12.5 + assert a.notes["metFrom"] == "KEGG" + assert a.notes["note"] == "metabolite note" + assert a.annotation["smiles"] == ["C1=NC2=C(N=CN2)N(C1=O)C"] + + # EC codes round-trip through cobra annotation (the cobra-native place, + # where geckopy reads them), not a RAVEN-only top-level/notes field. + assert r.annotation["ec-code"] == ["1.1.1.1"] + assert "eccodes" not in r.notes + assert r.notes["references"] == "PMID:123" + assert r.notes["rxnFrom"] == "manual" + assert r.notes["confidence_score"] == 2 + assert r.notes["deltaG"] == -5.0 + # rxnNotes (legacy key) gets normalised to notes on read. + assert r.notes["note"] == "legacy reaction note key" + + # Gene RAVEN extras. + assert reloaded.genes.get_by_id("G1").notes["protein"] == "P12345" + + # Provenance. + assert reloaded.notes["metaData"]["taxonomy"] == "taxonomy/559292" + assert reloaded.notes["version"] == "1.0" + + +def test_output_is_cobra_readable(src, tmp_path): + """The written file is valid cobra-native YAML; cobra.io can read + the core content (it doesn't know about RAVEN extras, but doesn't + choke on them either).""" + model = read_yaml_model(src) + out = tmp_path / "out.yml" + write_yaml_model(model, out) + cmodel = cobra.io.load_yaml_model(str(out)) + assert {m.id for m in cmodel.metabolites} == {"s_0001", "s_0002"} + assert cmodel.reactions.get_by_id("R1").bounds == (-1000.0, 1000.0) + # SMILES landed in annotation, not at metabolite top level. + assert cmodel.metabolites.get_by_id("s_0001").annotation["smiles"] == [ + "C1=NC2=C(N=CN2)N(C1=O)C" + ] + + +def test_output_carries_omap_tags(src, tmp_path): + """RAVEN MATLAB's reader is a line-based parser keyed on ``!!omap``; + the writer must emit those tags. (PR #17 originally dropped them + because _to_plain flattened OrderedDicts to plain dicts.)""" + model = read_yaml_model(src) + out = tmp_path / "out.yml" + write_yaml_model(model, out) + text = out.read_text() + # Per-entry !!omap on metabolites, reactions, genes: + assert text.count("!!omap") >= 1 + 1 + 2 + 1 + 1 # root + metaData + 2 mets + 1 rxn + 1 gene + # Each major section appears as a top-level entry. + for section in ("metabolites:", "reactions:", "genes:", "compartments:"): + assert f"- {section}" in text + + +def test_metadata_is_first(src, tmp_path): + """RAVEN MATLAB emits metaData as the first top-level section. + Producing the same layout means RAVEN MATLAB and raven_toolbox + files agree byte-for-byte on top-level ordering.""" + model = read_yaml_model(src) + out = tmp_path / "out.yml" + write_yaml_model(model, out) + text = out.read_text() + idx_meta = text.find("- metaData:") + idx_mets = text.find("- metabolites:") + assert 0 <= idx_meta < idx_mets, "metaData must precede metabolites" + + +def test_smiles_with_yaml_special_chars_quoted(src, tmp_path): + """The SMILES value above contains square brackets; an unquoted + bare scalar would be parsed as a flow sequence. The writer must + either keep ``smiles`` inside the annotation block (where SMILES + annotations naturally live) or quote it. Either way the loop- + back read must recover the exact string.""" + model = read_yaml_model(src) + out = tmp_path / "out.yml" + write_yaml_model(model, out) + # The verification is purely functional: reload, check the value. + reloaded = read_yaml_model(out) + assert reloaded.metabolites.get_by_id("s_0001").annotation["smiles"] == [ + "C1=NC2=C(N=CN2)N(C1=O)C" + ] + + +PRE_SHIM_YAML = """\ +--- +!!omap +- metaData: + id: "eciYali" + name: "Yarrowia lipolytica" + version: "1.0" + date: "2024-10-17" + geckoLight: "true" +- metabolites: + - !!omap + - id: "s_0001" + - name: "ATP" + - compartment: "c" + - formula: "C10H16N5O13P3" + - charge: -4 + - inchis: "InChI=1S/CH4" + - smiles: "[O-]P(=O)([O-])OP(=O)([O-])O" + - annotation: !!omap + - kegg.compound: "C00002" + - sbo: "SBO:0000247" + - deltaG: 12.5 + - notes: "metabolite note" + - metFrom: "KEGG" +- reactions: + - !!omap + - id: "r_0001" + - name: "hexokinase" + - metabolites: !!omap + - s_0001: -1 + - lower_bound: -1000 + - upper_bound: 1000 + - gene_reaction_rule: "G1" + - rxnFrom: "KEGG" + - eccodes: "2.7.1.1" + - references: "PMID:12345" + - subsystem: "Glycolysis" + - annotation: !!omap + - kegg.reaction: "R00299" + - sbo: "SBO:0000176" + - deltaG: -17.39 + - confidence_score: 2 + - rxnNotes: "old reaction note" +- genes: + - !!omap + - id: "G1" + - name: "HXK1" + - protein: "P01234" + - annotation: !!omap + - uniprot: "P01234" +- compartments: !!omap + - c: "cytoplasm" +- ec-rxns: + - !!omap + - id: "r_0001" + - kcat: 25.3 + - enzymes: !!omap + - P01234: 1 +- ec-enzymes: + - !!omap + - genes: "G1" + - enzymes: "P01234" + - mw: 50000 +""" + + +def test_pre_shim_format_loads(tmp_path): + """The pre-`feat/yeast-gem-shared` RAVEN MATLAB writer emitted a + file shape that differs from the current one in seven concrete + ways. The reader must continue to load every one of them: + + 1. ``---`` document-start marker (kept by old MATLAB writer) + 2. ``- metaData:`` as a plain block mapping (no ``!!omap`` tag) + 3. ``geckoLight: "true"`` *inside* metaData (now emitted as a + top-level ``gecko_light``) + 4. Metabolite ``smiles`` as a top-level entry key (now emitted + inside the ``annotation`` block) + 5. Reaction notes under the ``rxnNotes`` key (now emitted as + ``notes``) + 6. Integer-typed bounds / coefficients (now emitted as floats) + 7. Every string double-quoted (now bare unless YAML requires + quoting) + + Each item below maps to one of those seven cases. + """ + p = tmp_path / "pre_shim.yml" + p.write_text(PRE_SHIM_YAML) + model = read_yaml_model(p) + + # metaData survives + provenance is lifted onto the cobra-shape + # accessors (cases 1 + 2). + assert model.id == "eciYali" + assert model.name == "Yarrowia lipolytica" + assert model.notes["version"] == "1.0" + assert model.notes["metaData"]["taxonomy" if "taxonomy" in model.notes["metaData"] else "date"] + + # geckoLight in metaData populates the typed EcData (case 3). + assert model.ec is not None + assert model.ec.gecko_light is True + assert model.ec.rxns == ["r_0001"] + assert model.ec.kcat[0] == 25.3 + + # Top-level smiles lifted into annotation.smiles (case 4). + a = model.metabolites.get_by_id("s_0001") + assert a.annotation["smiles"] == ["[O-]P(=O)([O-])OP(=O)([O-])O"] + assert "smiles" not in a.notes # stays in annotation, not notes + + # rxnNotes read as the canonical notes key (case 5). + r = model.reactions.get_by_id("r_0001") + assert r.notes["note"] == "old reaction note" + + # Integer bounds become floats inside cobra (case 6). + assert r.bounds == (-1000.0, 1000.0) + assert isinstance(r.lower_bound, float) + + # Quoted strings unquote cleanly (case 7) — verified implicitly by + # all the equality assertions above. Spot check the metabolite + # name, which used double quotes in the source. + assert a.name == "ATP" + + # Other RAVEN extras still preserved. + assert a.notes["inchis"] == "InChI=1S/CH4" + assert a.notes["deltaG"] == 12.5 + assert a.notes["note"] == "metabolite note" + assert a.notes["metFrom"] == "KEGG" + assert r.notes["rxnFrom"] == "KEGG" + # legacy top-level eccodes lifted into the cobra-native annotation['ec-code'] + assert r.annotation["ec-code"] == ["2.7.1.1"] + assert "eccodes" not in r.notes + assert r.notes["references"] == "PMID:12345" + assert r.notes["confidence_score"] == 2 + assert r.notes["deltaG"] == -17.39 + assert model.genes.get_by_id("G1").notes["protein"] == "P01234" + + +def test_pre_shim_yeast_gem_loads_if_available(): + """The real pre-shim yeast-GEM.yml: 2748 mets, 4102 rxns, 1143 + genes. Skipped when the working copy isn't mounted (CI runners).""" + real = Path("/mnt/c/Work/GitHub/yeast-GEM/model/yeast-GEM.yml") + if not real.exists(): + pytest.skip("yeast-GEM.yml not available in this environment") + model = read_yaml_model(real) + assert model.id == "yeastGEM_develop" + assert len(model.metabolites) == 2748 + assert len(model.reactions) == 4102 + assert len(model.genes) == 1143 + # Every RAVEN extension we know about must come through. + assert sum(1 for r in model.reactions if r.annotation.get("ec-code")) == 2411 + assert sum(1 for r in model.reactions if r.notes.get("deltaG") is not None) == 3984 + assert sum(1 for m in model.metabolites if m.notes.get("deltaG") is not None) == 2696 + assert sum(1 for m in model.metabolites if "smiles" in (m.annotation or {})) == 1788 + assert sum(1 for r in model.reactions if r.notes.get("note")) == 1443 + + +def test_eccodes_round_trip_through_cobra_extras(src, tmp_path): + """EC codes round-trip as cobra annotation through a + raven_toolbox -> cobra -> raven_toolbox loop. They live in + ``annotation['ec-code']`` — the cobra-native place — so plain + ``cobra.io`` preserves them with no RAVEN-specific handling, and + geckopy (which reads ``annotation['ec-code']``) sees them.""" + model = read_yaml_model(src) + pass1 = tmp_path / "via_rp.yml" + write_yaml_model(model, pass1) + # Plain cobra reads annotation['ec-code'] natively — this is the + # interop the alignment guarantees. + via_cobra = cobra.io.load_yaml_model(str(pass1)) + assert via_cobra.reactions.get_by_id("R1").annotation["ec-code"] == ["1.1.1.1"] + pass2 = tmp_path / "via_rp2.yml" + again = read_yaml_model(pass1) + write_yaml_model(again, pass2) + final = read_yaml_model(pass2) + assert final.reactions.get_by_id("R1").annotation["ec-code"] == ["1.1.1.1"] + # And cobra can still read the final result. + cm = cobra.io.load_yaml_model(str(pass2)) + assert cm.reactions.get_by_id("R1").bounds == (-1000.0, 1000.0) diff --git a/tests/test_localization.py b/tests/test_localization.py index d177db5c..a7cdafc0 100644 --- a/tests/test_localization.py +++ b/tests/test_localization.py @@ -1,4 +1,4 @@ -"""Tests for raven_python.localization — predictor loaders + the MILP + apply (Phase 7).""" +"""Tests for raven_toolbox.localization — predictor loaders + the MILP + apply (Phase 7).""" from __future__ import annotations from textwrap import dedent @@ -7,7 +7,7 @@ import pandas as pd import pytest -from raven_python.localization import ( +from raven_toolbox.localization import ( LocalizationProposal, LocalizationResult, LocalizationScores, diff --git a/tests/test_manifest.py b/tests/test_manifest.py new file mode 100644 index 00000000..160c53ef --- /dev/null +++ b/tests/test_manifest.py @@ -0,0 +1,125 @@ +"""Tests for the data/binary manifest loader (manifest.py) and its wiring into the +resolvers. Uses file:// URLs + a tmp manifest to avoid the network.""" +import hashlib +import json +from pathlib import Path + +import pytest + +from raven_toolbox import binaries, data, manifest + +REPO_ROOT = Path(__file__).resolve().parents[1] + + +def _sha256(b: bytes) -> str: + return hashlib.sha256(b).hexdigest() + + +@pytest.fixture +def clean_registries(): + """Snapshot and restore the live registries so a test's loads don't leak.""" + data_snap = dict(data._DATA_REGISTRY) + bin_snap = dict(binaries._REGISTRY) + data._DATA_REGISTRY.clear() + binaries._REGISTRY.clear() + yield + data._DATA_REGISTRY.clear() + data._DATA_REGISTRY.update(data_snap) + binaries._REGISTRY.clear() + binaries._REGISTRY.update(bin_snap) + + +def _write_manifest(tmp_path: Path, payload: dict) -> Path: + p = tmp_path / "manifest.json" + p.write_text(json.dumps(payload), encoding="utf-8") + return p + + +def test_converters_strip_to_registry_shapes(): + m = { + "manifest_version": 1, + "data": { + "kegg": { + "version": "kegg116", + "license": "metadata that the registry should drop", + "files": {"x.gz": {"url": "https://e/x.gz", "sha256": "a" * 64, "bytes": 5}}, + } + }, + "binaries": { + "diamond": { + "version": "2.1.9", + "provides": ["diamond"], + "license": "GPL-3.0-only", + "platforms": {"linux-x86_64": {"url": "https://e/d.zip", "sha256": "b" * 64, "bytes": 9}}, + } + }, + } + assert manifest.to_data_registry(m) == { + "kegg": {"version": "kegg116", "files": {"x.gz": {"url": "https://e/x.gz", "sha256": "a" * 64}}} + } + assert manifest.to_binary_registry(m) == { + "diamond": { + "version": "2.1.9", + "provides": ["diamond"], + "platforms": {"linux-x86_64": {"url": "https://e/d.zip", "sha256": "b" * 64}}, + } + } + + +def test_load_manifest_rejects_unknown_version(tmp_path): + p = _write_manifest(tmp_path, {"manifest_version": 999}) + with pytest.raises(ValueError, match="manifest_version"): + manifest.load_manifest(p) + + +def test_load_manifest_requires_a_source(monkeypatch): + monkeypatch.delenv(manifest.ENV_MANIFEST, raising=False) + with pytest.raises(ValueError, match="No manifest source"): + manifest.load_manifest() + + +def test_load_into_registries_populates_both(tmp_path, clean_registries): + p = _write_manifest( + tmp_path, + { + "manifest_version": 1, + "data": {"kegg": {"version": "v1", "files": {"a": {"url": "https://e/a", "sha256": "0" * 64}}}}, + "binaries": {"diamond": {"version": "2", "provides": ["diamond"], "platforms": {}}}, + }, + ) + manifest.load_into_registries(p) + assert data._DATA_REGISTRY["kegg"]["version"] == "v1" + assert binaries._REGISTRY["diamond"]["provides"] == ["diamond"] + + +def test_resolver_lazy_autoload_via_env(tmp_path, monkeypatch, clean_registries): + # A real artefact file served over file://, registered through the manifest. + artefact = tmp_path / "reference_model.yml.gz" + artefact.write_bytes(b"hello kegg") + payload = { + "manifest_version": 1, + "data": { + "kegg": { + "version": "kegg-test", + "files": { + artefact.name: {"url": artefact.as_uri(), "sha256": _sha256(artefact.read_bytes())} + }, + } + }, + } + manifest_path = _write_manifest(tmp_path, payload) + monkeypatch.setenv(manifest.ENV_MANIFEST, str(manifest_path)) + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache")) + + # _DATA_REGISTRY is empty; ensure_data_file must lazily load the manifest and fetch. + got = data.ensure_data_file("kegg", artefact.name) + assert got.read_bytes() == b"hello kegg" + assert data._DATA_REGISTRY["kegg"]["version"] == "kegg-test" + + +@pytest.mark.parametrize("name", ["manifest.json", "manifest.example.json"]) +def test_repo_manifests_are_valid(name): + m = manifest.load_manifest(REPO_ROOT / "data" / name) + # Both must convert cleanly to the runtime registry shapes. + manifest.to_data_registry(m) + manifest.to_binary_registry(m) diff --git a/tests/test_manipulation_add.py b/tests/test_manipulation_add.py index 2a3a9d3b..fbf87b3e 100644 --- a/tests/test_manipulation_add.py +++ b/tests/test_manipulation_add.py @@ -1,9 +1,9 @@ -"""Tests for raven_python.manipulation.add (addRxns port).""" +"""Tests for raven_toolbox.manipulation.add (addRxns port).""" import cobra import pytest -from raven_python.manipulation import add_reactions_from_equations -from raven_python.utils.parse import parse_name_comp +from raven_toolbox.manipulation import add_reactions_from_equations +from raven_toolbox.utils.parse import parse_name_comp @pytest.fixture @@ -114,6 +114,25 @@ def test_name_mode_creates_new_met_with_auto_id(model): assert new[0].compartment == "c" +def test_name_mode_dedups_new_met_across_reactions(model): + # A new metabolite named on more than one reaction in the same call must be + # created once and shared — later tokens have to see mets created earlier in + # the call (the (name, comp) index is seeded once and updated on creation). + r1, r2 = add_reactions_from_equations( + model, + [ + {"id": "R1", "equation": "ATP --> AMP"}, + {"id": "R2", "equation": "AMP --> ADP"}, + ], + mets_by="name", + compartment="c", + ) + amp = [m for m in model.metabolites if m.name == "AMP"] + assert len(amp) == 1 # created once, not duplicated + assert amp[0] in r1.metabolites + assert amp[0] in r2.metabolites + + def test_name_mode_requires_compartment(model): with pytest.raises(ValueError, match="needs a compartment"): add_reactions_from_equations( diff --git a/tests/test_manipulation_change.py b/tests/test_manipulation_change.py index 8d54f587..dc11b01a 100644 --- a/tests/test_manipulation_change.py +++ b/tests/test_manipulation_change.py @@ -1,8 +1,8 @@ -"""Tests for raven_python.manipulation.change (changeRxns port).""" +"""Tests for raven_toolbox.manipulation.change (changeRxns port).""" import cobra import pytest -from raven_python.manipulation import add_reactions_from_equations, change_reaction_equations +from raven_toolbox.manipulation import add_reactions_from_equations, change_reaction_equations @pytest.fixture diff --git a/tests/test_manipulation_compartments.py b/tests/test_manipulation_compartments.py index 4d3fb3b0..8444222c 100644 --- a/tests/test_manipulation_compartments.py +++ b/tests/test_manipulation_compartments.py @@ -4,7 +4,7 @@ import cobra import pytest -from raven_python.manipulation.compartments import copy_to_compartment, merge_compartments +from raven_toolbox.manipulation.compartments import copy_to_compartment, merge_compartments def _two_compartment_model() -> cobra.Model: diff --git a/tests/test_manipulation_expand.py b/tests/test_manipulation_expand.py index 08cd2f2f..1e8b6f1a 100644 --- a/tests/test_manipulation_expand.py +++ b/tests/test_manipulation_expand.py @@ -4,8 +4,8 @@ """ import cobra -from raven_python.manipulation import expand_model -from raven_python.manipulation.expand import _gpr_to_dnf +from raven_toolbox.manipulation import expand_model +from raven_toolbox.manipulation.expand import gpr_to_dnf # --------------------------------------------------------------------------- # # DNF conversion (internal helper, worth testing directly) @@ -15,7 +15,7 @@ def _dnf_from_gpr_string(gpr_str: str) -> list[list[str]]: from cobra.core.gene import GPR gpr = GPR.from_string(gpr_str) - return _gpr_to_dnf(gpr) + return gpr_to_dnf(gpr) def test_dnf_empty_gpr(): diff --git a/tests/test_manipulation_find_duplicate.py b/tests/test_manipulation_find_duplicate.py new file mode 100644 index 00000000..94c978f3 --- /dev/null +++ b/tests/test_manipulation_find_duplicate.py @@ -0,0 +1,85 @@ +"""Tests for raven_toolbox.manipulation.find_duplicate_reactions.""" +from __future__ import annotations + +import cobra + +from raven_toolbox.manipulation import find_duplicate_reactions + + +def _mk_model() -> cobra.Model: + m = cobra.Model("m") + a = cobra.Metabolite("a_c", compartment="c") + b = cobra.Metabolite("b_c", compartment="c") + c = cobra.Metabolite("c_c", compartment="c") + m.add_metabolites([a, b, c]) + return m + + +def test_no_duplicates_returns_empty(): + m = _mk_model() + r1 = cobra.Reaction("r1") + r1.add_metabolites({m.metabolites.a_c: -1, m.metabolites.b_c: 1}) + r2 = cobra.Reaction("r2") + r2.add_metabolites({m.metabolites.b_c: -1, m.metabolites.c_c: 1}) + m.add_reactions([r1, r2]) + assert find_duplicate_reactions(m) == [] + + +def test_same_stoichiometry_grouped(): + m = _mk_model() + r1 = cobra.Reaction("r1", lower_bound=0, upper_bound=1000) + r2 = cobra.Reaction("r2", lower_bound=-100, upper_bound=100) + for r in (r1, r2): + r.add_metabolites({m.metabolites.a_c: -1, m.metabolites.b_c: 1}) + m.add_reactions([r1, r2]) + groups = find_duplicate_reactions(m) + assert len(groups) == 1 + assert {r.id for r in groups[0]} == {"r1", "r2"} + + +def test_ignore_direction_default_groups_reverse_pair(): + """yeast-GEM's findDuplicatedRxns matches A→B with B→A. That's the default.""" + m = _mk_model() + r1 = cobra.Reaction("r1") + r1.add_metabolites({m.metabolites.a_c: -1, m.metabolites.b_c: 1}) + r2 = cobra.Reaction("r2") + r2.add_metabolites({m.metabolites.a_c: 1, m.metabolites.b_c: -1}) # reversed + m.add_reactions([r1, r2]) + groups = find_duplicate_reactions(m) + assert len(groups) == 1 + + +def test_ignore_direction_false_keeps_them_separate(): + m = _mk_model() + r1 = cobra.Reaction("r1") + r1.add_metabolites({m.metabolites.a_c: -1, m.metabolites.b_c: 1}) + r2 = cobra.Reaction("r2") + r2.add_metabolites({m.metabolites.a_c: 1, m.metabolites.b_c: -1}) + m.add_reactions([r1, r2]) + assert find_duplicate_reactions(m, ignore_direction=False) == [] + + +def test_three_duplicates_in_one_group(): + m = _mk_model() + rxns = [] + for i in range(3): + r = cobra.Reaction(f"r{i}") + r.add_metabolites({m.metabolites.a_c: -1, m.metabolites.b_c: 1}) + rxns.append(r) + m.add_reactions(rxns) + groups = find_duplicate_reactions(m) + assert len(groups) == 1 + assert len(groups[0]) == 3 + + +def test_ignores_bounds_and_gpr_differences(): + """Bounds and GPRs are intentionally ignored — only stoichiometry.""" + m = _mk_model() + r1 = cobra.Reaction("r1", lower_bound=0, upper_bound=1000) + r2 = cobra.Reaction("r2", lower_bound=-50, upper_bound=50) + r1.add_metabolites({m.metabolites.a_c: -1, m.metabolites.b_c: 1}) + r2.add_metabolites({m.metabolites.a_c: -1, m.metabolites.b_c: 1}) + r1.gene_reaction_rule = "g1" + r2.gene_reaction_rule = "g2" + m.add_reactions([r1, r2]) + assert len(find_duplicate_reactions(m)) == 1 diff --git a/tests/test_manipulation_irreversible.py b/tests/test_manipulation_irreversible.py index e211fa31..ad4cd05d 100644 --- a/tests/test_manipulation_irreversible.py +++ b/tests/test_manipulation_irreversible.py @@ -5,7 +5,7 @@ """ import cobra -from raven_python.manipulation import convert_to_irreversible +from raven_toolbox.manipulation import convert_to_irreversible def _build_model_with_bounds( diff --git a/tests/test_manipulation_merge.py b/tests/test_manipulation_merge.py index a430f6eb..ab301c31 100644 --- a/tests/test_manipulation_merge.py +++ b/tests/test_manipulation_merge.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.manipulation import add_reactions_from_equations, merge_models +from raven_toolbox.manipulation import add_reactions_from_equations, merge_models def _model(mid, mets, reactions): diff --git a/tests/test_manipulation_remove.py b/tests/test_manipulation_remove.py index 2b659b9c..3bfe0bef 100644 --- a/tests/test_manipulation_remove.py +++ b/tests/test_manipulation_remove.py @@ -1,8 +1,8 @@ -"""Tests for raven_python.manipulation.remove (removeMets/removeGenes ports).""" +"""Tests for raven_toolbox.manipulation.remove (removeMets/removeGenes ports).""" import cobra import pytest -from raven_python.manipulation import ( +from raven_toolbox.manipulation import ( add_reactions_from_equations, remove_genes, remove_metabolites, diff --git a/tests/test_manipulation_simplify.py b/tests/test_manipulation_simplify.py index 586a0c39..a9ae8eb3 100644 --- a/tests/test_manipulation_simplify.py +++ b/tests/test_manipulation_simplify.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.manipulation import ( +from raven_toolbox.manipulation import ( add_reactions_from_equations, constrain_reversible_reactions, group_linear_reactions, diff --git a/tests/test_manipulation_transfer.py b/tests/test_manipulation_transfer.py index 61c2ac90..78b1de5d 100644 --- a/tests/test_manipulation_transfer.py +++ b/tests/test_manipulation_transfer.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.manipulation import add_reactions_from_equations, add_reactions_from_model +from raven_toolbox.manipulation import add_reactions_from_equations, add_reactions_from_model @pytest.fixture diff --git a/tests/test_manipulation_transport.py b/tests/test_manipulation_transport.py index e8fb2b68..4cc09a9b 100644 --- a/tests/test_manipulation_transport.py +++ b/tests/test_manipulation_transport.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.manipulation import add_transport_reactions +from raven_toolbox.manipulation import add_transport_reactions @pytest.fixture diff --git a/tests/test_omics_hpa.py b/tests/test_omics_hpa.py index 8b2c124f..fbc2da1a 100644 --- a/tests/test_omics_hpa.py +++ b/tests/test_omics_hpa.py @@ -5,7 +5,7 @@ import pytest -from raven_python.omics import ( +from raven_toolbox.omics import ( HPA_LEVEL_SCORES, HPAData, HPARnaData, diff --git a/tests/test_parameters.py b/tests/test_parameters.py index c0ab06cd..f681833a 100644 --- a/tests/test_parameters.py +++ b/tests/test_parameters.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.manipulation import add_reactions_from_equations, set_variance_bounds +from raven_toolbox.manipulation import add_reactions_from_equations, set_variance_bounds @pytest.fixture diff --git a/tests/test_reconstruction_blast.py b/tests/test_reconstruction_blast.py index 32af556b..184150d8 100644 --- a/tests/test_reconstruction_blast.py +++ b/tests/test_reconstruction_blast.py @@ -4,8 +4,8 @@ import pandas as pd import pytest -from raven_python.reconstruction.homology import HIT_COLUMNS, blast_from_table, run_blast -from raven_python.reconstruction.homology.blast import _parse_tabular +from raven_toolbox.reconstruction.homology import HIT_COLUMNS, blast_from_table, run_blast +from raven_toolbox.reconstruction.homology.blast import _parse_tabular _SEQ = ( "MSTNPKPQRKTKRNTNRRPQDVKFPGGGQIVGGVYLLPRRGPRLGVRATRKTSERSQPRGRRQPIPKARRPEGRTWAQPGYPWPLYGNEGCGWAGWLLSPRG" diff --git a/tests/test_reconstruction_homology.py b/tests/test_reconstruction_homology.py index 63ed72f1..10659a90 100644 --- a/tests/test_reconstruction_homology.py +++ b/tests/test_reconstruction_homology.py @@ -3,8 +3,8 @@ import pandas as pd import pytest -from raven_python.manipulation import add_reactions_from_equations -from raven_python.reconstruction.homology import ( +from raven_toolbox.manipulation import add_reactions_from_equations +from raven_toolbox.reconstruction.homology import ( HIT_COLUMNS, get_model_from_homology, make_ortholog_hits, diff --git a/tests/test_reconstruction_kegg_download.py b/tests/test_reconstruction_kegg_download.py index 38d2f445..46c4bfeb 100644 --- a/tests/test_reconstruction_kegg_download.py +++ b/tests/test_reconstruction_kegg_download.py @@ -11,7 +11,7 @@ import pytest -from raven_python.reconstruction.kegg.download import ( +from raven_toolbox.reconstruction.kegg.download import ( _resolve_auth, extract_kegg_dump, ) @@ -57,6 +57,14 @@ def test_resolve_auth_host_absent(tmp_path): _resolve_auth("ftp.kegg.net", netrc_path=netrc_file) +def test_resolve_auth_malformed_netrc(tmp_path): + netrc_file = tmp_path / ".netrc" + netrc_file.write_text("this is not a valid netrc line\n") + netrc_file.chmod(0o600) + with pytest.raises(ValueError, match="Could not read credentials"): + _resolve_auth("ftp.kegg.net", netrc_path=netrc_file) + + # --------------------------------------------------------------------------- # # Extract / arrange # --------------------------------------------------------------------------- # diff --git a/tests/test_reconstruction_kegg_hmm.py b/tests/test_reconstruction_kegg_hmm.py index 3f559eca..91b4890f 100644 --- a/tests/test_reconstruction_kegg_hmm.py +++ b/tests/test_reconstruction_kegg_hmm.py @@ -1,17 +1,21 @@ -"""Tests for KEGG HMM-library construction (taxonomy + hmm, step 3b.3).""" +"""Tests for KEGG HMM-library construction (taxonomy + hmm, step 3b.3). + +The ``kegg_dump`` fixture (tests/conftest.py) is a small, fully fictional dump — +no real KEGG content is committed. +""" from pathlib import Path import pandas as pd import pytest -from raven_python.reconstruction.kegg import ( +from raven_toolbox.reconstruction.kegg import ( build_ko_fastas, organism_domains, organisms_in_domain, parse_taxonomy, ) -from raven_python.reconstruction.kegg import hmm as hmm_mod -from raven_python.reconstruction.kegg.hmm import ( +from raven_toolbox.reconstruction.kegg import hmm as hmm_mod +from raven_toolbox.reconstruction.kegg.hmm import ( _cdhit_cmd, _cdhit_word_size, _fasta_stats, @@ -20,18 +24,16 @@ build_ko_hmm, ) -DUMP = Path(__file__).parent / "data" / "kegg_dump" - @pytest.fixture def organism_gene_ko(): return pd.DataFrame( [ - ("bsu", "BSU31050", "K01194"), - ("bsu", "BSU31060", "K01194"), - ("hsa", "124", "K01194"), - ("hsa", "125", "K01194"), - ("eco", "b0001", "K00002"), + ("aaa", "GENE01", "K90001"), + ("aaa", "GENE02", "K90001"), + ("ccc", "GENE04", "K90001"), + ("ccc", "GENE05", "K90001"), + ("bbb", "GENE03", "K90002"), ], columns=["organism", "gene", "ko"], ) @@ -40,24 +42,24 @@ def organism_gene_ko(): # --------------------------------------------------------------------------- # # Taxonomy # --------------------------------------------------------------------------- # -def test_parse_taxonomy_lineages(): - cats = parse_taxonomy(DUMP / "taxonomy") - assert cats["bsu"] == ["Prokaryotes", "Bacteria", "Firmicutes"] - assert cats["hsa"][0] == "Eukaryotes" - assert cats["eco"][1] == "Bacteria" - - -def test_organism_domains(): - assert organism_domains(DUMP / "taxonomy") == { - "bsu": "Prokaryotes", - "eco": "Prokaryotes", - "hsa": "Eukaryotes", +def test_parse_taxonomy_lineages(kegg_dump): + cats = parse_taxonomy(kegg_dump / "taxonomy") + assert cats["aaa"] == ["Prokaryotes", "Bacteria", "Firmicutes"] + assert cats["ccc"][0] == "Eukaryotes" + assert cats["bbb"][1] == "Bacteria" + + +def test_organism_domains(kegg_dump): + assert organism_domains(kegg_dump / "taxonomy") == { + "aaa": "Prokaryotes", + "bbb": "Prokaryotes", + "ccc": "Eukaryotes", } -def test_organisms_in_domain_prefix_match(): - assert organisms_in_domain(DUMP / "taxonomy", "prok") == {"bsu", "eco"} - assert organisms_in_domain(DUMP / "taxonomy", "Eukaryotes") == {"hsa"} +def test_organisms_in_domain_prefix_match(kegg_dump): + assert organisms_in_domain(kegg_dump / "taxonomy", "prok") == {"aaa", "bbb"} + assert organisms_in_domain(kegg_dump / "taxonomy", "Eukaryotes") == {"ccc"} def test_parse_taxonomy_handles_skipped_depth(tmp_path): @@ -79,29 +81,29 @@ def test_parse_taxonomy_handles_skipped_depth(tmp_path): # --------------------------------------------------------------------------- # # build_ko_fastas (constructMultiFasta) # --------------------------------------------------------------------------- # -def test_build_ko_fastas_groups_by_ko(organism_gene_ko, tmp_path): - written = build_ko_fastas(organism_gene_ko, DUMP / "genes.pep", tmp_path) - assert set(written) == {"K01194", "K00002"} - k01194 = (tmp_path / "K01194.fa").read_text() - assert k01194.count(">") == 4 # bsu x2 + hsa x2 - assert ">bsu:BSU31050" in k01194 - assert ">xxx:unused" not in k01194 # gene not in any KO is excluded +def test_build_ko_fastas_groups_by_ko(organism_gene_ko, kegg_dump, tmp_path): + written = build_ko_fastas(organism_gene_ko, kegg_dump / "genes.pep", tmp_path) + assert set(written) == {"K90001", "K90002"} + k90001 = (tmp_path / "K90001.fa").read_text() + assert k90001.count(">") == 4 # aaa x2 + ccc x2 + assert ">aaa:GENE01" in k90001 + assert ">zzz:GENE99" not in k90001 # gene not in any KO is excluded -def test_build_ko_fastas_domain_filter(organism_gene_ko, tmp_path): - prok = organisms_in_domain(DUMP / "taxonomy", "prokaryotes") - written = build_ko_fastas(organism_gene_ko, DUMP / "genes.pep", tmp_path, organisms=prok) - # Only prokaryote genes: K01194 keeps bsu (2), K00002 keeps eco (1). - assert (tmp_path / "K01194.fa").read_text().count(">") == 2 - assert ">hsa:" not in (tmp_path / "K01194.fa").read_text() - assert set(written) == {"K01194", "K00002"} +def test_build_ko_fastas_domain_filter(organism_gene_ko, kegg_dump, tmp_path): + prok = organisms_in_domain(kegg_dump / "taxonomy", "prokaryotes") + written = build_ko_fastas(organism_gene_ko, kegg_dump / "genes.pep", tmp_path, organisms=prok) + # Only prokaryote genes: K90001 keeps aaa (2), K90002 keeps bbb (1). + assert (tmp_path / "K90001.fa").read_text().count(">") == 2 + assert ">ccc:" not in (tmp_path / "K90001.fa").read_text() + assert set(written) == {"K90001", "K90002"} -def test_build_ko_fastas_sequences_intact(organism_gene_ko, tmp_path): - build_ko_fastas(organism_gene_ko, DUMP / "genes.pep", tmp_path) - text = (tmp_path / "K00002.fa").read_text() - assert text.startswith(">eco:b0001") - assert "MRVLKFGGTSVANAERFLRVADILESNARQGQVATVLSAPAKITNHLVAMIEKTISGQDA" in text +def test_build_ko_fastas_sequences_intact(organism_gene_ko, kegg_dump, tmp_path): + build_ko_fastas(organism_gene_ko, kegg_dump / "genes.pep", tmp_path) + text = (tmp_path / "K90002.fa").read_text() + assert text.startswith(">bbb:GENE03") + assert "MQFKTLVIDEGHKLPSTWYNACRMQFKTLVIDEGHKLPSTWYNACR" in text # --------------------------------------------------------------------------- # @@ -139,12 +141,12 @@ def test_command_builders(): # build_ko_hmm orchestration (binaries mocked) # --------------------------------------------------------------------------- # def test_build_ko_hmm_multi_sequence_runs_full_pipeline(tmp_path, monkeypatch): - fasta = tmp_path / "K01194.fa" + fasta = tmp_path / "K90001.fa" fasta.write_text(">a\nMKV\n>b\nMRV\n") calls = [] monkeypatch.setattr( - "raven_python.reconstruction.kegg.hmm.resolve_binary", + "raven_toolbox.reconstruction.kegg.hmm.resolve_binary", lambda exe, binary=None: binary or exe, ) @@ -159,8 +161,8 @@ def fake_run(cmd, *, stdout_path=None): Path(cmd[-2]).write_text("HMM\n") return "" - monkeypatch.setattr("raven_python.reconstruction.kegg.hmm._run", fake_run) - out = build_ko_hmm(fasta, tmp_path / "K01194.hmm") + monkeypatch.setattr("raven_toolbox.reconstruction.kegg.hmm._run", fake_run) + out = build_ko_hmm(fasta, tmp_path / "K90001.hmm") assert calls == ["cd-hit", "mafft", "hmmbuild"] assert out.read_text() == "HMM\n" @@ -170,7 +172,7 @@ def test_build_ko_hmm_single_sequence_skips_align(tmp_path, monkeypatch): fasta.write_text(">only\nMKV\n") calls = [] monkeypatch.setattr( - "raven_python.reconstruction.kegg.hmm.resolve_binary", + "raven_toolbox.reconstruction.kegg.hmm.resolve_binary", lambda exe, binary=None: binary or exe, ) @@ -180,16 +182,16 @@ def fake_run(cmd, *, stdout_path=None): Path(cmd[-2]).write_text("HMM\n") return "" - monkeypatch.setattr("raven_python.reconstruction.kegg.hmm._run", fake_run) + monkeypatch.setattr("raven_toolbox.reconstruction.kegg.hmm._run", fake_run) build_ko_hmm(fasta, tmp_path / "K9.hmm") assert calls == ["hmmbuild"] # no cd-hit / mafft for a lone sequence def test_build_ko_hmm_verbose_logs_each_stage(tmp_path, monkeypatch, caplog): - fasta = tmp_path / "K01194.fa" + fasta = tmp_path / "K90001.fa" fasta.write_text(">a\nMKV\n>b\nMRV\n") monkeypatch.setattr( - "raven_python.reconstruction.kegg.hmm.resolve_binary", lambda exe, binary=None: binary or exe + "raven_toolbox.reconstruction.kegg.hmm.resolve_binary", lambda exe, binary=None: binary or exe ) def fake_run(cmd, *, stdout_path=None): @@ -201,31 +203,31 @@ def fake_run(cmd, *, stdout_path=None): Path(cmd[-2]).write_text("HMM\n") return "" - monkeypatch.setattr("raven_python.reconstruction.kegg.hmm._run", fake_run) - with caplog.at_level("INFO", logger="raven_python.reconstruction.kegg.hmm"): - build_ko_hmm(fasta, tmp_path / "K01194.hmm", verbose=True) + monkeypatch.setattr("raven_toolbox.reconstruction.kegg.hmm._run", fake_run) + with caplog.at_level("INFO", logger="raven_toolbox.reconstruction.kegg.hmm"): + build_ko_hmm(fasta, tmp_path / "K90001.hmm", verbose=True) text = caplog.text # Each stage is logged, labelled with the KO id. - assert "[K01194] start: 2 sequences" in text - assert "[K01194] CD-HIT" in text - assert "[K01194] MAFFT" in text - assert "[K01194] hmmbuild: done in" in text + assert "[K90001] start: 2 sequences" in text + assert "[K90001] CD-HIT" in text + assert "[K90001] MAFFT" in text + assert "[K90001] hmmbuild: done in" in text # Each stage is a single line: the tool/params and the timing together, not split. assert "running" not in text - assert "[K01194] complete" in text + assert "[K90001] complete" in text def test_build_ko_hmm_quiet_by_default(tmp_path, monkeypatch, caplog): fasta = tmp_path / "K9.fa" fasta.write_text(">only\nMKV\n") monkeypatch.setattr( - "raven_python.reconstruction.kegg.hmm.resolve_binary", lambda exe, binary=None: binary or exe + "raven_toolbox.reconstruction.kegg.hmm.resolve_binary", lambda exe, binary=None: binary or exe ) monkeypatch.setattr( - "raven_python.reconstruction.kegg.hmm._run", + "raven_toolbox.reconstruction.kegg.hmm._run", lambda cmd, *, stdout_path=None: Path(cmd[-2]).write_text("HMM\n") and "", ) - with caplog.at_level("INFO", logger="raven_python.reconstruction.kegg.hmm"): + with caplog.at_level("INFO", logger="raven_toolbox.reconstruction.kegg.hmm"): build_ko_hmm(fasta, tmp_path / "K9.hmm") # verbose defaults False assert caplog.text == "" @@ -250,7 +252,7 @@ def test_auto_cost_budget_scales_with_memory(monkeypatch): def test_auto_cost_budget_warns_on_low_memory(monkeypatch, caplog): hmm_mod._auto_cost_budget.cache_clear() monkeypatch.setattr(hmm_mod, "_total_memory_bytes", lambda: 7 * 1024**3) - with caplog.at_level("WARNING", logger="raven_python.reconstruction.kegg.hmm"): + with caplog.at_level("WARNING", logger="raven_toolbox.reconstruction.kegg.hmm"): hmm_mod._auto_cost_budget() assert "Limited memory" in caplog.text hmm_mod._auto_cost_budget.cache_clear() @@ -259,7 +261,7 @@ def test_auto_cost_budget_warns_on_low_memory(monkeypatch, caplog): def test_auto_cost_budget_falls_back_without_detection(monkeypatch, caplog): hmm_mod._auto_cost_budget.cache_clear() monkeypatch.setattr(hmm_mod, "_total_memory_bytes", lambda: None) - with caplog.at_level("WARNING", logger="raven_python.reconstruction.kegg.hmm"): + with caplog.at_level("WARNING", logger="raven_toolbox.reconstruction.kegg.hmm"): assert hmm_mod._auto_cost_budget() == hmm_mod._DEFAULT_COST_BUDGET assert "Could not detect system memory" in caplog.text hmm_mod._auto_cost_budget.cache_clear() diff --git a/tests/test_reconstruction_kegg_organism.py b/tests/test_reconstruction_kegg_organism.py index f64f15b0..fbeeb897 100644 --- a/tests/test_reconstruction_kegg_organism.py +++ b/tests/test_reconstruction_kegg_organism.py @@ -1,11 +1,13 @@ -"""Tests for get_kegg_model_for_organism (KEGG organism-ID mode, step 3b.4).""" -from pathlib import Path +"""Tests for get_kegg_model_for_organism (KEGG organism-ID mode, step 3b.4). +The ``kegg_dump`` fixture (tests/conftest.py) is a small, fully fictional dump — +no real KEGG content is committed. +""" import cobra import pandas as pd import pytest -from raven_python.reconstruction.kegg import ( +from raven_toolbox.reconstruction.kegg import ( build_kegg_tables, build_reference_model, get_kegg_model_for_organism, @@ -15,17 +17,15 @@ parse_kegg_reactions, ) -DUMP = Path(__file__).parent / "data" / "kegg_dump" - @pytest.fixture(scope="module") -def artefacts(): - reactions = parse_kegg_reactions(DUMP) - compounds = parse_kegg_compounds(DUMP) +def artefacts(kegg_dump): + reactions = parse_kegg_reactions(kegg_dump) + compounds = parse_kegg_compounds(kegg_dump) linked = {ko for r in reactions for ko in r.kos} - from raven_python.reconstruction.kegg import parse_kegg_kos + from raven_toolbox.reconstruction.kegg import parse_kegg_kos - kos = parse_kegg_kos(DUMP, keep=linked) + kos = parse_kegg_kos(kegg_dump, keep=linked) model = build_reference_model(reactions, compounds) tables = build_kegg_tables(reactions, kos) return model, tables @@ -46,43 +46,43 @@ def _build(artefacts, organism_id, **kw): # --------------------------------------------------------------------------- # # Core behaviour # --------------------------------------------------------------------------- # -def test_eco_keeps_only_its_reactions(artefacts): - # eco has b0001 -> K00002 -> R00100 only. - model = _build(artefacts, "eco") - assert {r.id for r in model.reactions} == {"R00100"} - assert model.id == "eco" - - -def test_eco_gpr_and_gene_annotation(artefacts): - model = _build(artefacts, "eco") - r = model.reactions.get_by_id("R00100") - assert r.gene_reaction_rule == "b0001" - assert model.genes.get_by_id("b0001").annotation["kegg.genes"] == "eco:b0001" +def test_bbb_keeps_only_its_reactions(artefacts): + # bbb has GENE03 -> K90002 -> R90100 only. + model = _build(artefacts, "bbb") + assert {r.id for r in model.reactions} == {"R90100"} + assert model.id == "bbb" + + +def test_bbb_gpr_and_gene_annotation(artefacts): + model = _build(artefacts, "bbb") + r = model.reactions.get_by_id("R90100") + assert r.gene_reaction_rule == "GENE03" + assert model.genes.get_by_id("GENE03").annotation["kegg.genes"] == "bbb:GENE03" assert r.notes["note"].startswith("Included by get_kegg_model_for_organism") -def test_bsu_or_joins_multiple_genes(artefacts): - # bsu has BSU31050 + BSU31060, both -> K01194 -> R00010. - model = _build(artefacts, "bsu") - r = model.reactions.get_by_id("R00010") - assert set(r.genes) == {model.genes.get_by_id("BSU31050"), model.genes.get_by_id("BSU31060")} - assert r.gene_reaction_rule == "BSU31050 or BSU31060" +def test_aaa_or_joins_multiple_genes(artefacts): + # aaa has GENE01 + GENE02, both -> K90001 -> R90010. + model = _build(artefacts, "aaa") + r = model.reactions.get_by_id("R90010") + assert set(r.genes) == {model.genes.get_by_id("GENE01"), model.genes.get_by_id("GENE02")} + assert r.gene_reaction_rule == "GENE01 or GENE02" def test_case_insensitive_organism(artefacts): - assert "R00010" in _build(artefacts, "BSU").reactions + assert "R90010" in _build(artefacts, "AAA").reactions def test_orphan_metabolites_pruned(artefacts): - # eco keeps only R00100 (C00002, C00003); trehalose/glucose mets should go. - model = _build(artefacts, "eco") - assert {m.id for m in model.metabolites} == {"C00002", "C00003"} + # bbb keeps only R90100 (C90002, C90003); other mets should go. + model = _build(artefacts, "bbb") + assert {m.id for m in model.metabolites} == {"C90002", "C90003"} def test_reference_model_unmodified(artefacts): reference, _ = artefacts before = len(reference.reactions) - _build(artefacts, "eco") + _build(artefacts, "bbb") assert len(reference.reactions) == before # worked on a copy assert len(reference.genes) == 0 @@ -91,16 +91,16 @@ def test_reference_model_unmodified(artefacts): # Spontaneous handling # --------------------------------------------------------------------------- # def test_spontaneous_reaction_kept_without_genes(artefacts): - # R00100 is spontaneous; for bsu it has no genes but is kept (no GPR). - model = _build(artefacts, "bsu", keep_spontaneous=True) - assert "R00100" in model.reactions - assert model.reactions.get_by_id("R00100").gene_reaction_rule == "" + # R90100 is spontaneous; for aaa it has no genes but is kept (no GPR). + model = _build(artefacts, "aaa", keep_spontaneous=True) + assert "R90100" in model.reactions + assert model.reactions.get_by_id("R90100").gene_reaction_rule == "" def test_spontaneous_dropped_when_disabled(artefacts): - model = _build(artefacts, "bsu", keep_spontaneous=False) - assert "R00100" not in model.reactions - assert "R00010" in model.reactions # the gene-backed reaction stays + model = _build(artefacts, "aaa", keep_spontaneous=False) + assert "R90100" not in model.reactions + assert "R90010" in model.reactions # the gene-backed reaction stays # --------------------------------------------------------------------------- # @@ -142,7 +142,7 @@ def test_general_kept_when_enabled(): # --------------------------------------------------------------------------- # def test_unknown_organism_raises(artefacts): with pytest.raises(ValueError, match="no genes"): - _build(artefacts, "zzz") + _build(artefacts, "qqq") def test_domain_mode_needs_taxonomy(artefacts): @@ -150,30 +150,48 @@ def test_domain_mode_needs_taxonomy(artefacts): _build(artefacts, "eukaryotes") -def test_domain_mode_keeps_all_domain_organisms(artefacts): - # Prokaryotes (bsu + eco) -> R00010 (bsu genes) and R00100 (eco gene). - model = _build(artefacts, "prokaryotes", taxonomy=DUMP / "taxonomy") - assert "R00010" in model.reactions - assert "R00100" in model.reactions +def test_domain_mode_keeps_all_domain_organisms(artefacts, kegg_dump): + # Prokaryotes (aaa + bbb) -> R90010 (aaa genes) and R90100 (bbb genes). + model = _build(artefacts, "prokaryotes", taxonomy=kegg_dump / "taxonomy") + assert "R90010" in model.reactions + assert "R90100" in model.reactions # Genes are organism-qualified in domain mode to stay distinct. - assert {g.id for g in model.reactions.get_by_id("R00010").genes} == { - "bsu:BSU31050", - "bsu:BSU31060", + assert {g.id for g in model.reactions.get_by_id("R90010").genes} == { + "aaa:GENE01", + "aaa:GENE02", } -def test_domain_mode_eukaryotes(artefacts): - # Eukaryotes (hsa) -> R00010 via hsa:124/125; eco-only R00100 absent of genes - # but it is spontaneous, so kept without GPR. - model = _build(artefacts, "eukaryotes", taxonomy=DUMP / "taxonomy") - assert {g.id for g in model.reactions.get_by_id("R00010").genes} == { - "hsa:124", - "hsa:125", +def test_domain_mode_eukaryotes(artefacts, kegg_dump): + # Eukaryotes (ccc) -> R90010 via ccc:GENE04/GENE05; bbb-only R90100 absent of + # genes but it is spontaneous, so kept without GPR. + model = _build(artefacts, "eukaryotes", taxonomy=kegg_dump / "taxonomy") + assert {g.id for g in model.reactions.get_by_id("R90010").genes} == { + "ccc:GENE04", + "ccc:GENE05", } -def test_from_artefacts_roundtrip(tmp_path): - parse_kegg_dump(DUMP, tmp_path) - model = get_kegg_model_for_organism_from_artefacts("eco", tmp_path) - assert {r.id for r in model.reactions} == {"R00100"} - assert model.reactions.get_by_id("R00100").gene_reaction_rule == "b0001" +def test_from_artefacts_roundtrip(kegg_dump, tmp_path): + parse_kegg_dump(kegg_dump, tmp_path) + model = get_kegg_model_for_organism_from_artefacts("bbb", tmp_path) + assert {r.id for r in model.reactions} == {"R90100"} + assert model.reactions.get_by_id("R90100").gene_reaction_rule == "GENE03" + + +def test_from_artefacts_domain_mode_auto_resolves_taxonomy(kegg_dump, tmp_path): + # Domain mode must auto-resolve the taxonomy artefact from artefact_dir, without the + # caller passing taxonomy= (previously raised "Domain mode needs the KEGG taxonomy"). + import gzip + import shutil + + parse_kegg_dump(kegg_dump, tmp_path) + with open(kegg_dump / "taxonomy", "rb") as src, gzip.open(tmp_path / "taxonomy.gz", "wb") as out: + shutil.copyfileobj(src, out) + model = get_kegg_model_for_organism_from_artefacts("prokaryotes", tmp_path) + # Same prokaryote-domain content (aaa + bbb) as the explicit-taxonomy build. + assert "R90010" in model.reactions and "R90100" in model.reactions + assert {g.id for g in model.reactions.get_by_id("R90010").genes} == { + "aaa:GENE01", + "aaa:GENE02", + } diff --git a/tests/test_reconstruction_kegg_parse.py b/tests/test_reconstruction_kegg_parse.py index 23d8f71c..b560e133 100644 --- a/tests/test_reconstruction_kegg_parse.py +++ b/tests/test_reconstruction_kegg_parse.py @@ -1,9 +1,11 @@ -"""Tests for the KEGG dump parser (reconstruction/kegg/parse.py, step 3b.2).""" -from pathlib import Path +"""Tests for the KEGG dump parser (reconstruction/kegg/parse.py, step 3b.2). +The ``kegg_dump`` fixture (tests/conftest.py) is a small, fully fictional dump — +no real KEGG content is committed. +""" import pytest -from raven_python.reconstruction.kegg import ( +from raven_toolbox.reconstruction.kegg import ( build_kegg_tables, build_reference_model, parse_kegg_compounds, @@ -14,45 +16,43 @@ write_kegg_tables, ) -DUMP = Path(__file__).parent / "data" / "kegg_dump" - @pytest.fixture(scope="module") -def reactions(): - return parse_kegg_reactions(DUMP) +def reactions(kegg_dump): + return parse_kegg_reactions(kegg_dump) @pytest.fixture(scope="module") -def compounds(): - return parse_kegg_compounds(DUMP) +def compounds(kegg_dump): + return parse_kegg_compounds(kegg_dump) @pytest.fixture(scope="module") -def kos(): - linked = {ko for r in parse_kegg_reactions(DUMP) for ko in r.kos} - return parse_kegg_kos(DUMP, keep=linked) +def kos(kegg_dump): + linked = {ko for r in parse_kegg_reactions(kegg_dump) for ko in r.kos} + return parse_kegg_kos(kegg_dump, keep=linked) # --------------------------------------------------------------------------- # # Reactions # --------------------------------------------------------------------------- # def test_reactions_parsed(reactions): - assert {r.id for r in reactions} == {"R00010", "R00100", "R00200", "R00300", "R00400"} + assert {r.id for r in reactions} == {"R90010", "R90100", "R90200", "R90300", "R90400"} def test_reaction_fields(reactions): - r = next(r for r in reactions if r.id == "R00010") - assert r.name == "alpha,alpha-trehalose glucohydrolase" - assert r.eccodes == ["3.2.1.28"] - assert r.kos == ["K01194"] - # rn01100 is an overview map and must be skipped. - assert r.pathways == ["rn00500"] + r = next(r for r in reactions if r.id == "R90010") + assert r.name == "fictional glucohydrolase analogue" + assert r.eccodes == ["9.9.9.99"] + assert r.kos == ["K90001"] + # rn01199 is an overview map (rn011..) and must be skipped. + assert r.pathways == ["rn09500"] def test_stoichiometry_cached(reactions): """parse_kegg_reactions populates the cached stoichiometry so build_reference_model doesn't have to re-parse (known_issues.md D2).""" - r = next(r for r in reactions if r.id == "R00010") + r = next(r for r in reactions if r.id == "R90010") assert r.stoichiometry # non-empty # Reactants negative, products positive. assert all(c != 0 for c in r.stoichiometry.values()) @@ -61,56 +61,56 @@ def test_stoichiometry_cached(reactions): def test_spontaneous_flag(reactions): - assert next(r for r in reactions if r.id == "R00100").spontaneous - assert not next(r for r in reactions if r.id == "R00010").spontaneous + assert next(r for r in reactions if r.id == "R90100").spontaneous + assert not next(r for r in reactions if r.id == "R90010").spontaneous def test_general_flag(reactions): - assert next(r for r in reactions if r.id == "R00300").general + assert next(r for r in reactions if r.id == "R90300").general def test_undefined_stoich_flag(reactions): - assert next(r for r in reactions if r.id == "R00200").undefined_stoich - assert not next(r for r in reactions if r.id == "R00010").undefined_stoich + assert next(r for r in reactions if r.id == "R90200").undefined_stoich + assert not next(r for r in reactions if r.id == "R90010").undefined_stoich def test_mapformula_makes_irreversible(reactions): - # R00100 is drawn one direction in its only map -> irreversible. - assert not next(r for r in reactions if r.id == "R00100").reversible - # R00010 is drawn in conflicting directions across maps -> stays reversible. - assert next(r for r in reactions if r.id == "R00010").reversible + # R90100 is drawn one direction in its only map -> irreversible. + assert not next(r for r in reactions if r.id == "R90100").reversible + # R90010 is drawn in conflicting directions across maps -> stays reversible. + assert next(r for r in reactions if r.id == "R90010").reversible # --------------------------------------------------------------------------- # # Compounds # --------------------------------------------------------------------------- # def test_compound_first_name_only(compounds): - water = next(c for c in compounds if c.id == "C00001") - assert water.name == "H2O" - assert water.chebi == ["CHEBI:15377"] - assert water.pubchem == ["3303"] + water = next(c for c in compounds if c.id == "C90001") + assert water.name == "Aqualike" + assert water.chebi == ["CHEBI:95377"] + assert water.pubchem == ["9303"] def test_inchi_overrides_formula(compounds): - glucose = next(c for c in compounds if c.id == "C00031") + glucose = next(c for c in compounds if c.id == "C90031") assert glucose.inchi.startswith("InChI=") assert glucose.formula == "" # cleared when an InChI is available - assert glucose.chebi == ["CHEBI:4167", "CHEBI:17634"] + assert glucose.chebi == ["CHEBI:94167", "CHEBI:97634"] # --------------------------------------------------------------------------- # # KOs / genes # --------------------------------------------------------------------------- # def test_kos_limited_to_keep(kos): - # K99999 is unlinked (excluded by keep); K09999 is referenced but absent. - assert {ko.id for ko in kos} == {"K01194", "K00002"} + # K90099 is unlinked (excluded by keep); K90009 is referenced but absent. + assert {ko.id for ko in kos} == {"K90001", "K90002"} def test_ko_genes_lowercased_and_stripped(kos): - k = next(ko for ko in kos if ko.id == "K01194") - assert k.name == "alpha,alpha-trehalase [EC:3.2.1.28]" - assert ("bsu", "BSU31050") in k.genes # '(gbsB)' suffix stripped, org lowercased - assert ("hsa", "125") in k.genes + k = next(ko for ko in kos if ko.id == "K90001") + assert k.name == "fictional trehalase analogue [EC:9.9.9.99]" + assert ("aaa", "GENE01") in k.genes # '(alias1)' suffix stripped, org lowercased + assert ("ccc", "GENE05") in k.genes # --------------------------------------------------------------------------- # @@ -125,29 +125,29 @@ def test_reference_model_is_gene_free(reactions, compounds): def test_empty_reaction_dropped(reactions, compounds): model = build_reference_model(reactions, compounds) - assert "R00400" not in model.reactions # C00007 <=> C00007 cancels out - assert "C00007" not in model.metabolites # and its only metabolite is unused + assert "R90400" not in model.reactions # C90007 <=> C90007 cancels out + assert "C90007" not in model.metabolites # and its only metabolite is unused def test_reaction_bounds_follow_reversibility(reactions, compounds): model = build_reference_model(reactions, compounds) - assert model.reactions.get_by_id("R00010").bounds == (-1000.0, 1000.0) - assert model.reactions.get_by_id("R00100").bounds == (0.0, 1000.0) + assert model.reactions.get_by_id("R90010").bounds == (-1000.0, 1000.0) + assert model.reactions.get_by_id("R90100").bounds == (0.0, 1000.0) def test_reaction_stoichiometry_and_annotation(reactions, compounds): model = build_reference_model(reactions, compounds) - r = model.reactions.get_by_id("R00010") + r = model.reactions.get_by_id("R90010") coefs = {m.id: c for m, c in r.metabolites.items()} - assert coefs == {"C01083": -1.0, "C00001": -1.0, "C00031": 2.0} - assert r.annotation["kegg.orthology"] == ["K01194"] - assert r.annotation["ec-code"] == ["3.2.1.28"] + assert coefs == {"C91083": -1.0, "C90001": -1.0, "C90031": 2.0} + assert r.annotation["kegg.orthology"] == ["K90001"] + assert r.annotation["ec-code"] == ["9.9.9.99"] def test_metabolite_annotation(reactions, compounds): model = build_reference_model(reactions, compounds) - glucose = model.metabolites.get_by_id("C00031") - assert glucose.name == "D-Glucose" + glucose = model.metabolites.get_by_id("C90031") + assert glucose.name == "Glucolike" assert glucose.annotation["inchi"].startswith("InChI=") @@ -157,25 +157,25 @@ def test_metabolite_annotation(reactions, compounds): def test_ko_reaction_table(reactions, kos): tables = build_kegg_tables(reactions, kos) pairs = set(map(tuple, tables["ko_reaction"].to_numpy())) - assert ("K01194", "R00010") in pairs - assert ("K09999", "R00300") in pairs # kept even though KO entry is missing + assert ("K90001", "R90010") in pairs + assert ("K90009", "R90300") in pairs # kept even though KO entry is missing def test_organism_gene_ko_table(reactions, kos): tables = build_kegg_tables(reactions, kos) rows = set(map(tuple, tables["organism_gene_ko"].to_numpy())) - assert ("bsu", "BSU31050", "K01194") in rows - assert ("eco", "b0001", "K00002") in rows + assert ("aaa", "GENE01", "K90001") in rows + assert ("bbb", "GENE03", "K90002") in rows assert len(rows) == 5 def test_rxn_flags_table(reactions, kos): tables = build_kegg_tables(reactions, kos) flags = tables["rxn_flags"].set_index("reaction") - assert bool(flags.loc["R00100", "spontaneous"]) - assert bool(flags.loc["R00200", "undefined_stoich"]) - assert bool(flags.loc["R00300", "general"]) - assert not bool(flags.loc["R00010", "spontaneous"]) + assert bool(flags.loc["R90100", "spontaneous"]) + assert bool(flags.loc["R90200", "undefined_stoich"]) + assert bool(flags.loc["R90300", "general"]) + assert not bool(flags.loc["R90010", "spontaneous"]) # --------------------------------------------------------------------------- # @@ -189,32 +189,42 @@ def test_tables_roundtrip_gzipped_tsv(reactions, kos, tmp_path): assert set(map(tuple, back.to_numpy())) == set(map(tuple, tables["ko_reaction"].to_numpy())) -def test_parse_kegg_dump_writes_artefacts(tmp_path): - paths = parse_kegg_dump(DUMP, tmp_path) +def test_parse_kegg_dump_writes_artefacts(kegg_dump, tmp_path): + paths = parse_kegg_dump(kegg_dump, tmp_path) assert set(paths) >= { "ko_reaction", "ko_names", "organism_gene_ko", "rxn_flags", "reference_model" } assert (tmp_path / "reference_model.yml.gz").is_file() - # organism_gene_ko is streamed to a sorted, xz-compressed TSV. - assert paths["organism_gene_ko"].name == "organism_gene_ko.tsv.xz" + # organism_gene_ko is streamed to a sorted, gzipped TSV. + assert paths["organism_gene_ko"].name == "organism_gene_ko.tsv.gz" ogk = read_kegg_table(paths["organism_gene_ko"]) assert set(ogk.columns) == {"organism", "gene", "ko"} - assert ("eco", "b0001", "K00002") in set(map(tuple, ogk.to_numpy())) + assert ("bbb", "GENE03", "K90002") in set(map(tuple, ogk.to_numpy())) # Rows are sorted by (organism, gene) — the property that makes them compress. keys = list(zip(ogk["organism"], ogk["gene"], strict=True)) assert keys == sorted(keys) -def test_stream_organism_gene_ko_external_merge(tmp_path): +def test_parse_kegg_dump_version_prefixes_filenames(kegg_dump, tmp_path): + paths = parse_kegg_dump(kegg_dump, tmp_path, version="kegg116") + # Dict keys stay logical; the files on disk are version-prefixed. + assert set(paths) >= {"ko_reaction", "organism_gene_ko", "reference_model"} + assert paths["organism_gene_ko"].name == "kegg116_organism_gene_ko.tsv.gz" + assert paths["reference_model"].name == "kegg116_reference_model.yml.gz" + assert (tmp_path / "kegg116_ko_reaction.tsv.gz").is_file() + assert read_kegg_table(paths["organism_gene_ko"]).columns.tolist() == ["organism", "gene", "ko"] + + +def test_stream_organism_gene_ko_external_merge(kegg_dump, tmp_path): """A tiny chunk_rows forces multiple sorted runs to be merged; output stays sorted.""" - from raven_python.reconstruction.kegg.parse import stream_organism_gene_ko + from raven_toolbox.reconstruction.kegg.parse import stream_organism_gene_ko - out = tmp_path / "organism_gene_ko.tsv.xz" - keep = {ko.id for ko in parse_kegg_kos(DUMP)} - names = stream_organism_gene_ko(DUMP, keep, out, chunk_rows=1) + out = tmp_path / "organism_gene_ko.tsv.gz" + keep = {ko.id for ko in parse_kegg_kos(kegg_dump)} + names = stream_organism_gene_ko(kegg_dump, keep, out, chunk_rows=1) assert out.is_file() and not list(tmp_path.glob("ogk_sort_*")) # temp dir cleaned up ogk = read_kegg_table(out) keys = list(zip(ogk["organism"], ogk["gene"], strict=True)) assert keys == sorted(keys) - assert ("eco", "b0001", "K00002") in set(map(tuple, ogk.to_numpy())) + assert ("bbb", "GENE03", "K90002") in set(map(tuple, ogk.to_numpy())) assert set(names.columns) == {"ko", "name"} diff --git a/tests/test_reconstruction_kegg_phyl_dist.py b/tests/test_reconstruction_kegg_phyl_dist.py new file mode 100644 index 00000000..0139abc3 --- /dev/null +++ b/tests/test_reconstruction_kegg_phyl_dist.py @@ -0,0 +1,70 @@ +"""Tests for the KEGG phylogenetic distance generator (RAVEN getPhylDist port).""" +import gzip + +import numpy as np + +from raven_toolbox.reconstruction.kegg.taxonomy import parse_taxonomy_records, phyl_dist + +# Tiny taxonomy: two prokaryotes sharing a lineage, a 3-deep mammal, a 2-deep fungus. +TAXONOMY = ( + "# Prokaryotes\n" + "## Bacteria\n" + "T1\tbsu\tT1\tBacillus subtilis\n" + "T2\teco\tT2\tEscherichia coli\n" + "# Eukaryotes\n" + "## Animals\n" + "### Mammals\n" + "T3\thsa\tT3\tHomo sapiens (human)\n" + "## Fungi\n" + "T4\tsce\tT4\tSaccharomyces cerevisiae\n" +) + + +def _write(tmp_path): + p = tmp_path / "taxonomy" + p.write_text(TAXONOMY, encoding="utf-8") + return p + + +def test_parse_records_ids_names_lineages(tmp_path): + recs = parse_taxonomy_records(_write(tmp_path)) + assert [r[0] for r in recs] == ["bsu", "eco", "hsa", "sce"] + # Names keep RAVEN's trailing parenthetical. + assert [r[1] for r in recs] == [ + "Bacillus subtilis", + "Escherichia coli", + "Homo sapiens (human)", + "Saccharomyces cerevisiae", + ] + lineage = {r[0]: r[2] for r in recs} + assert lineage["hsa"] == ["Eukaryotes", "Animals", "Mammals"] + assert lineage["sce"] == ["Eukaryotes", "Fungi"] + + +def test_phyl_dist_matches_raven_formula(tmp_path): + pd = phyl_dist(_write(tmp_path)) + assert pd.ids == ["bsu", "eco", "hsa", "sce"] + # Hand-computed from RAVEN's distMat = (Li-Lj) + min(Li,Lj) - k. + expected = np.array( + [ + [0, 0, 0, 1], + [0, 0, 0, 1], + [2, 2, 0, 2], + [1, 1, 0, 0], + ] + ) + np.testing.assert_array_equal(pd.dist_matrix, expected) + assert np.all(np.diag(pd.dist_matrix) == 0) # self-distance is 0 + assert pd.dist_matrix[2, 0] != pd.dist_matrix[0, 2] # RAVEN metric is asymmetric + + +def test_phyl_dist_only_in_kingdom_blocks_cross_domain(tmp_path): + pd = phyl_dist(_write(tmp_path), only_in_kingdom=True) + assert np.isinf(pd.dist_matrix[0, 2]) and np.isinf(pd.dist_matrix[0, 3]) # prok vs euk + assert np.isfinite(pd.dist_matrix[2, 3]) # both eukaryotes + + +def test_phyl_dist_reads_gzip(tmp_path): + p = tmp_path / "taxonomy.gz" + p.write_bytes(gzip.compress(TAXONOMY.encode("utf-8"))) + assert phyl_dist(p).ids == ["bsu", "eco", "hsa", "sce"] diff --git a/tests/test_reconstruction_kegg_query.py b/tests/test_reconstruction_kegg_query.py index 49aae60a..0dd23f09 100644 --- a/tests/test_reconstruction_kegg_query.py +++ b/tests/test_reconstruction_kegg_query.py @@ -1,30 +1,30 @@ -"""Tests for the KEGG HMM-query path (reconstruction/kegg/query.py, step 3b.5).""" -from pathlib import Path +"""Tests for the KEGG HMM-query path (reconstruction/kegg/query.py, step 3b.5). +The ``kegg_dump`` fixture (tests/conftest.py) is a small, fully fictional dump — +no real KEGG content is committed. +""" import pandas as pd import pytest -from raven_python.reconstruction.kegg import ( +from raven_toolbox.reconstruction.kegg import ( assign_kos, build_kegg_tables, build_reference_model, get_kegg_model_from_sequences, - parse_hmmscan_tblout, + parse_hmmsearch_tblout, parse_kegg_compounds, parse_kegg_kos, parse_kegg_reactions, ) -DUMP = Path(__file__).parent / "data" / "kegg_dump" - -# A minimal hmmscan --tblout excerpt: target(KO) accession query(gene) ... evalue ... +# A minimal hmmsearch --tblout excerpt: target(gene) accession query(KO) ... evalue ... TBLOUT = """\ # --- full sequence ---- # target name accession query name accession E-value score bias #------------------- ---------- ----------- ---------- --------- ------ ----- -K01194 - gene1 - 1e-120 400.0 0.0 -K01194 - gene2 - 1e-100 350.0 0.0 -K00002 - gene1 - 1e-10 40.0 0.0 +gene1 - K90001 - 1e-120 400.0 0.0 +gene2 - K90001 - 1e-100 350.0 0.0 +gene1 - K90002 - 1e-10 40.0 0.0 """ @@ -32,32 +32,32 @@ # Parsing # --------------------------------------------------------------------------- # def test_parse_tblout_skips_comments(): - hits = parse_hmmscan_tblout(TBLOUT) + hits = parse_hmmsearch_tblout(TBLOUT) assert list(hits.columns) == ["ko", "gene", "evalue"] assert len(hits) == 3 - assert set(hits["ko"]) == {"K01194", "K00002"} + assert set(hits["ko"]) == {"K90001", "K90002"} assert hits.iloc[0]["evalue"] == 1e-120 def test_parse_tblout_empty(): - assert parse_hmmscan_tblout("# only a header\n").empty + assert parse_hmmsearch_tblout("# only a header\n").empty # --------------------------------------------------------------------------- # # assign_kos scoring/filters # --------------------------------------------------------------------------- # def test_cutoff_excludes_weak_hits(): - hits = parse_hmmscan_tblout(TBLOUT) - # gene1->K00002 has evalue 1e-10, above the default cutoff 1e-30: dropped. + hits = parse_hmmsearch_tblout(TBLOUT) + # gene1->K90002 has evalue 1e-10, above the default cutoff 1e-30: dropped. assigned = assign_kos(hits) - assert "K00002" not in assigned - assert set(assigned["K01194"]) == {"gene1", "gene2"} + assert "K90002" not in assigned + assert set(assigned["K90001"]) == {"gene1", "gene2"} def test_loose_cutoff_keeps_hit(): - hits = parse_hmmscan_tblout(TBLOUT) + hits = parse_hmmsearch_tblout(TBLOUT) assigned = assign_kos(hits, cutoff=1e-5, min_score_ratio_g=0.0, min_score_ratio_ko=0.0) - assert assigned.get("K00002") == ["gene1"] + assert assigned.get("K90002") == ["gene1"] def test_min_score_ratio_ko_prunes_weak_member(): @@ -95,25 +95,25 @@ def test_cutoff_ge_one_rejected(): # --------------------------------------------------------------------------- # -# Model assembly via the HMM path (hmmscan mocked) +# Model assembly via the HMM path (hmmsearch mocked) # --------------------------------------------------------------------------- # @pytest.fixture(scope="module") -def reference_and_tables(): - reactions = parse_kegg_reactions(DUMP) - compounds = parse_kegg_compounds(DUMP) +def reference_and_tables(kegg_dump): + reactions = parse_kegg_reactions(kegg_dump) + compounds = parse_kegg_compounds(kegg_dump) linked = {ko for r in reactions for ko in r.kos} - kos = parse_kegg_kos(DUMP, keep=linked) + kos = parse_kegg_kos(kegg_dump, keep=linked) return build_reference_model(reactions, compounds), build_kegg_tables(reactions, kos) def test_get_model_from_sequences(reference_and_tables, monkeypatch): model_ref, tables = reference_and_tables - # Mock the HMM search: K01194 -> myGeneA/myGeneB (-> R00010). + # Mock the HMM search: K90001 -> myGeneA/myGeneB (-> R90010). monkeypatch.setattr( - "raven_python.reconstruction.kegg.query.run_hmmscan", + "raven_toolbox.reconstruction.kegg.query.run_hmmsearch", lambda *a, **k: ( - "K01194 - myGeneA - 1e-120 400 0\n" - "K01194 - myGeneB - 1e-110 380 0\n" + "myGeneA - K90001 - 1e-120 400 0\n" + "myGeneB - K90001 - 1e-110 380 0\n" ), ) model = get_kegg_model_from_sequences( @@ -125,8 +125,8 @@ def test_get_model_from_sequences(reference_and_tables, monkeypatch): model_id="myorg", ) assert model.id == "myorg" - r = model.reactions.get_by_id("R00010") + r = model.reactions.get_by_id("R90010") assert set(r.gene_reaction_rule.split(" or ")) == {"myGeneA", "myGeneB"} assert r.notes["note"].endswith("(using HMMs)") - # R00200/R00300 had no matched KOs and are not spontaneous -> absent. - assert "R00200" not in model.reactions + # R90200/R90300 had no matched KOs and are not spontaneous -> absent. + assert "R90200" not in model.reactions diff --git a/tests/test_tasks.py b/tests/test_tasks.py index 1b5e6bd7..7d60db48 100644 --- a/tests/test_tasks.py +++ b/tests/test_tasks.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.tasks import Task, check_tasks, parse_task_list +from raven_toolbox.tasks import Task, check_tasks, parse_task_list TASK_TSV = ( "ID\tDESCRIPTION\tIN\tIN UB\tOUT\tOUT LB\tEQU\tSHOULD FAIL\n" diff --git a/tests/test_tasks_essential.py b/tests/test_tasks_essential.py index 53523789..d240e095 100644 --- a/tests/test_tasks_essential.py +++ b/tests/test_tasks_essential.py @@ -11,7 +11,7 @@ make_test_task, ) -from raven_python.tasks import ( +from raven_toolbox.tasks import ( EssentialReactionsResult, Task, find_task_essential_reactions, diff --git a/tests/test_utils_balance.py b/tests/test_utils_balance.py index aa1e47ed..776b8b7b 100644 --- a/tests/test_utils_balance.py +++ b/tests/test_utils_balance.py @@ -2,7 +2,7 @@ import cobra import pytest -from raven_python.utils import ElementalBalance, get_elemental_balance +from raven_toolbox.utils import ElementalBalance, get_elemental_balance @pytest.fixture diff --git a/tests/test_utils_gpr.py b/tests/test_utils_gpr.py index 275d020b..d0be6db5 100644 --- a/tests/test_utils_gpr.py +++ b/tests/test_utils_gpr.py @@ -1,8 +1,8 @@ -"""Tests for raven_python.utils.gpr (GPR linting).""" +"""Tests for raven_toolbox.utils.gpr (GPR linting).""" import cobra import pytest -from raven_python.utils import GPRIssue, find_non_dnf_grrules, is_dnf +from raven_toolbox.utils import GPRIssue, find_non_dnf_grrules, is_dnf @pytest.mark.parametrize( diff --git a/tests/test_utils_parse.py b/tests/test_utils_parse.py new file mode 100644 index 00000000..82946c42 --- /dev/null +++ b/tests/test_utils_parse.py @@ -0,0 +1,57 @@ +"""Tests for raven_toolbox.utils.parse helpers.""" +from __future__ import annotations + +import cobra + +from raven_toolbox.utils.parse import parse_name_comp, subsystem_to_str + + +def test_parse_name_comp_basic(): + assert parse_name_comp("ATP[c]") == ("ATP", "c") + assert parse_name_comp("ATP") == ("ATP", None) + assert parse_name_comp("weird[name][m]") == ("weird[name]", "m") + + +def test_subsystem_to_str_scalar_and_none(): + assert subsystem_to_str("glycolysis") == "glycolysis" + assert subsystem_to_str("") == "" + assert subsystem_to_str(None) == "" + + +def test_subsystem_to_str_joins_list_without_data_loss(): + # A multi-subsystem reaction keeps every part (unlike taking only the first). + assert subsystem_to_str(["glycolysis", "TCA cycle"]) == "glycolysis;TCA cycle" + # Empty parts are dropped; the rest survive. + assert subsystem_to_str(["", "x"]) == "x" + + +def test_subsystem_to_str_coerces_non_string_items(): + # Non-string items must not crash (the old excel.py ";".join did). + assert subsystem_to_str([1, "name"]) == "1;name" + + +def test_excel_export_handles_list_subsystem(tmp_path): + import pytest + + pytest.importorskip("openpyxl") + from openpyxl import load_workbook + + from raven_toolbox.io.excel import export_to_excel + + model = cobra.Model("m") + a = cobra.Metabolite("a_c", compartment="c") + b = cobra.Metabolite("b_c", compartment="c") + model.add_metabolites([a, b]) + r = cobra.Reaction("R1", lower_bound=-1000, upper_bound=1000) + model.add_reactions([r]) + r.add_metabolites({a: -1, b: 1}) + r.subsystem = ["glycolysis", "TCA cycle"] # list, would crash old ";".join + + out = tmp_path / "m.xlsx" + export_to_excel(model, out) + wb = load_workbook(out) + ws = wb["RXNS"] + header = [c.value for c in next(ws.iter_rows(max_row=1))] + sub_col = header.index("SUBSYSTEM") + row1 = [c.value for c in list(ws.iter_rows(min_row=2, max_row=2))[0]] + assert row1[sub_col] == "glycolysis;TCA cycle" diff --git a/tests/test_utils_sort.py b/tests/test_utils_sort.py index 18bca249..74d9b32f 100644 --- a/tests/test_utils_sort.py +++ b/tests/test_utils_sort.py @@ -1,9 +1,9 @@ """Tests for sort_identifiers and write_yaml_model(sort_ids=True).""" import cobra -from raven_python.io import read_yaml_model, write_yaml_model -from raven_python.manipulation import add_reactions_from_equations -from raven_python.utils import sort_identifiers +from raven_toolbox.io import read_yaml_model, write_yaml_model +from raven_toolbox.manipulation import add_reactions_from_equations +from raven_toolbox.utils import sort_identifiers def _model(): diff --git a/tests/test_utils_validate.py b/tests/test_utils_validate.py index 2d38e6f3..9707678a 100644 --- a/tests/test_utils_validate.py +++ b/tests/test_utils_validate.py @@ -2,8 +2,8 @@ import cobra import pytest -from raven_python.manipulation import add_reactions_from_equations -from raven_python.utils import ModelIssue, check_model +from raven_toolbox.manipulation import add_reactions_from_equations +from raven_toolbox.utils import ModelIssue, check_model def _categories(issues, category): diff --git a/tests/tinit_oracles.py b/tests/tinit_oracles.py index 638956dc..28dffcf2 100644 --- a/tests/tinit_oracles.py +++ b/tests/tinit_oracles.py @@ -93,7 +93,7 @@ def make_test_model() -> cobra.Model: # The task: generate e[s] from a[s] (RAVEN getTstModelTasks()). def make_test_task(): """RAVEN getTstModelTasks(): make e[s] from a[s].""" - from raven_python.tasks import Task + from raven_toolbox.tasks import Task return Task( id="Gen e[s] from a[s]", From e3061900908bc7ab590352429461937a76d9ec6b Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Wed, 17 Jun 2026 19:46:37 +0200 Subject: [PATCH 2/3] Fix add_sbo_terms docstring: use bold headers + unindented bullet lists (#45) The previous 4-space-indented bullet items under plain-text section headings rendered as code blocks in CommonMark (mkdocstrings/griffe) instead of list items. Replace with bold-header paragraphs followed by top-level bullet lists so the SBO assignment rules display as readable prose rather than raw ``- SBO:...`` code. --- src/raven_toolbox/annotation/sbo.py | 47 ++++++++++++++--------------- 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/src/raven_toolbox/annotation/sbo.py b/src/raven_toolbox/annotation/sbo.py index d0306cb6..44700031 100644 --- a/src/raven_toolbox/annotation/sbo.py +++ b/src/raven_toolbox/annotation/sbo.py @@ -32,30 +32,29 @@ def add_sbo_terms( ) -> cobra.Model: """Assign SBO terms to metabolites and reactions in-place. - Metabolite SBO assignment - - SBO:0000649 (Biomass) for metabolites whose ``name`` is in - ``biomass_met_names`` or ends with any of - ``biomass_met_suffixes`` (default: ``" backbone"`` / ``" chain"``). - - SBO:0000247 (Simple chemical) otherwise. - - Reaction SBO assignment - - SBO:0000176 (Metabolic reaction) default. - - Single-reactant reactions (exchange / sink / demand): - * extracellular → SBO:0000627 (exchange) - * coefficient < 0 → SBO:0000632 (sink) - * else → SBO:0000628 (demand) - - Transport reactions → SBO:0000655 (default detector: same - metabolite name appearing in ≥ 2 compartments). - - The reaction whose ``name`` matches ``biomass_rxn_name`` → - SBO:0000629. - - The reaction whose ``name`` matches ``ngam_rxn_name`` → - SBO:0000630. - - Other reactions whose ``name`` contains any - ``pseudoreaction_name_substrings`` → SBO:0000395. - - "fill" semantic: SBO is written to ``annotation['sbo']`` only when - that key is missing or empty — mirrors RAVEN's - ``editMiriam(..., 'fill')`` mode. + **Metabolite SBO assignment** + + - SBO:0000649 (Biomass) for metabolites whose ``name`` is in + ``biomass_met_names`` or ends with any of ``biomass_met_suffixes`` + (default: ``" backbone"`` / ``" chain"``). + - SBO:0000247 (Simple chemical) otherwise. + + **Reaction SBO assignment** + + - SBO:0000176 (Metabolic reaction) default. + - Single-reactant reactions (exchange / sink / demand): + * extracellular → SBO:0000627 (exchange) + * coefficient < 0 → SBO:0000632 (sink) + * else → SBO:0000628 (demand) + - Transport reactions → SBO:0000655 (default detector: same + metabolite name appearing in ≥ 2 compartments). + - The reaction whose ``name`` matches ``biomass_rxn_name`` → SBO:0000629. + - The reaction whose ``name`` matches ``ngam_rxn_name`` → SBO:0000630. + - Other reactions whose ``name`` contains any ``pseudoreaction_name_substrings`` + → SBO:0000395. + + "fill" semantic: SBO is written to ``annotation['sbo']`` only when that key is + missing or empty — mirrors RAVEN's ``editMiriam(..., 'fill')`` mode. Parameters ---------- From 57d0c42dbcdad7cc4f229419c6cc325ad392d1d6 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Fri, 28 Aug 2026 22:39:19 +0200 Subject: [PATCH 3/3] Release 0.4.0 (#111) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add a release workflow (GitHub release + PyPI trusted publishing) Builds sdist and wheel from the tag, verifies the built version matches it, takes the release notes from the CHANGELOG section for that version, creates the GitHub release, and publishes to PyPI via OIDC trusted publishing so no token is stored in the repo. Triggers on a v* tag push, plus a workflow_dispatch that takes a tag input so a tag that was already pushed can still be released -- push.tags cannot fire retroactively. * CI: run the release artifact actions on Node 24 actions/upload-artifact@v4 and actions/download-artifact@v4 target Node.js 20, so every release run is annotated with the Node 20 deprecation warning. The rest of the workflows moved to Node 24 action majors in #35 (checkout@v5, setup-python@v6); this brings the release workflow in line. The two actions' majors are offset by one, so they cannot both take the same number. v5 (upload) and v6 (download) only added preliminary Node 24 support and still default to Node 20, which would leave the warning in place; upload v6 and download v7 are the first to set 'runs.using: node24'. This takes the latest matched pair -- upload v7 with download v8 -- since upload v7's direct uploads are what download v8's direct downloads consume. Behaviour notes: * download-artifact v8 defaults 'digest-mismatch' to error rather than warn, so a corrupted download now fails the run instead of publishing silently. * Both require Actions Runner >= 2.327.1; the workflow runs on GitHub-hosted ubuntu-latest. * The multi-path upload (dist/ plus notes.md) is unaffected: the paths still resolve through their least common ancestor to the workspace root, so both downstream jobs see 'dist/*' beside 'notes.md'. v7's single-file restriction applies only with 'archive: false'. * docs: note diffModels grRule comparison to back-port into raven-toolbox MATLAB RAVEN's diffModels compares grRules as logic (DNF + sort, so "a and b" == "b and a"), while raven-toolbox's diff_models._normalise_gpr only lowercases and collapses whitespace. Record the AST-based comparison as a pending back-port into raven-toolbox. * fix: make the placement master deterministic across runs The score-maximising placement MILP iterated genes_in_scope, a set, so string hash randomisation reordered its variables and constraints between processes and a degenerate objective returned a different co-optimal placement on each run. Iterate the genes sorted, and pin the Gurobi solve (single thread, fixed seed, presolve level and zero MIP gap) so the same placement is returned every time. A parameter sweep on the yeast master showed thread count, seed and presolve level are the settings that change which co-optimal placement is chosen. * KEGG FASTA/HMM path: align model output with MATLAB RAVEN Bring the protein-FASTA reconstruction (get_kegg_model_from_sequences) into byte-parity with RAVEN develop3 getKEGGModelForOrganism: - Prune each gene-backed reaction's kegg.orthology annotation to the KOs that actually matched a gene, via a new prune_orthology flag on assemble_model_from_ko_genes (off by default, so the organism-annotation path keeps the full KO list). The order-preserving intersection also avoids the rxnMiriams index misalignment in the MATLAB pruning. - Default model.id to the FASTA stem when no model_id is given, so the draft never inherits the reference model's id. - Use a neutral reaction note ("Included by KEGG HMM reconstruction"). Note: the hmmsearch -Z parity item is handled on the MATLAB side (Python's -Z matches the hmmscan scale the K15 cut-offs were calibrated against). * Compare grRules as logic in diff_models (order-insensitive) Implements the back-port this branch previously only documented. diff_models' GPR check DNF-expands each rule via the existing manipulation.gpr_to_dnf, sorts genes within each isozyme clause and sorts the clauses, so operand order no longer registers as a difference: "a and b" == "b and a", "a or b" == "b or a". The previous _normalise_gpr only lowercased and collapsed whitespace, flagging logically identical rules that differed only in operand order. Uses the reaction's already-parsed cobra .gpr (no re-parse) and reports the original rule strings in the diff message while comparing canonically. A rule cobra cannot parse falls back to the old string comparison, so malformed rules are still compared rather than silently equated to empty. Brings diff_models in line with MATLAB RAVEN's diffModels (RAVEN #686); removes the now-resolved pending-back-port note. * Fix load_delta_g_csv stamping the ΔG missing-value sentinel as a real value (#73) * Fix load_delta_g_csv stamping the SEED "missing" sentinel as a real value The ModelSEED-derived side-car tables encode "no valid dG" as the magic value 10000000. load_delta_g_csv -- written for exactly these files, down to its Var1/Var2 column defaults -- had no notion of it and stamped it verbatim, so 777 of yeast-GEM's 4102 reactions (19.5%) carried a physically impossible 1e7 kJ/mol presented as a measurement. Anything reading notes["deltaG"] got garbage for a fifth of the model. yeast-GEM's own checkrxnDirection.m gates on the same value: if ~isequal(seed_rxnInfo{rxnIdx4(i),16},'10000000.0') %check if database contains valid deltaG value Treat the sentinel as missing, exactly as NaN already was, recognising it whichever dtype the MATLAB/pandas round-trip produces (10000000, 10000000.0, "10000000.0"). The new keyword-only missing_value (default SEED_DELTA_G_MISSING) tunes or disables it. Real dG coverage of yeast-GEM is 78.2% (3207/4102), not the 97.1% the loader previously implied. * Close the confidence facet set and rewrite the remaining plan The facet set is localization + equation + gene_association, and no further facet is planned. Drop the planned fourth facet from the study doc and from confidence.py's module docstring, which both still promised one. Replace the phasing section, whose P1/P2 entries only restated what had already shipped, with what is actually left: * Wire the facets together -- an annotate_confidence() umbrella, and let curation_priority read the record so a mark_curated reaction stops resurfacing in the review queue. That is the change that closes the score -> review -> curate loop. * Validate beyond one model. Every number in the doc is yeast-GEM's, and three bands never fire there. * Standards alignment for the paper (Thiele-Palsson / ECO); the SBO half is already done. * delta_g: drop unfounded ModelSEED provenance from the missing-ΔG sentinel The 10000000 "no valid ΔG" sentinel handled by load_delta_g_csv was described as ModelSEED's and the constant named SEED_DELTA_G_MISSING, but the ΔG side-car tables (e.g. yeast-GEM's model_rxnDeltaG.csv) are not ModelSEED-derived — the shared sentinel value does not establish that provenance. Describe it neutrally as the tables' own missing-value marker (still evidenced by yeast-GEM's checkrxnDirection.m gating on 10000000.0) and rename the constant to DELTA_G_MISSING. No behaviour change. * confidence: wire the facets together (#78) * confidence: Thiele-Palsson score mapping + verified ECO ids (§10.3) (#80) Add thiele_palsson_score(reaction): the Thiele & Palsson reconstruction confidence score (0-4) derived from the gene_association facet's basis (gpr+literature -> 3, gpr -> 2, no-gpr -> 1), the facet that captures reaction-inclusion evidence. Grounded in the correspondences already asserted in score_gene_association_confidence. Respects the study doc's two cautions: the map runs forward only (basis -> score), so the ambiguous Thiele-Palsson 2 is never reverse-mapped to an evidence class, and curated is left unmapped (a curator's assertion does not name its evidence class). The ECO evidence-class ids in §9 were each verified against EBI's ECO (OLS) - ECO:0000044 sequence similarity, ECO:0000015 mutant phenotype, ECO:0000002 direct assay - and the assertion-method term ECO:0000305 is explicitly excluded as it is not an evidence class. Docs: §9 gains the concrete basis -> score -> ECO table; §10.3 marked mapping-shipped. Tests for the mapping and the end-to-end basis path. * confidence: cross-model validation of the facet bands (§10.2) (#81) Run measure_confidence_facets.py over three further models beyond yeast-GEM (Human-GEM, iYali, panAsp/pAo) and record the result: - Two of the three bands §6 could not exercise DO fire on other real models -- formula-generic on all three, formula-unparseable on both drafts -- so they are calibrated, not yeast artefacts. Only charge-unknown stays fixture-only across all four (a mass-balanced reaction with a charge-less metabolite is genuinely rare). - gpr+literature fires only where reactions carry a pubmed annotation. - The gene-rubric-vs-recorded-Confidence-Level check needs a per-reaction-curated model: iYali and pAo record a blanket value, so they are not graded ground truth the way yeast-GEM is. Also generalise the script to load SBML or cobra YAML/JSON by extension (Human-GEM ships as YAML). Docs: §6 caveat resolved with a pointer to §10.2; §10.2 written up and marked done. -W docs build clean. * Deterministic score-aligned reaction placement in the assignment master The placement master maximises a gene-localisation objective that never mentions the per-reaction placement variable, so each reaction's compartment was a free co-optimum: the pinned solver returned it reproducibly but arbitrarily, giving 52.8% reaction agreement with curated yeast-GEM (an earlier un-pinned build happened to land ~72%). Add a lexicographic second pass: fix the gene layout to the primary optimum (fixing the y solution, not the objective value, so there is no tolerance to tune), then place each reaction in the compartment its own enzymes are predicted to occupy -- the summed DeepLoc score of the reaction's genes, with a small default_compartment prior so genes-free and score-tied reactions fall there deterministically. Yeast reaction agreement rises to 72.5% (1408/1943) and now rests on the localisation evidence rather than a solver tie-break; gene agreement is unchanged (88.7%, 716/807 -- the gene layout is untouched); coherent placement adds fewer transports (1001 -> 967); growth and blocked fraction unchanged; the warm-started second solve adds ~4s. Reproducible across independent runs. Docs updated (yeast_validation, localization_redesign, multiorganism_validation footnote); the multi-organism reaction-agreement rows await re-measurement. * Refresh multi-organism validation on the deterministic placement (stacks on #83) (#84) * Deterministic score-aligned reaction placement in the assignment master The placement master maximises a gene-localisation objective that never mentions the per-reaction placement variable, so each reaction's compartment was a free co-optimum: the pinned solver returned it reproducibly but arbitrarily, giving 52.8% reaction agreement with curated yeast-GEM (an earlier un-pinned build happened to land ~72%). Add a lexicographic second pass: fix the gene layout to the primary optimum (fixing the y solution, not the objective value, so there is no tolerance to tune), then place each reaction in the compartment its own enzymes are predicted to occupy -- the summed DeepLoc score of the reaction's genes, with a small default_compartment prior so genes-free and score-tied reactions fall there deterministically. Yeast reaction agreement rises to 72.5% (1408/1943) and now rests on the localisation evidence rather than a solver tie-break; gene agreement is unchanged (88.7%, 716/807 -- the gene layout is untouched); coherent placement adds fewer transports (1001 -> 967); growth and blocked fraction unchanged; the warm-started second solve adds ~4s. Reproducible across independent runs. Docs updated (yeast_validation, localization_redesign, multiorganism_validation footnote); the multi-organism reaction-agreement rows await re-measurement. * Refresh multi-organism validation numbers on the deterministic placement Re-measure Human-GEM, AraCore, and iCre1355 (scripts/benchmark_certified_multiorg.py) on the deterministic score-aligned reaction placement, and refresh the yeast row to match yeast_validation.md. The previous reaction-agreement figures came from the old arbitrary co-optimum placement. Reaction agreement rose or held (Human-GEM 52.8% -> 56.9%, the largest move; AraCore 81.8% -> 82.1%; iCre1355 53.4% -> 54.1%; yeast 72.0% -> 72.5%), and coherent placement needs fewer transports everywhere. Gene agreement is unchanged (the tie-break leaves the gene layout untouched), confirmed per model. One non-obvious shift: iCre1355's blocked fraction rose 19.0% -> 32.9% as tighter, less transport-heavy placement leaves more reactions dead-ended (still certifies). Removes the "await re-measurement" footnote; updates the prose reaction counts to the current Human-GEM version (12854 -> 12877). * assign_compartments gap-fill: reliable flux-based fill, not cobra's MILP (#82) * assign_compartments gap-fill: reliable flux-based fill, not cobra's MILP * gap-fill: run on a copy, batch-add candidates, warn on namespace mismatch Harden the flux-based _gapfill (universal-DB fill in localization.certify): operate on a model copy so the caller's model is never mutated; add candidates in one batch (per-reaction adds are super-linear at scale); return the flux-carrying set sorted, so the result does not depend on which co-optimal vertex the solver picked; and warn when most universal candidates share no metabolite id with the model, so a namespace mismatch is distinguishable from a genuine empty fill instead of returning [] silently. Knockout-recovery 60/60 (exact reaction each time) and 12/12 on realistic incomplete drafts, vs cobra's 45% and 0/12. * feat(ftinit): opt-in deterministic extraction (strict gap + canonical optimum) (#85) Two keyword flags on ftinit() (and the underlying run_ftinit / fill_tasks), both default off, so existing behaviour is unchanged. * strict_gap — one solve per step proven to a fixed absolute gap (0.05, below the 0.1 reaction-score granularity), replacing the relative-gap escalation whose final near-zero-objective run otherwise accepts an arbitrary within-gap incumbent. * canonical — a lexicographic phase 2 over the removable reactions, applied to each extraction step and to the task gap-fill: hold the primary objective at its optimum, minimise the count of kept/added reactions, then their summed id rank. * Both guarded by a solution-availability check: a solve that ends at the time limit with no incumbent raises a clear error (or, for the canonical phase, falls back to the primary optimum) instead of an opaque AttributeError on a null primal. * The extraction seed becomes a module constant (_EXTRACT_SEED, still 1234) so the determinism probe in the study can vary it without editing source. These pin which of many equally data-consistent optima is returned; they are not a fix for gene-essentiality determinism. On Human-GEM/DLD1 they cut the reaction-level seed-swing 125 -> 34 but worsened essential-gene flips 5 -> 19 at a 3-7x build-time cost. Measurements, mechanism and limitations: docs/studies/ftinit_determinism.md. * manipulation: export merge_compartments and copy_to_compartment (#86) Both live in manipulation/compartments.py and have done since the package was renamed, but manipulation/__init__.py never imported the module, so neither name was public API. The package docstring already advertises "compartment merge / copy", and docs/reference/migration.md documents both as ported RAVEN functions (mergeCompartments, copyToComps) with source links, so the omission contradicted both. The tests reached into raven_toolbox.manipulation.compartments directly, which is why nothing caught it. They now import through the package, plus an explicit assertion that both names are in __all__. No other module has the same problem: the remaining non-re-exported public names are genuine internal helpers, and utils.parse_name_comp is documented in IMPROVEMENTS.md as deliberately internal. * convert_to_irreversible: carry annotations, subsystem and notes to _REV (#88) MATLAB's convertToIrrev copies the per-reaction fields (eccodes, rxnMiriams, subSystems, rxnNotes, ...) onto the reverse reaction when it splits a reversible one; the Python port copied only the name and the gene rule. Downstream that silently cost the reverse reaction its EC code: in geckopy, fill_eccodes_from_gem returned '' for every _REV reaction, fuzzy BRENDA matching then found no kcat, and the reverse direction was left unconstrained. GECKO's own unit-test suite expects kcat 10 on the ecTestGEM reverse reactions where geckopy produced 0. Annotations and notes are deep-copied, so editing one direction does not write through to the other. * Parity harness: make agreement with MATLAB RAVEN a test (#87) * fix(io): read RAVEN YAML models that omit an empty section RAVEN MATLAB writes no `genes:` block at all for a model without genes -- its own tutorial/small.yml is such a model. cobra's model_from_dict indexes obj["genes"] directly, so reading one raised KeyError: 'genes' and raven-toolbox could not open a valid file written by the toolbox it is the counterpart of. Every YAML fixture in this repository happened to have genes, so no test covered it. Found by reading RAVEN's own shipped models, which is now a standing check (tests/parity/test_yaml_interop.py). The three entity sections are defaulted to empty lists before handing the data to cobra. A file that omits `metabolites:` while its reactions reference metabolites is still an error, and still fails loudly -- that is a broken file, not a RAVEN one. * test(parity): make agreement with MATLAB RAVEN a test, not a claim The Human-GEM, yeast and multi-organism validations live in study documents. Nothing fails if the two implementations drift apart, so "validated against MATLAB RAVEN" has been a statement about the past rather than a property of the current code. tests/parity/ starts the harness, run with `pytest -m parity` and in its own CI job. Its README states the contract in three tiers, because "identical output" is achievable for some functions and meaningless for others: exact for deterministic transformations, set-level for MILP results that have many optima of equal value, statistical for sampling. Putting a function in the wrong tier gives either a flaky test or a vacuous one. Reference values come from two places: - RAVEN's own artefacts. Its repository ships models written by MATLAB RAVEN, so reading them is a real cross-language check needing no MATLAB. RAVEN is GPL and this package is MIT, so they are read from a checkout via RAVEN_ROOT and never vendored here; the CI job checks RAVEN out separately. This is what found the KeyError fixed in the previous commit -- with that fix reverted, test_reads_every_raven_authored_model fails on small.yml. - Recorded oracles, for behaviour no file can show: how RAVEN grades elemental balance, what it normalises a gene rule to. scripts/parity/generate_oracles.m runs RAVEN once over tests/data/parity/tiny.yml -- authored here, so no GPL material is involved -- and writes JSON. Those tests skip with instructions until someone generates them: a visible skip is honest, a fabricated oracle is not. The comparison logic was verified end to end against throwaway oracles, which were then deleted. test_determinism.py is not cross-language but guards the property the tiers rest on: comparing against MATLAB means nothing if raven-toolbox itself answers differently each run. Placement and gap-filling were made deterministic in #76, #83 and the assignment rework with no regression test. Both tests assert first that the fixture still exercises a real choice, so they cannot quietly become vacuous. * docs: add the development roadmap, and rework the open-work backlog todo.md was a 24-line pointer page. It is now the item-level backlog, and roadmap.md is the plan it serves: four dependency-ordered phases with what each delivers, what it depends on, and how you know it is finished. The phases are driven by what stops a newcomer trusting the port rather than by what is left to write: nothing tells a user how the two toolboxes differ (phase 0/1), nothing enforces the parity claims (phase 2, started in this branch), the documentation site has no Python content at all (phase 3), and no one can tell which functions are mature (phase 4). Decisions taken and recorded: two new dual-language reconstruction tutorials rather than porting the published hanpo-GEM protocol; SLIME lipid curation out of scope while that method work proceeds; no external anchor for 1.0, so phases 3 and 4 may overlap. Open decisions are listed with what they block. * Parity: add the set-level tier with a recorded baseline (#89) * test(parity): add the set-level tier with a recorded baseline PR #87 stated the tier-2 contract and left it without tests. Extraction is a MILP, so there is no single right answer to assert -- but there is a wrong outcome worth catching: the result quietly moving. test_set_level.py runs run_init over RAVEN's smallYeast.yml and compares the kept reactions against a baseline recorded by scripts/parity/record_baseline.py. On failure it prints the Jaccard overlap and both directions of the diff, so a one-reaction shift is immediately distinguishable from a rewrite, and says how to re-record when the move is intended. The assertion is exact set equality rather than an overlap band because that was measured, not assumed: on this fixture GLPK and Gurobi return the same 13 of 53 reactions, and each is identical across three runs. A difference means this package changed, not that the solver picked another optimum. On a fixture where the solvers genuinely disagree the right form is a band with a measured floor -- not a loosened threshold here. The baseline is seeded from raven-toolbox, so this guards against regression rather than proving agreement with MATLAB. The file's `source` field says so and the failure message repeats it; when generate_oracles.m grows an extraction oracle the same comparison becomes a real parity check. A second test asserts the baseline still keeps some but not all reactions, so the check cannot become vacuous if the fixture or scores change. * test(parity): genome-scale checks and the nightly job that can run them The Human-GEM, yeast and multi-organism validations are the strongest evidence this package agrees with MATLAB RAVEN, and the least protected: measured once, by hand, on models no free runner's solver licence will accept. parity-nightly.yml runs them on a Gurobi Web License Service licence, supplied as GUROBI_WLSACCESSID / GUROBI_WLSSECRET / GUROBI_LICENSEID. WLS is the licence type meant for containers and CI; a named-user academic licence is tied to a machine and does not belong on a hosted runner. The job refuses to be quietly useless. If the secrets are missing it fails with a message rather than skipping, and before any test runs, check_license.py solves a 20,000-variable MIP -- far past the 2000-variable cap of the licence bundled with gurobipy -- so a silent fallback to that licence cannot leave the genome-scale checks skipped under a green tick. test_genome_scale.py compares as a *band*, not exact equality, which is the opposite choice from the small fixture and for a reason: at this size the MILP has many optima of equal value and the solve is bounded (mip_gap, time_limit), so a different optimum is legitimate. It also asserts a sanity floor that holds whichever optimum turns up -- non-empty, not everything, no duplicates -- so the run is checked even before a baseline exists. The band floor will be recorded alongside the run that produces it rather than guessed now. The pull-request job keeps running everything except these (`-m "parity and not slow"`) and now installs gurobipy: its bundled licence is useless at genome scale but ample for the 53-reaction fixture, which gives the cross-solver check a second solver to compare against. * Parity: keep the genome-scale numbers, budget the nightly, and check determinism across processes (#90) * test(parity): keep the genome-scale numbers, and budget the job realistically Ran the genome-scale check locally against the real Human-GEM (12,931 reactions) with a full Gurobi licence. It passed, and exposed two problems with what #89 shipped. **The numbers were thrown away.** pytest captures stdout for a passing test, so the extraction's reaction count and timing -- the whole point of the run, and the candidate for the next baseline -- were invisible in precisely the case that matters. The fixture now writes a JSON report to $RAVEN_PARITY_REPORT_DIR, appends a line to the GitHub job summary, and the workflow uploads the report as an artefact even on failure, since a failure is when the actual reaction set most needs inspecting. **The job budget was wrong.** The run took 68 minutes on a fast workstation against a 90-minute job timeout, which a two-core hosted runner would not have met. Raised to 150, with the reasoning recorded: only ~2 of those 68 minutes are parsing and copying (measured: 94s to read the YAML, 29s to copy the model), and run_init's time_limit does not bound the call -- a 60-second limit still ran for well over fifteen minutes. pytest-timeout is therefore the real stop, and timeout-minutes only has to exceed it so an overrun is reported as a test failure rather than a killed job. That time_limit behaviour is a finding about run_init rather than about the test, and is being measured separately. * test(parity): check determinism across processes, not just within one The existing determinism tests repeat a call in one process, which cannot see the failure mode the placement fixes were actually about: Python randomises string hashing per process, so a set iterated to build constraint rows or a dict deciding a tie-break gives a stable answer within a run and a different one in the next. Repeating in-process, however many times, is blind to it. The new test runs the gap-fill in three subprocesses under PYTHONHASHSEED 0, 1 and 12345 and compares digests of the result -- which is what scripts/determinism_probe.py was doing by hand while those fixes were being made. All three agree today. The worker sets cobra's process count to 1. The gap-filler runs FVA, which otherwise starts a worker pool per invocation; with three workers doing that at once the test took over ten minutes and spawned nineteen processes. Single -process, on a four-reaction model, it takes 53 seconds. Like the other determinism tests, it first asserts the fixture still requires a fill, so it cannot pass by checking nothing. * fix(homology): match RAVEN's BLAST e-value default (#91) * fix(homology): match RAVEN's BLAST e-value default run_blast defaulted to 1e-5 where MATLAB RAVEN's getBlast hardcodes -evalue 10e-5 -- which is 1e-4, ten times looser. The same two proteomes therefore gave the two toolboxes different hit sets, before any of the homology logic ran. Found while building the homology parity scenario in raven-gecko-parity: the BLAST checkpoint had to pass RAVEN's value explicitly for the comparison to mean anything. With the defaults aligned it does not. RAVEN hardcodes the cutoff; here it stays an argument, so a stricter search is still available to anyone who wants one. run_diamond is left alone and documented instead: getDiamond passes no --evalue at all, so DIAMOND's own default of 1e-3 applies on the MATLAB side -- a third value, looser than either. Picking one needs a decision about which is right, not a quiet change. * fix(homology): match RAVEN's DIAMOND e-value default too run_diamond defaulted to 1e-5. getDiamond passes no --evalue at all, so DIAMOND's own default of 1e-3 applies on the MATLAB side -- two hundred times looser. Same reasoning as the BLAST default in the previous commit: MATLAB came first, and nothing measured here says otherwise. The two aligners still disagree with each other (1e-4 for BLAST, 1e-3 for DIAMOND) because RAVEN's own defaults disagree; that is now stated in both docstrings instead of being implicit in an unpassed flag. Worth noting what this is *not*: the KO-assignment cutoffs in assign_kos do diverge from RAVEN deliberately (1e-30 against RAVEN's 1e-50, gene ratio 0.9 against 0.8), and that divergence is backed by the measurements in docs/studies/kegg_hmm_cutoff_calibration.md. No comparable study exists for the homology e-value on either side, so there is nothing to weigh against following MATLAB here. * Homology cut-offs: measured, and min_align_len lowered to 100 (#92) * docs: protocol for calibrating the homology cut-offs RAVEN and raven-toolbox share three thresholds that decide which template reactions transfer to a new organism -- max_evalue 1e-30, min_align_len 200, min_identity 40 -- and neither records where they came from. This is the design for measuring them, written before any numbers exist so the success criterion cannot be chosen to fit the curves. It retargets the question. The aligner's own cut-off is inert: run_blast and getBlast collect at 1e-4, then get_model_from_homology filters again at 1e-30, twenty-six orders stricter, so every hit that survives model building would have survived collection. Tuning the collection threshold changes nothing; the levers are the three above plus bidirectionality. Ground truth comes from two sources whose weaknesses do not overlap. KEGG cross-organism models share a reaction namespace, so the comparison needs no id mapping and scales to a distance series -- but KEGG's own orthology is built on all-against-all BLAST, so it measures agreement with KEGG's homology calls, not correctness, and the protocol says so where the numbers will be read. Curated GEM pairs (yeast-GEM against hanpo-GEM and rhto-GEM) supply the non-circular check. Measurement is at hit level first -- do the two genes share a KO -- because reaction-level agreement is buffered: a reaction transfers if any one of its template genes hits, so the sweep would look flat for reasons unrelated to the parameter. The sweep is cheap: align once per organism pair, cache the hit table, and every threshold combination is post-processing of it. The objective is fixed in the protocol, the very-distant pairs act as a constraint rather than a target so recall cannot be maximised until everything transfers, and "the current values are as good as anything measured" is named as an acceptable outcome. * docs: run the homology cut-off study -- defaults unchanged The measurements are in. Under the criterion fixed before any data existed, no change to max_evalue / min_align_len / min_identity is justified, and they stay as they are. Three findings do not depend on that criterion. max_evalue does nothing. From 1e-4 to 1e-50 the extracted model is identical -- on both ground truths, on every organism. Only 1e-100 changes anything, because identity and alignment length have already excluded whatever a looser e-value would admit. It is not a tuning knob and calibration effort spent on it is wasted. min_identity is the only real lever, and its best value moves with phylogenetic distance: ~30 for a close relative, ~25 for medium and distant. The default of 40 costs 1 point of F1 on Kluyveromyces and 10 on Aspergillus. No single global default is right for every reconstruction, which the documentation should say. min_align_len is nearly inert below 150; the default 200 costs about 2 points. The curated-GEM arm had to be retired, and the study can show why rather than just suspect it. hanpo-GEM's draft was built by getModelFromHomology at (1e-30, 150, 35), and the sweep's optimum lands exactly there. The discriminating test: walk the thresholds loose one step at a time and ask what fraction of newly admitted reactions are in the curated model. The rate holds between 0.62 and 0.85 up to the build settings and collapses to 0.06 one step past them. That cliff is the reference remembering its own construction, so any optimisation against it returns the build settings whether or not they are good. Most curated non-model fungal GEMs are RAVEN drafts from yeast-GEM, so this applies to the class, not just this model. The KEGG arm ran because UniProt carries KEGG cross-references, which solved the gene-id problem that the missing (subscription-only) KEGG proteomes had created. What is left open is a value judgement rather than a measurement: the criterion blocked a change that improves F1 on all four organisms, including the constraint organism, because it counted calls rather than correctness. Rewriting it after seeing the curves is what fixing it in advance was meant to prevent, so it stands, and the real question is stated instead -- what a wrongly transferred reaction costs relative to a missed one. Gap-filling can recover the second; the first pollutes the model silently. Answering that gives a loss function, and the cached hit tables can be re-scored in minutes. Both drivers are committed; the KEGG one reproduces every published figure from the cached tables. * docs: rescore the homology study at beta = 0.5 -- defaults confirmed The loss function was the one thing the study deliberately left open, since it is a value judgement rather than a measurement. It is now decided on the asymmetry it turns on: a wrongly transferred reaction is worse than a missed one, because gap-filling can recover the second while the first pollutes the model and its gene associations silently. So precision is weighted above recall, beta = 0.5. Rescored, the conclusion reverses in the most useful direction: min_identity 40 is optimal on the close and very-distant organisms outright and within 0.01 of the best on the other two. The 10-point deficit at distance that the F1 scoring showed was entirely an artefact of counting a missed reaction as costly as a wrong one. The default was right. One candidate change misses narrowly and is recorded rather than adopted: min_align_len 50 improves all four organisms (+0.009 to +0.020) at essentially unchanged precision, but eco calls rise 137 -> 152, violating the constraint, and the improvement of 0.011 is smaller than the 0.056 within-band spread the margin rule compares against. Both gates are doing debatable work at this scale -- comparing an improvement against the spread between organisms of different difficulty is not a noise estimate -- so the evidence is stated and the decision left to a maintainer rather than taken here. The driver takes --beta, so the same cached hit tables can be re-scored under a different weighting in a couple of minutes without realigning anything. That the same measurements recommend an identity of 25 or 40 depending on beta alone is the study's most transferable finding: a threshold recommendation without a stated weighting is not a recommendation. * feat(homology): lower min_align_len to 100, measured against KEGG orthology The three filters that decide which template reactions transfer had never been measured in either toolbox. Scored against KEGG's own orthology across a distance series -- close yeast, medium, distant fungus, bacterium -- with precision weighted above recall, because gap-filling can recover a missing reaction while a wrongly transferred one is hard to find and harder to remove. min_align_len 200 -> 100. Everything at or below 150 measures the same; the loss appears between 150 and 200. Dropping to 100 recovers 3-4 points of recall on every organism tested while precision moves by at most 0.006 -- real orthologs that 200 was discarding for nothing. 50 and 100 are identical to three decimals, so 100 is taken as the less permissive of two equivalent values. min_identity 40 is confirmed: it wins outright on the closest and most distant organisms and trails by under 0.01 on the two between. It is also the only filter that materially decides anything. max_evalue is inert. Any value from 1e-4 to 1e-50 yields the identical model, because identity and length have already excluded whatever it would exclude. It stays at 1e-30 for continuity with RAVEN, and the docstring now says it is not worth tuning. Worth recording: at beta = 1 the same measurements recommend identity 25 and call the default a 10-point deficit at distance. At beta = 0.5 they confirm 40. A cut-off recommendation without a stated weighting is not a recommendation. min_align_len now diverges from RAVEN's 200, so a back-port is worth proposing rather than leaving the toolboxes quietly different. * feat(homology): report the near misses instead of discarding them The filters are strict on purpose: a wrongly transferred reaction is hard to spot and harder to remove, while a missing one can be gap-filled. That argument justifies rejecting borderline hits. It does not justify throwing away the evidence for them, which is what happened -- a reaction that missed the identity threshold by half a percent left no trace. get_model_from_homology now takes review_identity (e.g. 25). Reactions that would have transferred at that looser identity are returned in HomologyResult.candidates with the hit supporting each, sorted strongest first, and are *not* added to the model. On a real reconstruction (yeast-GEM to H. polymorpha) that is 303 candidates beside a 2,078-reaction draft: too many to accept blindly, few enough to read. The reported identity is the *limiting* one. Matching is bidirectional, so a pair has to clear the threshold both ways; the first version of this reported the better direction and the top candidates came out at 44 % against a threshold of 40, which would have left a curator wondering why a comfortable match was turned away. Now they come out at 39.7 %, 39.6 %, 39.5 % -- the near misses they actually are. Covered by a regression test. * docs: add the OMA and DIAMOND results, and trim the study to its findings OMA infers orthologs without using BLAST, so it answers the obvious objection to the KEGG numbers: that they partly measure agreement with the method being tested. Both references put the best identity between 35 and 45 and neither supports anything looser, so 40 stands on two independent sources rather than one. DIAMOND reaches the same optimum as BLAST on every organism, within 0.006. It finds about half as many matches, but nearly all the missing ones are weak ones these settings discard anyway -- after filtering the two agree on 87-92% of what survives, while DIAMOND runs 10-20x faster. That matters because both toolboxes apply the same three settings to whichever aligner ran, which until now was an assumption. The driver grew --aligner and --reference so all three combinations reproduce from the cached alignments; verified against every published figure. The write-up is now about what was found rather than how it was set up, and is roughly half the length. * Fix the nightly's Gurobi licence lookup parity-nightly.yml had failed on its licence check every night since it was added: it read GUROBI_WLSACCESSID / GUROBI_WLSSECRET / GUROBI_LICENSEID, but only the first exists. The licence the organisation shares is GUROBI_EDUARD, holding a licence file verbatim. - Accept either shape, and report which kind was found without printing it. A named-user licence cannot authenticate on a hosted runner. - Still fails loudly when neither is configured. - Add a test that parses every workflow run: block with bash -n; nothing else checks a scheduled workflows shell until the night it runs. * docs: give evidence_aware_transport_cost a Returns section (#93) The closing paragraph of the docstring ("Returns every metabolite base -> cost ...") sat directly against the Parameters list at the same indentation as a parameter name, so the numpydoc parser read it as two further parameters, named "Returns" and "is". Both then rendered as parameter rows on the API page, and the generator warned four times: No types or annotations for parameters ['Returns'] Parameter 'Returns' does not appear in the function signature Move that paragraph into a proper Returns section, and document `model`, `annotation` and `base_metabolite`, which the Parameters list had left out -- six of the nine arguments were described. The `name:` parameter style is unchanged: it is what the rest of the package uses, and it resolves types from the signature without complaint. Documentation only; no functional change. * Make every executable in a downloaded bundle executable zipfile does not restore Unix permission bits, so extracted files arrive non-executable on Linux and macOS. ensure_binary marked only the executable it was asked for, and returned a cached bundle without marking anything at all. A bundle can provide several tools: BLAST ships blastp and makeblastdb. Fetching blastp then calling makeblastdb raised PermissionError, which is what broke the parity nightly. Affects any Linux or macOS user, not only CI. * io.yaml: adopt a shared RAVEN/raven-toolbox YAML layout (#97) * io.yaml: adopt a shared RAVEN/raven-toolbox YAML layout The two writers previously disagreed on line folding, number formatting and how an empty subsystem was represented, which made yeast-GEM / Human-GEM churn on every write regardless of which implementation touched them last. - write_yaml_model now dumps through its own ruamel instance with a very large width instead of cobra's shared module-level one, so long scalars are never folded onto a continuation line. Mutating the shared instance's width would also change cobra.io.save_yaml_model for the rest of the process. - Every numeric leaf is coerced to float before dumping, so a whole number always gets an explicit ".0" (matching RAVEN's writer, which cannot distinguish an int-typed field from a float-typed one since its model struct stores everything as double). - A reaction's subsystem is omitted entirely when empty, like every other optional field, instead of round-tripping cobra's own round-trip preservation of older RAVEN files' "blank list entry" convention. A non-empty subsystem is now always written as a list, even a single one, matching writeYAMLmodel.m and the reader's existing list-form support (a reaction can carry more than one). Indentation and quoting were already ruamel's own defaults and needed no change. Two tests asserted the previous round-trip shape for a plain-string subsystem input; updated to expect the new single-item list, which is what the format now guarantees regardless of how the in-memory Reaction object represented it. Verified byte-identical against RAVEN's writeYAMLmodel.m on both a plain model and one exercising every optional field. Full test suite: 874 passed, 0 failed. * io.yaml: trim comments to statements, not arguments Comments were re-litigating why the layout diverges from cobrapy's rather than just describing what it is; cut down to the latter. * Default a new metabolite's name to its id in add_reactions_from_equations (#98) The mets_by="id" path created a metabolite via Metabolite(token, compartment=compartment), leaving name empty, where addRxns on the MATLAB side names it after its id. A nameless metabolite is invisible to anything that keys on name -- merge_compartments (whose own docstring tells callers to pass base_metabolite=lambda m: m.name for name-keyed models), add_reactions_from_model, and mergeCompartments on the MATLAB side. * Teach apply_condition's reset_exchanges the direction RAVEN reads it as (#99) cobrapy has no concept of RAVEN's in/out exchange directions, so any truthy reset_exchanges value reset every exchange reaction regardless of what was actually named. applyCondition forwards the value straight through to getExchangeRxns as a direction filter; a condition naming "out" specifically -- exactly the value both docstrings use as their own worked example -- silently reopened uptake reactions it never asked to touch. _exchange_direction replicates getExchangeRxns' own in/out rule (nothing produced at all -> boundary met is implicitly the product, "out"; nothing consumed -> "in"), not restricted to single-metabolite reactions the way cobra's own Reaction.boundary is. "all"/"both", or any other truthy value that isn't a recognised direction keyword, falls back to resetting every exchange in either direction -- preserving prior behaviour for a bare `true`. Verified against the exact scenario reproducer: reset_exchanges: "out" on smallYeast now leaves glcIN/o2IN/ethIN untouched and reports growth=0.0, matching RAVEN exactly (apply_condition_smallyeast MATCHes end to end). * Default check_tasks/find_task_essential_reactions to RAVEN's additive boundary semantics (#100) close_boundaries defaulted to True, closing every exchange/sink/demand reaction before applying a task's constraints -- the opposite of RAVEN's own checkTasks, which relaxes metabolite balance for a task's declared inputs/outputs and leaves the model's own reactions, including open exchanges, untouched. A task the model can already satisfy through its own boundary without the task naming it as an input/output passed in RAVEN and failed here. Confirmed on smallYeast: GLC_AER_TIGHT (declares only CO2 as output; the model ships ethOUT/glyOUT/acOUT open) now agrees with RAVEN's feasible verdict instead of reporting infeasible. close_boundaries=True remains available for the stricter reading, where a task's declared inputs/outputs are the complete boundary of the system for that check. * Hide RXNS bound cells matching the model's declared default in export_to_excel (#101) export_to_excel always wrote LOWER BOUND / UPPER BOUND literally; exportToExcelFormat leaves the cell blank when the bound equals the model's own declared default (an irreversible reaction's lower bound is hidden separately whenever it is exactly 0, regardless of the declared default). Not a corner case: on smallYeast, whose bounds are drawn entirely from {-1000, 0, 1000} (its own declared defaults plus zero), nearly every reaction's bound cells were affected. _rxn_bound_cells reads the same defaultLB/defaultUB metadata.notes entries the MODEL sheet already reads, and replicates exportToExcelFormat's hiding rule exactly. Verified against the exact scenario reproducer: PGI/FBA/HXK now come back fully blank and glcIN/o2IN blank-lb/literal-ub, matching RAVEN's output cell for cell (export_to_excel_smallyeast MATCHes end to end). * Make remove_duplicate_reactions' round trip with expand_model lossless (#103) Matches RAVEN's contractModel, which this was a lossy counterpart to in three ways: the survivor was the last-encountered reaction rather than the first; its id kept an "_EXP_N" suffix rather than having it stripped; and only the survivor's own GPR was kept rather than the union of every duplicate's. The isozyme relationship contractModel restores by design -- three reactions that expand_model split out of one, each carrying one of the original's isozyme genes -- was previously lost on the Python side: two of the three genes came out unassociated with the reaction. makeEcModel runs this exact expand-then-contract round trip, so the gap reached the gecko pair too. _top_level_or_clauses walks cobra's own GPR AST rather than reimplementing RAVEN's bracket-matching string search; the union is deduplicated and AND-clauses are parenthesised for readability. Verified against the exact reproducer: expanding smallYeast's HXK (an isozyme of three genes) and contracting back now returns HXK (not HXK_EXP_3) with all three genes restored, matching RAVEN's own documented output for this case exactly. * Default load_delta_g_csv to stamping every value literally, matching deltaGCSV (#102) * Default load_delta_g_csv to stamping every value literally, matching deltaGCSV missing_value defaulted to DELTA_G_MISSING (1e7), so a matched CSV value equal to yeast-GEM's own "no measurement" sentinel was silently left unstamped instead of recorded. RAVEN's deltaGCSV has no sentinel concept at all and stores whatever the CSV says, verbatim -- applying the identical CSV to the identical model left the two implementations disagreeing about whether an entity had a value on record at all. Both sides are now symmetric: literal by default, with an explicit, opt-in way to treat a chosen value as missing instead (deltaGCSV's new missingValue argument; here, passing DELTA_G_MISSING explicitly rather than relying on the default). * test_annotation: opt these two into the sentinel default explicitly load_delta_g_csv's missing_value default flipped to None (stamp verbatim); these two tests were exercising the old skip-the-sentinel default and need to request it explicitly now. * fix: merge_compartments kept neither exchanges nor the objective (#96) * fix: merge_compartments kept neither exchanges nor the objective Two defects, both silent, both turning a working model into one that cannot grow. 1. Every reaction left with a single metabolite after merging was deleted, including the ones that had a single metabolite to begin with -- the exchanges. On the small yeast model used by the documentation site that removed all eight boundary reactions plus biomassOUT: 11 reactions dropped where only 3 were genuine transports. Only reactions that *become* trivial through the merge are dropped now, which is what RAVEN does (it records the single-metabolite reactions before merging and excludes them from deletion). 2. The merged model is rebuilt from scratch and the objective was not carried over, so the result optimised to 0.0 with nothing to indicate why. It is copied now, with the direction, and a warning is raised in the case where every objective-carrying reaction really was removed. Together these took that model from growth 0.1222 to 0.0000 with no error and no warning; it now flattens to 0.1268, slightly higher, as expected once compartmentalisation no longer constrains it. The existing tests used a toy model with no exchange reactions and no objective, so neither path was covered. Adds a test for each. * compartments: fix import order (ruff I001) * io/yaml: pin the writer's block-sequence indent explicitly (#105) Everything about the writer's list layout already matched RAVEN's writeYAMLmodel.m (each list entry two spaces past its parent key), but only because ruamel's own indent defaults happened to produce that. Set them explicitly so a future ruamel version can't silently change the on-disk layout out from under either writer. Verified byte-identical against the previous (implicit) output. * manipulation: add set_exchange_bounds, RAVEN's setExchangeBounds port (#104) cobra's model.medium covers the common single-direction, close-the- rest case, but not everything setExchangeBounds does: independent lb and ub per metabolite (medium sets only whichever bound is "active" for a reaction, from one scalar), a media-only compartment filter, and a check that every exchange reaction agrees on which sign of flux is import before touching any of them (closing "import" on the rest needs a consistent answer to that, or it closes the wrong bound). Matches metabolites against the model's exchanged set by id or name (case-insensitive), reports ones that weren't found, and warns (same as RAVEN) when a metabolite is exchanged in more than one reaction -- RAVEN's own version of that specific warning has a latent bug (an index-range mismatch between two differently-sized arrays means it never actually fires); this one does. Cross-validated directly against setExchangeBounds.m on smallYeast: explicit bounds plus close_others under RAVEN's own mixed-direction warning path, unused-metabolite reporting, the no-metabolites-given default, and the media_only error path (smallYeast has no extracellular compartment) -- all four cases match bound-for-bound. * io/yaml: match writeYAMLmodel.m's metaData order and MIRIAM collapsing (#106) * io/yaml: match writeYAMLmodel.m's metaData order and MIRIAM collapsing Closes three concrete gaps found while byte-diffing MATLAB and Python output for the same model: - metaData now follows writeYAMLmodel.m's fixed field order (id, name, version, date, then the annotation-style fields), with the same defaults: id/name fall back to "blankID"/"blankName", date falls back to today. Previously the block just echoed whatever order model.notes['metaData'] happened to be in. - defaultLB/defaultUB are now recomputed from the model's current bounds and included, matching readYAMLmodel.m's own min(model.lb)/max(model.ub) derivation — previously never emitted at all unless already present in the source. - A singleton annotation value (e.g. one kegg.compound id) now collapses to a bare scalar, matching writeYAMLmodel.m's MIRIAM handling — except ec-code/smiles, which RAVEN's own writer always keeps as a list. The reader gained the matching read-side normalisation (wrap a bare scalar into a one-item list before model_from_dict), since cobra doesn't do that itself and a collapsed-then-reread value would otherwise come back as a bare string instead of list[str]. A fourth candidate (defaulting an unset metabolite charge to 0.0) was investigated and deliberately not replicated: readYAMLmodel.m's actual behaviour there is a position-dependent side effect of generic fill-missing-fields code (a mid-list gap parses as NaN and is omitted; only a gap at the very end of the metabolite list happens to hit a different tail-padding branch and become a real 0), not a "charge defaults to neutral" convention worth reproducing. Verified against real RAVEN-authored files: reading and rewriting tutorial/smallYeast.yml through this writer and through RAVEN MATLAB's writeYAMLmodel.m now produces byte-identical output (same SHA256). * io/yaml: default a metabolite's missing compartment to the first one Matches readYAMLmodel.m's own "assume first compartment" convention (just fixed on the MATLAB side for a position-dependent bug in SysBioChalmers/RAVEN#709) for a metabolite with no explicit compartment: cobra's model_from_dict has no equivalent default and would otherwise leave met.compartment as None. Verified against the same probe file used to confirm the MATLAB fix: both readers now resolve it to the identical compartment assignment. * io/yaml: quote with double quotes, matching Prettier's default (#107) write_yaml_model relied on ruamel's own default (single-quote a scalar when quoting is required), which was never a deliberate choice, just what the writer happened to inherit. Switched to double quotes to match Prettier's YAML default instead, since byte-parity with a particular tool's default isn't the actual constraint here, only staying importable across RAVEN and raven-toolbox is. ruamel's round-trip dumper has no global "prefer this quote character" switch, only a per-value style override, so this needed its own needs_quote check (ported from writeYAMLmodel.m's, so both writers quote exactly the same set of values) and a recursive pass wrapping exactly those strings in DoubleQuotedScalarString before dumping -- wrapping unconditionally would have quoted every scalar regardless of whether YAML requires it. No reader change needed: read_yaml_model goes through ruamel's real parser, which already handles either quote style and its escaping correctly. * Prepare 0.4.0 release (#109) Bump version 0.3.0 -> 0.4.0 and complete the CHANGELOG 0.4.0 section: six MATLAB RAVEN parity fixes found by the new raven-gecko-parity harness, the YAML writer brought to byte-parity with writeYAMLmodel.m, BLAST/homology defaults matched to RAVEN, deterministic compartment placement, confidence-score interoperability (Thiele-Palsson, ECO), a new set_exchange_bounds port, and merge_compartments made public and fixed. --- .github/workflows/ci.yml | 36 ++ .github/workflows/parity-nightly.yml | 142 +++++++ .github/workflows/release.yml | 119 ++++++ CHANGELOG.md | 161 ++++++++ docs/guide/context_specific.md | 8 + docs/reference/index.md | 5 + docs/reference/roadmap.md | 200 ++++++++++ docs/reference/todo.md | 296 ++++++++++++++- docs/studies/confidence_tracking.md | 110 ++++-- docs/studies/ftinit_determinism.md | 142 +++++++ docs/studies/homology_cutoff_calibration.md | 136 +++++++ docs/studies/index.md | 9 + docs/studies/localization_redesign.md | 7 + docs/studies/multiorganism_validation.md | 28 +- docs/studies/yeast_validation.md | 16 +- pyproject.toml | 6 +- scripts/homology_cutoff_kegg.py | 345 ++++++++++++++++++ scripts/measure_confidence_facets.py | 12 +- scripts/parity/check_license.py | 42 +++ scripts/parity/generate_oracles.m | 129 +++++++ scripts/parity/record_baseline.py | 112 ++++++ src/raven_toolbox/annotation/delta_g.py | 37 +- src/raven_toolbox/binaries.py | 26 +- src/raven_toolbox/comparison/diff.py | 43 ++- src/raven_toolbox/conditions/apply.py | 50 ++- src/raven_toolbox/confidence.py | 83 ++++- src/raven_toolbox/init/ftinit.py | 184 +++++++++- src/raven_toolbox/init/taskfill.py | 66 +++- src/raven_toolbox/io/excel.py | 29 +- src/raven_toolbox/io/yaml.py | 320 +++++++++++++++- src/raven_toolbox/localization/certify.py | 162 ++++++-- src/raven_toolbox/localization/curation.py | 22 ++ .../localization/transport_evidence.py | 18 +- src/raven_toolbox/manipulation/__init__.py | 6 +- src/raven_toolbox/manipulation/add.py | 5 +- .../manipulation/compartments.py | 38 +- .../manipulation/irreversible.py | 14 +- src/raven_toolbox/manipulation/parameters.py | 159 +++++++- src/raven_toolbox/manipulation/simplify.py | 64 +++- .../reconstruction/homology/blast.py | 15 +- .../reconstruction/homology/homology.py | 145 +++++++- .../reconstruction/kegg/assemble.py | 26 +- .../reconstruction/kegg/query.py | 8 +- src/raven_toolbox/tasks/check.py | 34 +- .../parity/baselines/init_smallyeast.json | 28 ++ tests/data/parity/tiny.yml | 90 +++++ tests/parity/README.md | 130 +++++++ tests/parity/_hashseed_worker.py | 85 +++++ tests/parity/conftest.py | 76 ++++ tests/parity/test_determinism.py | 178 +++++++++ tests/parity/test_genome_scale.py | 160 ++++++++ tests/parity/test_oracles.py | 102 ++++++ tests/parity/test_set_level.py | 147 ++++++++ tests/parity/test_yaml_interop.py | 112 ++++++ tests/test_annotation.py | 38 ++ tests/test_binaries.py | 52 +++ tests/test_comparison_diff.py | 51 ++- tests/test_confidence.py | 66 ++++ tests/test_init_ftinit.py | 47 +++ tests/test_init_pipeline.py | 23 ++ tests/test_init_taskfill.py | 26 ++ tests/test_io_yaml.py | 6 +- tests/test_io_yaml_parity.py | 6 +- tests/test_io_yaml_raven_sections.py | 101 +++++ tests/test_localization_certify.py | 43 +++ tests/test_localization_curation.py | 20 + tests/test_manipulation_compartments.py | 48 ++- tests/test_manipulation_irreversible.py | 35 ++ tests/test_parameters.py | 97 ++++- ...test_reconstruction_homology_candidates.py | 138 +++++++ tests/test_reconstruction_kegg_query.py | 46 ++- tests/test_workflows.py | 102 ++++++ 72 files changed, 5459 insertions(+), 209 deletions(-) create mode 100644 .github/workflows/parity-nightly.yml create mode 100644 .github/workflows/release.yml create mode 100644 docs/reference/roadmap.md create mode 100644 docs/studies/ftinit_determinism.md create mode 100644 docs/studies/homology_cutoff_calibration.md create mode 100644 scripts/homology_cutoff_kegg.py create mode 100644 scripts/parity/check_license.py create mode 100644 scripts/parity/generate_oracles.m create mode 100644 scripts/parity/record_baseline.py create mode 100644 tests/data/parity/baselines/init_smallyeast.json create mode 100644 tests/data/parity/tiny.yml create mode 100644 tests/parity/README.md create mode 100644 tests/parity/_hashseed_worker.py create mode 100644 tests/parity/conftest.py create mode 100644 tests/parity/test_determinism.py create mode 100644 tests/parity/test_genome_scale.py create mode 100644 tests/parity/test_oracles.py create mode 100644 tests/parity/test_set_level.py create mode 100644 tests/parity/test_yaml_interop.py create mode 100644 tests/test_io_yaml_raven_sections.py create mode 100644 tests/test_reconstruction_homology_candidates.py create mode 100644 tests/test_workflows.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 421987c9..19a0919b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -78,6 +78,42 @@ jobs: env: PYTHONUNBUFFERED: "1" + parity: + name: parity vs MATLAB RAVEN + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@v5 + # RAVEN is GPL and this package is MIT, so its files are never vendored + # here -- the cross-language checks read them from a checkout instead. + # develop3 is the branch raven-docs documents and raven-toolbox tracks. + - name: Check out MATLAB RAVEN + uses: actions/checkout@v5 + with: + repository: SysBioChalmers/RAVEN + ref: develop3 + path: .raven + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: pyproject.toml + - run: pip install --upgrade pip + # gurobipy ships a size-limited licence (2000 variables / 2000 + # constraints). That is useless at genome scale but ample for the small + # fixtures here, and it gives the cross-solver check a second solver to + # compare against. Genome-scale work runs in parity-nightly.yml, which + # uses a WLS licence. + - run: pip install -e ".[dev]" gurobipy + # -rs reports what skipped: oracle-backed checks skip until someone runs + # scripts/parity/generate_oracles.m in MATLAB, and that should be visible + # in the log rather than looking like coverage that exists. + - name: Run parity checks + run: pytest -m "parity and not slow" -v -rs --timeout=600 --timeout-method=thread + env: + RAVEN_ROOT: ${{ github.workspace }}/.raven + PYTHONUNBUFFERED: "1" + docs: name: docs runs-on: ubuntu-latest diff --git a/.github/workflows/parity-nightly.yml b/.github/workflows/parity-nightly.yml new file mode 100644 index 00000000..bdaaa50a --- /dev/null +++ b/.github/workflows/parity-nightly.yml @@ -0,0 +1,142 @@ +name: Parity (nightly, licensed solver) + +# The genome-scale checks the studies rest on need a Gurobi licence that will +# accept a model with thousands of reactions. The licence bundled with +# pip-installed gurobipy caps out well below that, so this workflow uses a +# Gurobi Web License Service (WLS) licence, supplied in either of two shapes: +# +# GUROBI_EDUARD an organisation secret holding a licence file verbatim +# +# or, failing that, three separate secrets: +# +# GUROBI_WLSACCESSID the WLS access id +# GUROBI_WLSSECRET the WLS secret +# GUROBI_LICENSEID the numeric licence id +# +# WLS is the licence type intended for containers and CI; a named-user academic +# licence is tied to a specific machine and is not appropriate for a hosted +# runner. + +on: + schedule: + # 03:17 UTC, offset off the hour so it does not queue behind everything + # else scheduled on the hour. + - cron: "17 3 * * *" + workflow_dispatch: + +concurrency: + group: parity-nightly + cancel-in-progress: false + +jobs: + genome-scale: + name: genome-scale parity + # Forks do not have the secrets, and a scheduled run there would fail every + # night for no reason. + if: github.repository == 'SysBioChalmers/raven-toolbox' + runs-on: ubuntu-latest + # Measured: the Human-GEM extraction takes ~68 minutes on a fast + # workstation, of which only ~2 are parsing and copying. run_init's + # time_limit does not bound the call -- a 60-second limit still ran for + # well over 15 minutes -- so the real stop is pytest-timeout below, and + # this budget only has to be comfortably larger than that so the timeout + # is reported as a test failure instead of a killed job. + timeout-minutes: 150 + + steps: + - uses: actions/checkout@v5 + + - name: Check out MATLAB RAVEN + uses: actions/checkout@v5 + with: + repository: SysBioChalmers/RAVEN + ref: develop3 + path: .raven + + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: pyproject.toml + + - run: pip install --upgrade pip + - run: pip install -e ".[dev]" gurobipy + + # Fail here, loudly, rather than let the tests skip and the job go green + # having checked nothing -- which is the failure mode this whole harness + # exists to avoid. + - name: Install the Gurobi WLS licence + env: + # Two shapes exist in the organisation. GUROBI_EDUARD is an organisation + # secret holding a licence file verbatim; the three-field form below is + # what this workflow was originally written against. Only WLSACCESSID was + # ever created, so the three-field branch has never fired and this job has + # failed here every night since it was added. Accept either. + LICENCE_FILE: ${{ secrets.GUROBI_EDUARD }} + WLSACCESSID: ${{ secrets.GUROBI_WLSACCESSID }} + WLSSECRET: ${{ secrets.GUROBI_WLSSECRET }} + LICENSEID: ${{ secrets.GUROBI_LICENSEID }} + run: | + umask 077 + licence="$HOME/gurobi.lic" + if [ -n "$LICENCE_FILE" ]; then + printf '%s\n' "$LICENCE_FILE" > "$licence" + elif [ -n "$WLSACCESSID" ] && [ -n "$WLSSECRET" ] && [ -n "$LICENSEID" ]; then + { echo "WLSACCESSID=$WLSACCESSID" + echo "WLSSECRET=$WLSSECRET" + echo "LICENSEID=$LICENSEID" + } > "$licence" + else + # Fail here, loudly, rather than let the tests skip and the job go green + # having checked nothing -- which is the failure mode this whole harness + # exists to avoid. + echo "::error::No Gurobi licence configured. Set the GUROBI_EDUARD" \ + "organisation secret to a WLS licence file, or set GUROBI_WLSACCESSID," \ + "GUROBI_WLSSECRET and GUROBI_LICENSEID. The genome-scale checks cannot" \ + "run without one." + exit 1 + fi + + # Report the shape without printing the value. A named-user academic licence + # is tied to one machine and cannot authenticate on a hosted runner, so + # finding one here explains a failure that otherwise looks like a solver bug. + if grep -q '^[[:space:]]*WLSACCESSID=' "$licence"; then + echo "Gurobi WLS licence installed" + else + echo "::warning::the configured licence is not a WLS licence. A named-user" \ + "licence is tied to a specific machine and will not authenticate on a" \ + "hosted runner." + fi + echo "GRB_LICENSE_FILE=$licence" >> "$GITHUB_ENV" + + # Well past the 2000x2000 limit of the bundled licence, so this step fails + # unless the WLS licence really is in effect -- better to find out here + # than to have the tests skip and the job go green having checked nothing. + - name: Verify the licence accepts a genome-scale problem + run: | + python scripts/parity/check_license.py + + - name: Download Human-GEM + run: | + curl -fsSL -o "$RUNNER_TEMP/Human-GEM.yml" \ + https://raw.githubusercontent.com/SysBioChalmers/Human-GEM/main/model/Human-GEM.yml + ls -lh "$RUNNER_TEMP/Human-GEM.yml" + + - name: Run the parity suite, including the slow checks + run: pytest -m parity -v -rs --timeout=5400 --timeout-method=thread + env: + RAVEN_ROOT: ${{ github.workspace }}/.raven + HUMAN_GEM_YML: ${{ runner.temp }}/Human-GEM.yml + RAVEN_PARITY_REPORT_DIR: ${{ runner.temp }}/parity-report + PYTHONUNBUFFERED: "1" + + # The numbers are what make the run useful: they are the candidate for the + # next recorded baseline. Uploaded even when the tests fail, since a + # failure is exactly when the actual reaction set needs inspecting. + - name: Upload the run report + if: always() + uses: actions/upload-artifact@v4 + with: + name: genome-scale-parity-report + path: ${{ runner.temp }}/parity-report + if-no-files-found: warn diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 00000000..b4070e19 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,119 @@ +name: Release + +on: + push: + tags: ["v*"] + # Lets an already-pushed tag be released: the tag is cut first and this is + # dispatched against it afterwards. ``push.tags`` cannot fire retroactively. + workflow_dispatch: + inputs: + tag: + description: "Existing tag to release (e.g. v0.3.0)" + required: true + type: string + +# Least privilege by default; each job widens only what it needs. +permissions: {} + +jobs: + build: + name: build + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + with: + # On a tag push this is the tag itself; on manual dispatch it is the + # tag the operator named. Either way the build comes from the tag, + # never from a branch that may have moved on. + ref: ${{ inputs.tag || github.ref }} + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: pyproject.toml + - run: pip install --upgrade pip + - run: pip install build + - run: python -m build + + # The tag is the source of truth. A version is immutable once uploaded to + # PyPI, so a mismatch between the tag and the built dist is the one + # release mistake that cannot be undone — fail before publishing, not after. + - name: Verify the built version matches the tag + run: | + tag="${{ inputs.tag || github.ref_name }}" + version="${tag#v}" + for f in "dist/raven_toolbox-${version}-py3-none-any.whl" "dist/raven_toolbox-${version}.tar.gz"; do + test -f "$f" || { + echo "::error::expected $f; dist/ holds: $(ls dist/)" + exit 1 + } + done + + # The CHANGELOG section for this version becomes the release notes, which + # is how the 0.2.0 notes were written by hand. Keeps one source of truth. + - name: Extract the release notes from CHANGELOG.md + run: | + tag="${{ inputs.tag || github.ref_name }}" + version="${tag#v}" + awk -v v="$version" ' + $0 ~ "^## " v "( |$)" { found = 1; next } + found && /^## / { exit } + found { print } + ' CHANGELOG.md > notes.md + test -s notes.md || { + echo "::error::no '## ${version}' section in CHANGELOG.md" + exit 1 + } + cat notes.md + + # dist/ and notes.md share the workspace root as their least common + # ancestor, so the artifact holds ``dist/*`` beside ``notes.md`` — the + # layout both downstream jobs expect. + - uses: actions/upload-artifact@v7 + with: + name: dist + path: | + dist/ + notes.md + + github-release: + name: github release + needs: build + runs-on: ubuntu-latest + permissions: + contents: write # create the release and upload its assets + steps: + - uses: actions/download-artifact@v8 + with: + name: dist + - name: Create the release + env: + GH_TOKEN: ${{ github.token }} + run: | + tag="${{ inputs.tag || github.ref_name }}" + # --verify-tag refuses to invent a tag that does not already exist. + gh release create "$tag" dist/* \ + --repo "${{ github.repository }}" \ + --title "raven-toolbox ${tag#v}" \ + --notes-file notes.md \ + --verify-tag \ + --latest + + pypi: + name: pypi + needs: build + runs-on: ubuntu-latest + # Trusted Publishing: PyPI verifies this workflow's OIDC identity, so no + # long-lived token is stored in the repo. Requires a one-time publisher on + # PyPI for raven-toolbox with: + # owner: SysBioChalmers repo: raven-toolbox + # workflow: release.yml environment: pypi + environment: pypi + permissions: + id-token: write + steps: + - uses: actions/download-artifact@v8 + with: + name: dist + - uses: pypa/gh-action-pypi-publish@release/v1 + # Only dist/ is uploaded; notes.md sits outside it and is ignored. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e197d65..cd4b424e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,167 @@ Milestones in the raven-toolbox port. For function-level status see [docs/raven_migration.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/migration.md); for open work see [docs/todo.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/reference/todo.md). +## 0.4.0 — 2026-08-28 + +A `raven-gecko-parity` cross-validation harness went live this release and immediately paid for itself: +six behavioural divergences from MATLAB RAVEN closed (task checking, condition application, duplicate- +reaction contraction, Excel export, ΔG loading, reaction creation), the YAML writer brought to byte-parity +with `writeYAMLmodel.m`, and BLAST/homology defaults matched to RAVEN's own. Also: deterministic +compartment placement, confidence-score interoperability (Thiele-Palsson, ECO), a new `set_exchange_bounds` +port, and `merge_compartments` promoted to public API and fixed. + +* **`convert_to_irreversible` now carries the reaction's annotations, subsystem and notes onto the + `_REV` reaction.** MATLAB's `convertToIrrev` copies the per-reaction fields (`eccodes`, `rxnMiriams`, + `subSystems`, `rxnNotes`, ...) across when it splits a reversible reaction; the Python port copied only + the name and the gene rule. Downstream that silently cost the reverse reaction its EC code — in geckopy, + `fill_eccodes_from_gem` returned `''` for every `_REV` reaction, fuzzy BRENDA matching then found no + kcat, and the reverse direction ended up unconstrained. + +* **Opt-in deterministic ftINIT extraction: `ftinit(..., strict_gap=True, canonical=True)`.** The extraction + MILP is highly degenerate — many reaction subsets reach the same score optimum — so `Threads=1` + a fixed + `Seed` make the solver's choice *reproducible* but not *unique*: a different Gurobi version or platform + lands on a different optimum. Two new keyword flags, **both default off** so existing behaviour is + unchanged, pin it instead. `strict_gap` proves each step to a fixed *absolute* gap (0.05, below the 0.1 + reaction-score granularity) in a single solve, replacing RAVEN's relative-gap escalation, whose final + near-zero-objective run otherwise accepts an arbitrary within-gap incumbent. `canonical` adds a + lexicographic phase 2 over the removable reactions — hold the score objective at its optimum, minimise the + count of kept reactions, then their summed id rank — selecting the sparsest, lowest-id optimum instead of a + solver tie-break; it applies to both each extraction step and the task gap-fill (`fill_tasks(..., + canonical=True)`). A solve that ends at the time limit with no incumbent now raises a clear error (or, for + the canonical phase, falls back to the primary optimum) rather than an opaque `AttributeError` on a null + primal. **These are a reproducibility/parsimony tool, not an accuracy or a gene-essentiality fix**: on + Human-GEM/DLD1 they cut the reaction-level seed-swing 125 → 34 but *worsened* essential-gene determinism + (5 → 19 flips) at a 3–7× build-time cost, because the sparser model is more sensitive to the residual + transport degeneracy. For reproducible essentiality, pin the solver stack instead. Measurements and the + full trade-off: [docs/studies/ftinit_determinism.md](https://github.com/SysBioChalmers/raven-toolbox/blob/develop/docs/studies/ftinit_determinism.md). +* **`assign_compartments` gap-fill: a reliable flux-based fill instead of cobra's MILP.** The + universal-DB gap-fill in `localization.certify` no longer calls `cobra.flux_analysis.gapfill`, whose + indicator MILP, at genome scale, fails to find a valid fill in the **majority** of cases *even when the + exact reaction that restores growth is present in the universal* (it returns an incumbent its own + validation rejects, and raises rather than offering a fill). The replacement adds the candidates on a + working copy, holds biomass at the floor, runs pFBA, and returns the flux-carrying additions — **sorted**, + so the result does not depend on which co-optimal vertex the solver picked. A plain LP cannot have the + MILP's failure mode: a returned set is a real flux solution that reaches the floor. On single-reaction + knockout-recovery (remove an essential reaction, fill from a universal that contains it), recall is + **60/60 — the exact reaction each time — vs cobra's 45 %**; on realistic incomplete drafts (12% of + internal reactions dropped, many simultaneous gaps) it restores growth **12/12 vs cobra's 0/12**. The universal must share the draft's + metabolite namespace: candidates are matched by id, as cobra's gapfill required, and a mismatch now + **warns** instead of returning `[]` silently. The set is flux-parsimonious (pFBA) but not guaranteed + reaction-count-minimal; the caller re-certifies with a real FBA, so no false certificate is possible. +* **Deterministic, score-aligned reaction placement in `assign_compartments`.** The placement master + maximises a gene-localisation objective that never mentions the per-reaction placement variable, so + each reaction's compartment was a free co-optimum: the pinned solver returned it reproducibly but + *arbitrarily*, and reaction agreement with curated yeast-GEM was only **52.8 %** (an earlier un-pinned + build happened to land ~72 %). A lexicographic second pass now fixes the gene layout to the primary + optimum, then places each reaction in the compartment its own enzymes are predicted to occupy — the + summed DeepLoc score of the reaction's genes, with a small `default_compartment` prior so genes-free + and score-tied reactions fall there deterministically. Yeast reaction agreement rises to **72.5 %** + (1408/1943) and now rests on the localisation evidence rather than a solver tie-break; **gene agreement + is unchanged** (88.7 %, 716/807 — the gene layout is untouched); coherent placement fragments fewer + metabolites, so it adds **fewer transports** (1001 → 967); growth and blocked-fraction are unchanged; + the second solve is warm-started and adds ~4 s. Reproducible run-to-run (identical placement across + independent runs). +* **`diff_models` compares grRules as logic, not text.** The GPR check now DNF-expands each rule (via the + existing `manipulation.gpr_to_dnf`), sorts the genes within each isozyme clause and sorts the clauses, so + operand order no longer registers as a difference: `a and b` == `b and a` and `a or b` == `b or a`. The + previous heuristic only lowercased and collapsed whitespace, so it flagged logically identical rules that + differed only in operand order. This brings `diff_models` in line with MATLAB RAVEN's `diffModels` + ([RAVEN #686](https://github.com/SysBioChalmers/RAVEN/pull/686)); a rule cobra cannot parse falls back to + the old string comparison, so malformed rules are still compared rather than silently equated. +* **`load_delta_g_csv` defaults to stamping every matched value literally, matching `deltaGCSV` + (raven-gecko-parity#16).** An earlier change taught the loader to treat the ΔG side-car tables' "no + measurement" sentinel (`10000000`, matching yeast-GEM's own `checkrxnDirection.m`) as missing rather than + a real value, defaulting to skip it. RAVEN's `deltaGCSV` has no sentinel concept at all and stores + whatever the CSV says, verbatim, so the two implementations disagreed about whether an entity had a value + on record. The default is now literal stamping, matching RAVEN; the sentinel-skipping behaviour remains + available by passing the keyword-only `missing_value=DELTA_G_MISSING` explicitly. +* **Wire the confidence facets together.** `confidence.annotate_confidence(model, proposal=..., scores=...)` + runs every applicable scorer in one call and returns `{facet: reactions_scored}` — `equation` and + `gene_association` need only the model, `localization` runs only when a proposal and its scores are given + (skipped, not failed, otherwise). `curation_priority` now drops a placement a curator has settled with + `mark_curated` from the review queue (new `include_curated=False`), so a settled reaction stops + resurfacing; `include_curated=True` keeps it. The no-SBO-terms warning now names its remedy, + `raven_toolbox.annotation.add_sbo_terms(model)`. +* **Confidence facets mapped onto the field's standard vocabulary.** New `thiele_palsson_score(reaction)` + derives the Thiele & Palsson 2010 reconstruction confidence score (0–4) from the `gene_association` + facet's evidence `basis` (`gpr+literature` → 3, `gpr` → 2, `no-gpr` → 1); every ECO evidence id used + (`ECO:0000044`, `ECO:0000015`, `ECO:0000002`) was checked against EBI's OLS. Cross-model validation on + Human-GEM, iYali and panAsp/pAo confirmed the `equation` facet's bands are calibrated rather than + yeast-GEM-fixture artefacts (two of its three bands fire on the other models too). +* **`manipulation.merge_compartments` / `copy_to_compartment` are now public API, and `merge_compartments` + no longer breaks growth.** Both functions existed in `manipulation/compartments.py` but were never + imported into `manipulation/__init__.py`, so neither was reachable as documented despite `migration.md` + listing both as ported. Exporting them also surfaced two silent defects: any reaction left with a single + metabolite after merging was dropped — including reactions that already had one metabolite *before* + merging, i.e. every exchange reaction — and the merged model's objective was never carried over, so a + flattened model silently optimised to `0.0`. Both now match RAVEN's `mergeCompartments`: single-metabolite + reactions are recorded before the merge and excluded from deletion, and the objective (with its direction) + is copied onto the rebuilt model, raising a warning rather than returning a zero-objective model if every + objective-carrying reaction genuinely was removed. +* **Six behavioural divergences from MATLAB RAVEN closed, found by the new `raven-gecko-parity` + cross-validation harness:** + - `check_tasks` / `find_task_essential_reactions` now default to RAVEN `checkTasks`' additive boundary + semantics (`close_boundaries=False`): a task is checked by relaxing metabolite balance for its declared + inputs/outputs, not by closing every exchange first. A task the model can already satisfy through its + own open boundary, without naming it, now agrees with RAVEN's feasible verdict instead of reporting + infeasible (raven-gecko-parity#7). + - `apply_condition`'s `reset_exchanges` now honours the direction it names ("in"/"out"), replicating + `getExchangeRxns`' own rule, instead of resetting every exchange reaction for any truthy value + (raven-gecko-parity#15). + - `remove_duplicate_reactions` now round-trips losslessly with `expand_model`, matching RAVEN's + `contractModel`: it keeps the first-encountered survivor with the `_EXP_N` suffix stripped, and unions + every duplicate's GPR instead of keeping only the survivor's own — previously an isozyme relationship + `expand_model` had split out came back with two of three genes unassociated (raven-gecko-parity#8). + - `add_reactions_from_equations` now defaults a newly created metabolite's name to its id, matching + RAVEN's `addRxns`, instead of leaving it empty — a nameless metabolite defeated the name-keyed matching + in `merge_compartments` and `add_reactions_from_model` (raven-gecko-parity#9). + - `export_to_excel` now hides `LOWER BOUND` / `UPPER BOUND` cells that equal the model's declared default, + matching `exportToExcelFormat`, instead of always writing the literal number — on a model whose bounds + are drawn from its own declared defaults (the common case), nearly every bound cell was affected + (raven-gecko-parity#17). +* **YAML writer brought to byte-parity with `writeYAMLmodel.m`.** Closes a series of concrete diffs found + comparing MATLAB and Python output for the same model: numeric leaves are always coerced to `float` (a + whole number gets an explicit `.0`, since RAVEN's model struct cannot distinguish int- from float-typed + fields); an empty `subsystem` is omitted rather than round-tripping a "blank list entry", and a non-empty + one is always written as a list; `metaData` follows RAVEN's fixed field order and defaults + (`id`/`name` → `"blankID"`/`"blankName"`, `date` → today); `defaultLB`/`defaultUB` are recomputed from the + model's current bounds, matching `readYAMLmodel.m`'s derivation; a singleton MIRIAM annotation collapses + to a bare scalar (except `ec-code`/`smiles`, which RAVEN's writer always keeps as a list); a metabolite + with no explicit compartment now defaults to the model's first compartment; the block-sequence indent + (`mapping=2, sequence=2, offset=0`) is pinned explicitly rather than relying on ruamel's own defaults; and + quoted scalars use double quotes, matching Prettier's YAML default. Separately, `read_yaml_model` no + longer raises `KeyError` on a RAVEN-authored model that omits an empty `genes:`/`metabolites:`/ + `reactions:` block entirely (RAVEN's own `tutorial/small.yml` is one such file) — the three entity + sections now default to empty lists. Verified byte-identical against `writeYAMLmodel.m` output, including + a cross-read of MATLAB's own double-quoted output. +* **BLAST/homology defaults matched to RAVEN's, and measured rather than guessed.** `run_blast`'s + `evalue` default changed from `1e-5` to `1e-4`, matching RAVEN `getBlast`'s hardcoded `-evalue 10e-5` + (`run_diamond` is deliberately left alone — RAVEN's `getDiamond` passes no `--evalue` flag, so DIAMOND's + own `1e-3` default applies there, a third, distinct value). Separately, `min_align_len` drops from 200 to + 100: reconstructing four organisms from an *S. cerevisiae* template and checking every surviving match + against both KEGG and OMA found 200 was discarding 3–4% of real matches for a wrong-match-rate change of + at most 0.6 points; `min_identity=40` and `max_evalue=1e-30` were confirmed already correct. New + `review_identity` on `get_model_from_homology` surfaces near-miss candidates that a stricter identity + threshold excluded, each with its supporting match, instead of discarding that evidence silently. +* **New `set_exchange_bounds`, RAVEN's `setExchangeBounds` port (raven-gecko-parity#19).** Covers what + `model.medium` cannot: independent lower and upper bounds per metabolite, a `media_only` compartment + filter, and a check that every exchange reaction agrees on which flux sign is import before any bound is + touched. Matches metabolites against the model's exchanged set by id or name, case-insensitively, and + warns when a metabolite is exchanged in more than one reaction — a warning RAVEN's own implementation has + a latent bug that prevents from ever firing. +* **Cross-language parity testing harness.** `tests/parity/` turns "validated against MATLAB RAVEN" from a + claim in `docs/studies/` into an enforced property: an **exact** tier for I/O round-trips, task parsing, + GPR normalisation and similar (compared value-for-value); a **set-level** tier for INIT/ftINIT extraction, + gap-filling and compartment assignment (Jaccard overlap against a recorded baseline, since these have many + equally-valid optima); and a stated but not-yet-tested **statistical** tier for flux/random sampling. A + nightly job runs the set-level check against genome-scale Human-GEM under a real Gurobi licence. Building + the harness immediately found that `read_yaml_model` could not read a RAVEN-authored model with no genes + (see above) — the kind of gap unit tests, which all happened to use fixtures with genes, could not catch. +* **Fix a downloaded binary bundle leaving its non-primary executables non-executable.** A bundle providing + several tools (BLAST ships both `blastp` and `makeblastdb`) only `chmod`'d the one that was actually + requested, and a cached path returned before reaching the `chmod` at all — fetching `blastp` then calling + `makeblastdb` raised `PermissionError`. + ## 0.3.0 — 2026-07-16 Compartment localisation and per-reaction confidence tracking, new gap-filling and flux-sampling diff --git a/docs/guide/context_specific.md b/docs/guide/context_specific.md index fab1075f..58faae13 100644 --- a/docs/guide/context_specific.md +++ b/docs/guide/context_specific.md @@ -33,6 +33,14 @@ in the [parameter-calibration study](../studies/init_param_calibration.md), and equivalence to MATLAB RAVEN is established in the [Human-GEM validation](../studies/humangem_validation.md). +`ftinit()` also takes two opt-in determinism flags, `strict_gap` and `canonical` (both default +off → exact RAVEN behaviour), which pin *which* of the degenerate MILP's many equal-score optima +is returned, giving a more parsimonious and more reproducible extraction. They do not make the +model biologically more accurate, they cost 3–7× build time, and they are **not** a fix for +gene-essentiality reproducibility — measurements and the trade-off are in the +[extraction-determinism study](../studies/ftinit_determinism.md). For reproducible results +across runs, pin the solver stack (raven-toolbox commit + `gurobipy` version) instead. + :::{important} Genome-scale (f)tINIT MILPs currently require **Gurobi** for tractable solve times; toy and unit-test problems run on GLPK. See the diff --git a/docs/reference/index.md b/docs/reference/index.md index de545f93..eaf8fd83 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -2,6 +2,10 @@ Conceptual and API reference for raven-toolbox. +- **[Roadmap](roadmap.md)** — the phased development plan: what ships in which + order, what each phase depends on, and the decisions still open. The item-level + backlog behind it is [Open work](todo.md). + - **[RAVEN ↔ raven-toolbox migration map](migration.md)** — the function-by-function map from MATLAB RAVEN to raven-toolbox (and cobrapy where appropriate). Start here if you're porting RAVEN code. @@ -30,6 +34,7 @@ Conceptual and API reference for raven-toolbox. ```{toctree} :hidden: +roadmap migration yaml_format matlab_raven_backports diff --git a/docs/reference/roadmap.md b/docs/reference/roadmap.md new file mode 100644 index 00000000..18e43e33 --- /dev/null +++ b/docs/reference/roadmap.md @@ -0,0 +1,200 @@ +# Roadmap + +Where raven-toolbox and its documentation are going, in the order the work has to happen. +This page is the **plan**: phases, what each one delivers, what it depends on, and how you +know it is finished. [todo.md](todo.md) is the **backlog**: the item-level detail behind +each phase. + +Two repositories are in play: + +* **raven-toolbox** (this repo) — code, tests, CI, maintainer reference documents. +* **[raven-docs](https://github.com/edkerk/raven-docs)** — the dual-language documentation + site (MkDocs + Material on Read the Docs) that serves both MATLAB RAVEN and + raven-toolbox, with both toolboxes as git submodules. All user-facing prose belongs there. + +Phases are **dependency-ordered, not calendar-dated**. Phase 0 blocks nothing and should +happen immediately; phases 1 and 2 run in parallel; phase 3 depends on decisions D2 and D4 +below. There is no external 1.0 anchor (decision D5) — 1.0 ships when phase 4's exit +criteria are met, so **phases 3 and 4 may overlap freely** rather than running in sequence. + +--- + +## The problem this roadmap solves + +The port itself is essentially done — the functional scope of MATLAB RAVEN is covered, with +three deliberate omissions, and the results are validated on Human-GEM, yeast and a +multi-organism set. What is *not* done is everything that makes that work usable and +trustworthy by someone who is not the author: + +1. **A user cannot find out how the two toolboxes differ.** The one page that tries to say + so is factually wrong (it names Python functions that do not exist). +2. **Nothing enforces the parity claims.** They live in prose in study documents; no test + would fail if the two implementations diverged tomorrow. +3. **There is no Python tutorial anywhere.** The site's guides are 100 % MATLAB while + claiming to be dual-language. +4. **No one can tell which functions are mature.** A five-times-validated pipeline and a + thin wrapper look identical from the outside. + +Each phase below closes one of those, in the order that makes the next one cheaper. + +--- + +## Phase 0 — Stop publishing things that are false + +*No dependencies. Small. Do it first.* + +The documentation site currently misleads readers, and every day it stays up is a day +someone copies a function name that does not exist. + +* Remove or correct the invented Python names in `differences.md` (`get_blast`, + `fill_gaps`, `import_model`, `export_to_excel_format`, `solve_lp`, …). +* Drop the claim that RAVEN Excel round-trips between the two toolboxes — Python has no + Excel import by design. +* Correct `protocol/index.md`, which promises protocols "in both MATLAB and Python with + MATLAB/Python tabs" when the site has no Python content at all. +* Unbreak the `matlab-vs-python.md` reference — it is both an in-page link and a `mkdocs.yml` + nav entry pointing at a file that does not exist. + +**Exit criterion:** nothing on the published site is factually wrong. Interim honesty +("Python examples coming") is acceptable; incorrect specifics are not. + +## Phase 1 — Make the differences documentation true *and* self-maintaining + +*Depends on: phase 0. Runs parallel to phase 2.* + +Fixing the names by hand solves today and rots by next release. The deliverable is a +**generator**, and a set of small pages it feeds. + +* A build-time generator in raven-docs that emits the MATLAB ↔ Python mapping from the two + submodules' API data and **fails the build on any hand-written function name that does + not resolve**. This is the durable fix, and the reason phase 1 is worth more than a + correction pass. +* The `differences/` section, one page per question: `index`, `mapping`, `python-only`, + `matlab-only`, `behaviour`, `parity`. +* Behind it, in this repo: extend [migration.md](migration.md) with the reverse direction + (121 of 185 public names have no row today) so the generator has something to read. + +**Exit criterion:** a wrong or stale function name cannot reach the published site, and +every public function of either toolbox appears in the mapping with a counterpart or an +explicit "no equivalent, because …". + +**Note:** `differences/behaviour.md` and `differences/parity.md` can be *drafted* here from +what is already documented, but they are only *finished* by phase 2 — that is what supplies +the evidence. + +## Phase 2 — Make parity a test, not a claim + +*Depends on: phase 0 only. Runs parallel to phase 1; supplies phase 1's last two pages.* + +Today parity rests on hand-transcribed oracles, one YAML gate, and prose in study +documents. The deliverable is a harness that fails a build when the two implementations +disagree. + +* **Tier the contract first** — exact / set-level / statistical — because "identical output" + is not achievable uniformly, and pretending otherwise produces either flaky tests or + vacuous ones. This tiering is also the content of `differences/parity.md`. +* Golden-fixture harness: committed small inputs, a MATLAB driver that writes oracles, and + `pytest -m parity` that runs green with no MATLAB installed. +* Determinism regressions for the recently-fixed placement and gap-fill paths (#76, #83, + `c239d2e`), which currently have no guard. +* A nightly job on a licensed runner for the Gurobi-dependent tier-2 checks, so the study + numbers stop being refreshed by hand. + +**Exit criterion:** every parity claim on the site is backed by a test that would fail if it +stopped being true, and the behaviour table is populated from measured divergences rather +than from memory. + +## Phase 3 — A new tutorial, dual-language from the first line + +*Depends on: decisions D1 and D2 below; benefits from phase 2's fixtures.* + +The site has one protocol — the published hanpo-GEM reconstruction — and it is MATLAB-only. +**This roadmap does not port it.** It stays as it is: a faithful rendering of a published +protocol, MATLAB, untouched, alongside the untouched legacy tutorials 1–5. + +Instead we **define a new tutorial**, written dual-language from the start. It may +legitimately resemble the hanpo-GEM protocol in structure and in several steps — homology +draft, biomass, gap-filling, simulation, curation are the natural arc of any reconstruction +— but it is its own exercise with its own organism and its own runnable scripts in both +languages. + +Deliberately **out of scope for now:** the SLIME lipid-curation treatment. Parallel work is +under way on those methods, so the new tutorial does not depend on them and does not attempt +a Python equivalent. This removes what would otherwise have been the one hard blocker. + +**Two reconstruction tutorials, one per route** (decision D1). RAVEN has two ways to build a +draft, they suit different situations, and no current guide covers the second at all: + +1. **Reconstruction by homology — a non-conventional yeast.** A curated template plus + BLAST/DIAMOND, structurally parallel to the existing protocol but its own organism and + inputs. Written first: the arc is well understood, the inputs are small, and no new + infrastructure is needed. **It stops short of lipid curation** — biomass is defined + without the SLIME treatment, which keeps the tutorial clear of the parallel method work. +2. **Reconstruction *de novo* from KEGG — a small prokaryote.** The route with no coverage + anywhere on the site, and a raven-toolbox strength (artefact builders, HMM cutoff + calibration). Second, because it carries an infrastructure dependency the homology route + does not: the KEGG artefacts have to be fetched, so the tutorial has to say clearly what + is downloaded, how large it is, and what can be skipped. The existing artefact/manifest + tooling and [maintaining_kegg_data.md](../maintenance/maintaining_kegg_data.md) are the + starting point. + +Then, in order: + +3. **GEM extraction (ftINIT).** Adapt the existing material, which is already in good shape, + rather than writing it fresh. +4. **GEM comparison.** Already listed as planned on the site. +5. Then, as capacity allows: compartmentalisation, confidence tracking, ecModel handoff — + see [todo.md](todo.md) §3. + +Both reconstruction tutorials are dual-language, both lanes runnable, both scripts +version-controlled and transcluded into the pages so neither can rot. + +**Exit criterion:** a reader can complete a full reconstruction in Python, following the +site, without consulting the source; and every code block on the page comes from a script +that CI executes. + +## Phase 4 — Maturity, and what 1.0 means + +*Draws evidence from phases 1–3, but may run alongside phase 3 (decision D5). Gates 1.0.* + +* Coverage reporting with a floor, and a first pass over the modules no test obviously + touches. +* A per-function maturity table — **stable** / **provisional** / **experimental** — against + a written rubric, published on the site. +* The known functional gaps stated in one place (ftINIT metabolomics, no Excel import, no + MetaCyc, no dynamic FBA, Gurobi in practice for genome-scale work). +* API pages and docstrings for the subsystems that have neither (`confidence`, `biomass`, + `conditions`, `curation`, `annotation`, `manifest`, `data`, `binaries`) — thin docstrings + here render as thin pages on the site. +* Resolve the open 💡 proposals in [improvements.md](improvements.md): implement or formally + drop. +* A stability policy: what counts as public API, the deprecation window, and how `notes` / + `annotation` key names are versioned. + +**Exit criterion:** every public function carries a maturity label backed by evidence, and +the API surface is one we are willing to freeze. + +## Continuous + +Not phased — these run alongside: + +* Land or close the open PRs ([#75](https://github.com/SysBioChalmers/raven-toolbox/pull/75), + [#85](https://github.com/SysBioChalmers/raven-toolbox/pull/85)). +* Triage the 14 untracked scripts in `scripts/` — several are the cross-language drivers + phase 2 needs. +* Track the upstream optlang/HiGHS and GLPK blockers. +* Back-port queue to MATLAB RAVEN: `assignCompartments.m`, then FS4 / B2 / G1 / G5. + +--- + +## Decisions this roadmap is waiting on + +| # | Decision | Blocks | Status | +|---|---|---|---| +| **D1** | **Vehicle for the new tutorials** — **both routes**: reconstruction by homology for a non-conventional yeast (first), and *de novo* from KEGG for a small prokaryote (second). Neither depends on SLIME lipid methods. Still to pick within this: the specific organisms, templates, and the published phenotype each is validated against — the way *H. polymorpha*'s methylotrophy anchors the existing protocol. | Phase 3 | **Decided** (2026-08-20); organisms TBD | +| **D2** | **How Python-only material presents** on a site whose convention is linked MATLAB/Python tabs defaulting to MATLAB. Compartmentalisation and confidence tracking have no MATLAB counterpart. Badge-and-explain, or a separate grouping? | Phase 3 | **Open** | +| **D3** | **SLIME lipid curation** — out of scope for the new tutorial while the parallel method work proceeds. | — | **Decided** (2026-08-20) | +| **D4** | **Where the tutorial's runnable scripts live** — a dedicated tutorial repository, the model repository, or this repo's `scripts/`. Determines what CI can execute. | Phase 3 | **Open** | +| **D5** | **What anchors 1.0** — nothing external. 1.0 ships when phase 4's exit criteria are met, so phases 3 and 4 may overlap. | Phase 4 | **Decided** (2026-08-20) | + +Record D1–D2 and D4 in the raven-docs decisions log, since they shape the site. diff --git a/docs/reference/todo.md b/docs/reference/todo.md index 8d5b3d49..6eaadcf1 100644 --- a/docs/reference/todo.md +++ b/docs/reference/todo.md @@ -1,24 +1,286 @@ # Open work -What's still on the books. See [raven_migration.md](migration.md) for the function-by- -function port status (most of it is done); see [IMPROVEMENTS.md](improvements.md) for the -catalogue of raven-toolbox improvements that should also be back-ported into MATLAB RAVEN. +The item-level backlog. For the **plan** — phases, dependencies, exit criteria and the open +decisions — see [roadmap.md](roadmap.md); this page is the detail behind it. -## Smaller items +Items are tagged **P0** (do next), **P1** (should land before 1.0), **P2** (nice to have / +watching brief), and by where the work lands: -* [known_issues.md](known_issues.md) — backlog of low-priority edge cases / robustness gaps / - dead code from the full-codebase review. None affects correctness on well-formed inputs. -* [IMPROVEMENTS.md](improvements.md) — items marked 💡 *proposed* are candidates to - implement (and back-port). -* [matlab_raven_backports.md](matlab_raven_backports.md) — the MATLAB ↔ raven-toolbox - **differences** record: improvements to back-port to MATLAB RAVEN, and functionality - MATLAB has that raven-toolbox deliberately doesn't (MetaCyc, dynamic FBA, ftINIT - metabolomics scoring). +* **→ raven-docs** — the dual-language documentation site + ([edkerk/raven-docs](https://github.com/edkerk/raven-docs), MkDocs + Material on Read the + Docs; RAVEN and raven-toolbox are git submodules, tracking `develop3` and `develop`). + User-facing prose belongs there so it is online and covers both languages. Its + `DESIGN.md` is the source of intent — read it before writing pages, and record decisions + in its decisions log. +* **→ raven-toolbox** — code, tests, CI, and the internal reference documents that back the + public pages. + +**Page-granularity rule for raven-docs:** one topic per page, kept short. Prefer several +small linked pages over one long scrolling document; the site's left sidebar is the index. +The existing reconstruction protocol is the model to follow — one protocol is a *nav +section* of ~40–95-line chapter pages, not a single page. + +Background references in this repo: [migration.md](migration.md) (function-by-function port +status), [matlab_raven_backports.md](matlab_raven_backports.md) (MATLAB ↔ Python differences +record), [improvements.md](improvements.md) (design decisions + proposed improvements), +[known_issues.md](known_issues.md) (review backlog — sections A–F are all closed). + +--- + +## 1. MATLAB ↔ Python differences (→ raven-docs) + +`docs/differences.md` on the site is a single page that mixes orientation, function mapping, +per-language feature lists and solver notes — and it is **currently wrong in ways that will +mislead users**: + +* It lists Python functions that do not exist: `get_blast`, `get_diamond` (real names + `run_blast` / `run_diamond`), `fill_gaps` (real names `connect_blocked_reactions`, + `fill_gaps_fast_lp`, `fill_gaps_kumar_milp`), `import_model`, `export_model`, + `import_excel_model`, `export_to_excel_format` (real name `export_to_excel`), `solve_lp`. +* It states that both toolboxes read and write RAVEN Excel "making models fully portable". + raven-toolbox exports Excel but **deliberately has no Excel import**. +* It links twice to `matlab-vs-python.md`, which does not exist — and `mkdocs.yml` lists + that same missing file as a **nav entry** under "MATLAB vs Python", so the build emits a + broken nav reference. + +The underlying cause is that the page was written by analogy (camelCase → snake_case) +instead of from the source, so the fix has to be structural, not just a correction pass. + +* **P0 — Split `differences.md` into a small section**, one page per question, replacing the + single page: + * `differences/index.md` — orientation only: two independent implementations, large + overlap, why the Python one is built on cobrapy. Short, with cards to the pages below. + * `differences/mapping.md` — the full MATLAB ↔ Python function table the nav and the page + body already link to as `matlab-vs-python.md`. Per `DESIGN.md` §4.3: RAVEN ↔ + raven-toolbox pairs auto-generated from the API data, cobrapy-alternative rows + hand-curated. + * `differences/python-only.md` — functionality with no MATLAB counterpart. From this + repo's `__all__`s, 121 of 185 public names have no row in + [migration.md](migration.md) today, including whole subsystems: confidence tracking, + biomass helpers, growth conditions, batch curation, ΔG/SBO annotation, transport + evidence, `assign_compartments`, `diff_models`, the KEGG artefact builders. + * `differences/matlab-only.md` — `ravenCobraWrapper`, the `drawMap` family, MetaCyc + reconstruction (flagged for removal upstream), dynamic FBA, ftINIT metabolomics scoring, + Excel import, `printFluxes`; each with the reason it is absent and what to use instead. + * `differences/behaviour.md` — **same function, different answer.** The page that does not + exist anywhere today and is the most valuable one: differing defaults, arguments, return + shapes, ordering/tie-breaking, and solver dependence. Rows already known from this repo: + `check_tasks` (one model reused vs copy-per-task), `reporter_metabolites` (one-sided + p-value, z-sorted vs RAVEN's two-tailed ordering), `fseof` (abs-slope classifier), + `get_elemental_balance` (graded `unknown` class), `run_init` vs `ftinit` score-0 + semantics, `merge_models` / `add_reactions_from_model` (`name[comp]` matching), + `convert_to_irreversible` / `expand_model` (geckopy-derived), `write_yaml_model` + (`!!omap`, `metaData` first), `diff_models` (order-insensitive grRule logic), plus the + two the protocol port will surface: `fillGaps` vs the three Python gap-fillers, and + `contractModel` vs `remove_duplicate_reactions`. + * `differences/parity.md` — what "the same result" means per function class: exact, + set-level (alternate MILP optima), or statistical. Pairs with the test tiers in §2. +* **P0 — Generate the mapping, and validate names against the submodules.** A build-time + generator (raven-docs `scripts/`, alongside `gen_api_pages.py`) that emits the pairs and + **fails the build on any hand-written function name that no longer resolves** in either + submodule. This is what prevents a repeat of the wrong names above. +* **P1 — Pin the baseline.** Record which RAVEN commit each differences page was verified + against, plus a "last verified" date, so drift is visible. +* **P1 — Keep the source-of-truth split clear.** raven-docs carries the user-facing pages; + this repo keeps [migration.md](migration.md) (port decisions), + [matlab_raven_backports.md](matlab_raven_backports.md) (upstream port plans) and + [improvements.md](improvements.md) (rationale) as the maintainer record the pages are + derived from. Cross-link both ways; do not duplicate prose. + +## 2. Cross-language equivalence tests (→ raven-toolbox) + +What exists today: hand-transcribed RAVEN oracles (`tests/tinit_oracles.py`, from +`tinitTests.m`), the YAML round-trip parity gate (`tests/test_io_yaml_parity.py`), and the +validation studies (Human-GEM Jaccard 0.975–0.980, yeast, multi-organism) which are +*reported* in `docs/studies/` but never *asserted* in CI. Nothing regenerates MATLAB output +automatically, so a parity claim cannot fail a build. + +* **P0 — Golden-fixture harness.** Committed small inputs plus a MATLAB driver that runs the + RAVEN counterpart and writes JSON/CSV oracles into `tests/data/matlab/`; pytest loads + those and asserts equality. Fixtures small enough to commit and to run without Gurobi. + *Done when:* `pytest -m parity` passes on a clean checkout with no MATLAB installed, and a + documented regeneration step reruns MATLAB to refresh the oracles. +* **P0 — Tier the parity contract**, because "identical output" is not achievable uniformly. + The tiers are also what `differences/parity.md` publishes: + 1. **Exact** — YAML/Excel/SIF export, task-list parsing, GPR normalisation, elemental + balance, `sort_identifiers`, `merge_models`, `convert_to_irreversible`, `expand_model`, + KEGG table parsing, homology ortholog maps. + 2. **Set-level** — MILP outcomes ((f)tINIT extraction, gap-filling, localisation): assert + Jaccard/containment bands against a recorded baseline rather than identity, since + alternate optima are legitimate. + 3. **Statistical** — flux sampling and random sampling: distributional checks at a fixed + seed. +* **P1 — The new tutorial is itself a parity fixture.** Written dual-language from the + start (§3), it gives an end-to-end reconstruction with a MATLAB lane and a Python lane + over identical inputs — the most realistic equivalence test available, and far better + evidence than toy models. Design its fixtures so the nightly job can run both lanes and + diff the resulting models. (The published hanpo-GEM protocol, with its known anticipated + results, remains a useful *reference* benchmark even though the tutorial is not based on + it.) +* **P1 — Determinism regression tests.** Recent fixes (#76, #83, `c239d2e`) made placement + and gap-fill deterministic, but no CI test would catch a regression. Add repeated-run + identity assertions for `assign_compartments`, `predict_localization`, and + `run_init`/`ftinit` on toy models, plus row/column ordering of the built MILPs. The + untracked `scripts/determinism_probe.py` / `master_determinism_probe.py` are the starting + point. +* **P1 — Solver-dependent parity job.** Genome-scale (f)tINIT needs Gurobi, which free + runners cannot install. A nightly / manually-triggered workflow on a licensed runner that + runs the tier-2 checks and reports the Jaccard numbers, so the study documents stop being + hand-refreshed. +* **P1 — Promote or delete the 14 untracked scripts** now sitting in `scripts/` + (`cross_py_on_mat.py`, `full_pipeline_py.py`, `diff_drafts.py`, `export_draft*.py`, + `export_scope.py`, `param_sweep.py`, `py_mps.py`, `roworder_test.py`, …). Several are + exactly the cross-language drivers the harness above needs; the rest are spent scratch + work. *Done when:* `git status` is clean and `scripts/README.md` documents what survived. +* **P2 — Assert the validation-study headline numbers** (Jaccard bands, task pass counts) as + tolerance-checked tests in the nightly job, so a study document can never disagree with + the code. + +## 3. Protocols and tutorials (→ raven-docs) + +### What is actually on the site + +Reconciled against the checkout, not the repository listing: + +* **One protocol exists, in ten chapter pages** — the hanpo-GEM homology reconstruction + (Zorrilla & Kerkhoven 2022, *MMB* 2513), nav section "GEM reconstruction (*H. polymorpha*)": + introduction → materials → template models → homology → biomass → lipid curation → + gap-filling → save & simulate → manual curation → anticipated results. Its authoritative + source is `code/reconstructionProtocol.m` in the hanpo-GEM submodule, transcluded whole + into the last page. +* **Legacy tutorials 1–5** — MATLAB only, RAVEN 1 exercises, staying untouched. +* **"GEM extraction" and "GEM comparison" are listed as planned** on `protocol/index.md`, + with nothing behind them yet. + +### The finding that reorders this section + +**There is no Python anywhere in the guides.** Across the ten protocol chapters and the five +legacy tutorials there are 72 MATLAB code blocks, **zero Python blocks and zero +MATLAB/Python tabs** — while `protocol/index.md` tells the reader protocols "are provided in +**both MATLAB and Python** with code shown in MATLAB/Python tabs", and `DESIGN.md` §4.4 and +§5 require linked, persistent tabs site-wide. The site currently promises a dual-language +protocol it does not have. + +### What we are doing about it + +**The existing protocol is not ported.** It stays as it is — a faithful rendering of a +published MATLAB protocol — alongside the untouched legacy tutorials. Instead we **define new +tutorials, dual-language from the first line**. The homology one may resemble the hanpo-GEM +protocol in structure and in several individual steps (homology draft → biomass → +gap-filling → simulation → curation is the natural arc of any reconstruction), but it is its +own exercise, with its own organism, its own inputs, and its own runnable scripts in both +languages. + +**SLIME lipid curation is out of scope** while the parallel work on those methods proceeds. +The new tutorials do not depend on it and do not attempt a Python equivalent — which removes +what would otherwise have been the one hard blocker. + +* **P0 — Correct the claim on `protocol/index.md` now.** Until Python material exists, the + index must not state that protocols are provided in both languages. A one-line fix that + stops the site over-promising. +* **P0 — Two new reconstruction tutorials, one per route** (roadmap decision **D1**), each a + nav section of chapter pages with MATLAB/Python tabs throughout: + * **Homology — a non-conventional yeast.** First: well-understood arc, small inputs, no new + infrastructure. Stops short of lipid curation; biomass without the SLIME treatment. + * **De novo from KEGG — a small prokaryote.** Second: the route with no coverage on the + site today, but it must be explicit about artefact downloads and their size. + Still to pick: the specific organisms and templates, and the published phenotype each + tutorial validates against. +* **Feasibility, measured.** I checked every MATLAB call in the existing protocol's ten + chapters against this package, as a proxy for what a Python lane of *any* reconstruction + tutorial needs: **seven of the eight code-bearing chapters are already portable**, and the + one that is not is the SLIME chapter now out of scope. + + | Chapter | MATLAB | Python | Status | + |---|---|---|---| + | Template models | `importModel`, `exportToExcelFormat`, `exportModel` | `cobra.io.read_sbml_model`, `export_to_excel`, `cobra.io.write_sbml_model` | ✅ portable | + | Draft from homology | `getBlast`, `getModelFromHomology`, `contractModel`, `addRxnsGenesMets` | `run_blast`, `get_model_from_homology`, `remove_duplicate_reactions`, `add_reactions_from_model` | ✅ portable (`contractModel` ≠ `remove_duplicate_reactions` exactly — behaviour row) | + | Biomass composition | `addRxnsGenesMets`, `changeRxns` | `add_reactions_from_model`, `change_reaction_equations` | ✅ portable | + | Lipid curation (SLIME) | `addLipidReactions`, `addSLIMEreactions`, `scaleLipids` | — | ⏸️ **out of scope** — SLIMEr / hanpo-GEM helpers, not RAVEN core; parallel method work ongoing, so the new tutorial avoids this ground | + | Gap-filling | `setParam`, `getExchangeRxns`, `removeReactions`, `fillGaps`, `solveLP`, `printFluxes`, `deleteUnusedGenes` | bounds/objective one-liners, `model.exchanges`, `remove_reactions`, `connect_blocked_reactions` / `fill_gaps_fast_lp` / `fill_gaps_kumar_milp`, `model.optimize`, — , `remove_genes` | ⚠️ portable, but `fillGaps` maps to three functions with different semantics, and `printFluxes` has no equivalent | + | Save and simulate | `model.annotation.*`, `newCommit`, `newRelease`, `solveLP` | YAML `metaData`, `export_for_git`, `model.optimize` | ⚠️ portable; `newCommit`/`newRelease` are hanpo-GEM repo helpers over `exportForGit`, so the Python lane needs its own equivalents | + | Manual curation | `changeGrRules`, `addRxns`, `addTransport`, `addRxnsGenesMets` | `change_gene_reaction_rules`, `add_reactions_from_equations`, `add_transport_reactions`, `add_reactions_from_model` | ✅ portable | + | Gap-analysis tip | `canProduce`, `canConsume`, `checkProduction`, `getAllSubGraphs`, `haveFlux` | `analyse_topology` + `cobra.flux_analysis.find_blocked_reactions` | ✅ portable | + +* **P1 — Then the two already-planned protocols**, each as its own nav section of chapter + pages: **GEM extraction** (ftINIT — adapt the existing good material from the + Human-GEM-guide rather than writing fresh, per the maintainer's instruction) and **GEM + comparison** (`compare_models` / `diff_models`, reporter metabolites). +* **P1 — New protocols beyond those.** Revised against what the reconstruction chapters + already cover — my earlier "draft to a model that grows" and "simulation and analysis" + proposals are **dropped**, since the gap-filling and save-and-simulate chapters cover that + ground: + * **Compartmentalisation** — `predictLocalization` vs `assign_compartments`, evidence + sources, reading the certification report. Largely Python-only. + * **Confidence tracking** — running the facets, using the bands for curation triage. + Python-only; currently only a study in this repo. + * **ecModel handoff** — YAML `ec-*` round-trip with geckopy. +* **P1 — Decide how Python-only protocols present.** Several of the above have no MATLAB + counterpart, which the MATLAB-default linked-tab convention does not cover. Options: a + "Python only" badge with the MATLAB tab explaining the absence, or a separate Python-only + grouping. Record the decision in the raven-docs decisions log. +* **P1 — Keep tutorial code executable.** The existing protocol transcludes a real `.m` + script, so its MATLAB lane cannot rot silently. The new tutorial needs that on both sides: + a runnable MATLAB script and a runnable Python script, version-controlled and transcluded + into the pages, exercised in CI (roadmap decision **D4** fixes where they live). Without + it the Python lane will drift exactly the way `differences.md` did. +* **P2 — Retire the thin capability guides in this repo** (`docs/guide/*.md`, 18–69 lines + each) once the protocols cover the same ground, leaving the API reference and the studies + here and the narrative on the site. + +## 4. Maturity checklist (→ raven-toolbox, surfaced in raven-docs) + +There is no single view of which functions are production-ready. The information exists +(tests, studies, `known_issues.md`) but is not aggregated, so users cannot tell a +five-times-validated path from a thin wrapper. + +* **P0 — Per-function maturity table** with an explicit rubric — **stable** (unit-tested + + validated on a real model + API frozen), **provisional** (tested, API may change), + **experimental** (works, little validation). Maintained here; published as its own small + page in the raven-docs differences section. +* **P0 — Coverage report in CI** with a floor, plus a first pass over the thin spots. + Modules not obviously exercised by any test module today, to confirm or fix: + `localization/transport_evidence.py`, `localization/triage.py`, + `localization/substrate_ontology.py`, `localization/transporter_tables.py`, + `manipulation/parameters.py`, `manipulation/boundary.py`, `biomass/*`, + `conditions/apply.py`, `curation/batch.py`, `annotation/delta_g.py`, `annotation/sbo.py`, + `reconstruction/kegg/assemble.py`, `reconstruction/homology/hits.py`. +* **P1 — Gaps the protocol port exposes.** `printFluxes` has no Python equivalent (a small, + obviously useful addition); `contractModel`'s grRule-merging semantics differ from + `remove_duplicate_reactions`; there is no Python counterpart to hanpo-GEM's + `newCommit`/`newRelease` convenience layer over `export_for_git`. Decide port-or-document + for each. +* **P1 — Docstring completeness for the generated API.** raven-docs renders this package's + NumPy docstrings via `mkdocstrings`/griffe, so a thin docstring here is a thin page there. + Audit the subsystems with no local API page — `confidence`, `biomass`, `conditions`, + `curation`, `annotation`, `manifest`, `data`, `binaries` — and add them to + `docs/reference/api/`. +* **P1 — State the known functional gaps in one place**: ftINIT metabolomics scoring + (`NotImplementedError`), no Excel import, no MetaCyc reconstruction, no dynamic FBA, + genome-scale (f)tINIT effectively requiring Gurobi. Feeds `differences/matlab-only.md`. +* **P1 — Implement or formally drop the 💡 proposals** in `improvements.md` (A4 compartment + inference from structured metabolite ids, Y4 a first-class home for + `deltaG`/`confidence_score`, R4 the `remove_metabolites` wrapper review, G7 + follow-through). +* **P2 — Stability policy for 0.x → 1.0**: what counts as public API, the deprecation + window, and how `notes` / `annotation` key names are versioned. + +## 5. Housekeeping + +* **P1 — Land or close the open PRs**: + [#75](https://github.com/SysBioChalmers/raven-toolbox/pull/75) (assignment ablation + benchmark) and [#85](https://github.com/SysBioChalmers/raven-toolbox/pull/85) (opt-in + deterministic ftINIT extraction, still draft). +* **P1 — Post-0.3.0 doc debt**: restore the documentation that PR #53 removed, and land the + unmerged empirical parameter-default results. +* **P2 — MATLAB-side back-port queue**: `assignCompartments.m` (plan already written in + `matlab_raven_backports.md`), plus the FS4 / B2 / G1 / G5 items. ## Upstream blockers (not raven-toolbox work, but worth tracking) -* `optlang.hybrid_interface.Configuration.clone()` bug — blocks HiGHS at any scale (CI catches - it in `tests/test_init_solvers.py`). -* GLPK's MIP solve ignores `configuration.timeout` at genome scale — blocks GLPK on large MILPs. -* Both documented in [init_solver_benchmark.md](../studies/init_solver_benchmark.md) with concrete fix - suggestions. +* `optlang.hybrid_interface.Configuration.clone()` bug — blocks HiGHS at any scale (CI + catches it in `tests/test_init_solvers.py`). +* GLPK's MIP solve ignores `configuration.timeout` at genome scale — blocks GLPK on large + MILPs. +* Both documented in [init_solver_benchmark.md](../studies/init_solver_benchmark.md) with + concrete fix suggestions. diff --git a/docs/studies/confidence_tracking.md b/docs/studies/confidence_tracking.md index 3570d527..d1814b3a 100644 --- a/docs/studies/confidence_tracking.md +++ b/docs/studies/confidence_tracking.md @@ -1,9 +1,9 @@ # Per-reaction confidence tracking -**Status: P1 + P2 shipped.** The data model, the notes round-trip (YAML + SBML), and the `localization`, -`equation` and `gene_association` scorers live in +**Status: the three facets are shipped.** The data model, the notes round-trip (YAML + SBML), and the +`localization`, `equation` and `gene_association` scorers live in [`raven_toolbox/confidence.py`](../../src/raven_toolbox/confidence.py) (tests in `tests/test_confidence.py`). -The `reversibility` facet (P3) and the standards mapping (P4) below remain planned. +The facet set is **closed** — see [§10](#10-what-is-left) for the remaining work. Every reaction carries a small structured record scoring how well-supported each of its **facets** is, persisted in the model file, computed from evidence, updated by curation, and consumed by the raven-toolbox @@ -43,7 +43,6 @@ no facet is left. Those two facts, taken together, force the design: | `localization` | the compartment assignment | DeepLoc support at the assigned compartment + FBA certification | | `equation` | mass & charge balance, formula completeness | `get_elemental_balance` + a recomputed charge sum | | `gene_association` | is there gene evidence, and is it corroborated | GPR presence + a `pubmed` annotation | -| `reversibility` *(planned)* | are the bounds thermodynamically justified | ΔG hook / FVA-attainable direction | Each facet is scored independently, so a model can be annotated one facet at a time and the record grows incrementally. A `ConfidenceEntry` is a continuous `score` in [0, 1] plus optional provenance: a categorical @@ -82,8 +81,8 @@ and must balance. Detecting biomass by *name* is likewise refused — `\bgrowth\ model carries **no** reaction SBO terms at all, the scorers warn: they then cannot tell a pseudo-reaction from a defect. -Note also that `reaction.boundary` is `len(metabolites) == 1`, independent of id, bounds and reversibility — -so it catches an exchange reaction that is not named `EX_`, and a blocked `(0, 0)` one. +Note also that `reaction.boundary` is `len(metabolites) == 1`, independent of id and of bounds — so it +catches an exchange reaction that is not named `EX_`, and a blocked `(0, 0)` one. ### Bands @@ -180,10 +179,11 @@ disagreements are informative rather than embarrassing: 43 gene-less reactions c ### What yeast-GEM cannot test -Three bands never fire on it, and are therefore exercised only by synthetic fixtures: `formula-unparseable` -(no parenthesised formulas), `formula-generic` (no residual lands in an R group), and `charge-unknown` (all -2748 metabolites carry a charge). This is stated rather than hidden — a distribution measured on one model is -not a guarantee about another. +Three bands never fire on it: `formula-unparseable` (no parenthesised formulas), `formula-generic` (no +residual lands in an R group), and `charge-unknown` (all 2748 metabolites carry a charge). This is stated +rather than hidden — a distribution measured on one model is not a guarantee about another. The cross-model +sweep in [§10.2](#102-validate-beyond-one-model) resolves it: `formula-generic` and `formula-unparseable` +both fire on other real models; only `charge-unknown` stays exercised solely by synthetic fixtures. ## 7. Integration with the existing tools @@ -204,30 +204,76 @@ not a guarantee about another. **Shipped:** `ConfidenceEntry`, `ReactionConfidence`, `get_confidence` / `set_confidence` / `clear_confidence`, `read_confidence`, `mark_curated`, `equation_exempt`, `gene_association_exempt`, `score_localization_confidence`, `score_equation_confidence`, `score_gene_association_confidence`, -`confidence_report`, `facet_summary`. Storage lives in `reaction.notes["raven_confidence"]`; there is no -separate save step — the record serialises with the model. +`annotate_confidence`, `confidence_report`, `facet_summary`. Storage lives in +`reaction.notes["raven_confidence"]`; there is no separate save step — the record serialises with the model. -**Planned (P3):** `score_reversibility_confidence` and an umbrella `annotate_confidence(model, types=[...])`. +## 9. Standards alignment (for the paper) -## 9. Standards alignment (P4, for the paper) +The **Thiele & Palsson reconstruction confidence score (0–4)** classifies a reaction by the *type* of +evidence for including it. It therefore maps from the `gene_association` facet — the one that measures +reaction-inclusion evidence — via its `basis`, **not** from a facet's `level` (a support-strength band) or +the `overall` score. `raven_toolbox.confidence.thiele_palsson_score(reaction)` returns it: -Map the categorical `level` onto the established **Thiele & Palsson reconstruction confidence score (0–4)** so -it is familiar to modellers and reviewers, and reference **ECO** (Evidence & Conclusion Ontology) terms where a -facet maps to an evidence class. Two cautions carried forward from the design work: +| `gene_association` basis | Thiele-Palsson | ECO evidence class | +|---|---|---| +| `gpr+literature` | 3 — experimental / genetic | assign from the cited evidence, e.g. `ECO:0000015` (mutant phenotype) or `ECO:0000002` (direct assay); not inferable from the presence of a citation alone | +| `gpr` | 2 — sequence data | `ECO:0000044` sequence similarity evidence | +| `no-gpr` | 1 — modelling inference | — | +| `curated` | set by the curator's own evidence | set by the curator's own evidence | + +Two cautions govern the mapping: - Thiele & Palsson's table assigns score **2 to two different evidence classes** (physiological data *and* - sequence data), so a recorded 2 does not determine an evidence class and must not be mapped to one. -- Assertion-method ECO terms (e.g. "inferred by curator") are not evidence classes and must not be used as - such. Any ECO id must be checked against the ontology before it reaches the paper — a wrong ontology id is - worse than an omitted one. The SBO ids used above were each verified against EBI's SBO: `SBO:0000629` - biomass production, `SBO:0000395` encapsulating process, `SBO:0000630` ATP maintenance, `SBO:0000672` - spontaneous reaction, `SBO:0000655` transport reaction. - -## 10. Phasing - -- **P1 — foundation + the facet we already had** *(shipped)*: data model, storage/round-trip helpers (YAML - **and** SBML), the `localization` scorer, `confidence_report`. -- **P2 — cheap structural facets** *(shipped)*: `equation` (mass/charge/formula) + `gene_association`, the - abstain-vs-zero discipline, `facet_summary`, and the exemption predicates. -- **P3 — reversibility:** the bounds-vs-FVA heuristic, with a ΔG hook. -- **P4 — paper:** Thiele-Palsson / ECO / SBO mapping and documentation. + sequence data), so the mapping only runs *forward* (basis → score); a recorded 2 does not determine an + evidence class and must not be mapped back to one. `curated` is likewise left unmapped — a curator's + assertion does not, on its own, name the evidence class it rests on. +- Assertion-method ECO terms are **not** evidence classes and must not be used as such: `ECO:0000305` + "curator inference used in manual assertion" describes *how* a call was made, not the evidence for it. Any + ECO id must be checked against the ontology before it reaches the paper — a wrong ontology id is worse than + an omitted one. Every ECO id above was verified against EBI's ECO (OLS): `ECO:0000044` sequence similarity + evidence, `ECO:0000015` mutant phenotype evidence, `ECO:0000002` direct assay evidence. The SBO ids used + earlier were likewise verified: `SBO:0000629` biomass production, `SBO:0000395` encapsulating process, + `SBO:0000630` ATP maintenance, `SBO:0000672` spontaneous reaction, `SBO:0000655` transport reaction. + +## 10. What is left + +The **facet set is closed**: `localization`, `equation` and `gene_association` are shipped, and no further +facet is planned. The wiring (§10.1), the standards mapping (§10.3) and the cross-model validation (§10.2) +are done; what remains is the paper write-up. + +### 10.1 Wire the facets together — done + +- **`annotate_confidence(model, *, proposal=None, scores=None, facets=None)`** runs every applicable scorer + in one call and skips a facet whose inputs are absent (`localization` needs a proposal and a score table; + the other two need only the model) rather than failing — the same abstain-rather-than-guess rule the + scores themselves follow. +- **`curation_priority` now skips curated placements.** A reaction whose `localization` facet is + `mark_curated` is dropped from the review queue (pass `include_curated=True` to keep it), closing the + score → review → curate loop so a settled reaction stops resurfacing. +- **The SBO-precondition warning now names its remedy** — `raven_toolbox.annotation.add_sbo_terms(model)`. + +### 10.2 Validate beyond one model + +Done. `scripts/measure_confidence_facets.py` (now reads SBML or YAML/JSON) was run on three more models — +**Human-GEM** (12 877 rxns), **iYali** (1924) and **panAsp/pAo** (1917) — vs yeast-GEM's §6 baseline: + +| equation band | yeast-GEM | Human-GEM | iYali | pAo | +|---|---|---|---|---| +| `formula-generic` (0.3) | 0 | 66 | 40 | 26 | +| `formula-unparseable` (0.3) | 0 | 0 | 8 | 67 | +| `charge-unknown` (0.6) | 0 | 0 | 0 | 0 | + +Two of the three bands yeast-GEM could not exercise fire on other real models, so they are calibrated, not +yeast artefacts. Only `charge-unknown` stays fixture-only — a mass-balanced reaction with a charge-less +metabolite is rare (curated models charge everything; drafts fail mass balance first). + +`gpr+literature` (0.9) fires only where reactions carry a `pubmed` annotation (iYali 749; the others 0). The +gene-rubric-vs-`Confidence Level` check needs a per-reaction-curated model: iYali and pAo record a blanket +value (all 2, all 0), so only yeast-GEM's varied notes are usable ground truth. + +### 10.3 Standards alignment for the paper — mapping shipped + +§9 above. The Thiele-Palsson mapping is shipped as `raven_toolbox.confidence.thiele_palsson_score` +(`gene_association` basis → 0–4), and its ECO evidence-class ids are verified against OLS. What remains is +paper-side, not code: assigning the specific ECO class for `gpr+literature` and `curated` reactions from the +actual cited/curator evidence (which the facet cannot infer), and the write-up itself. diff --git a/docs/studies/ftinit_determinism.md b/docs/studies/ftinit_determinism.md new file mode 100644 index 00000000..15d6ebf4 --- /dev/null +++ b/docs/studies/ftinit_determinism.md @@ -0,0 +1,142 @@ +# ftINIT extraction determinism — `strict_gap` and `canonical` + +What the two opt-in determinism flags on `ftinit()` actually buy on a genome-scale model, +and — the reason this page exists — what they *do not* buy. Companion to the flags' +API documentation in `raven_toolbox.init.ftinit`. + +## The problem + +The ftINIT extraction MILP is highly degenerate: many different reaction subsets reach the +same score optimum, and the solver returns whichever one its branch-and-cut happens to +land on. raven-toolbox already matches RAVEN's reproducibility levers — `Threads=1` (the +dominant one; multi-threaded Gurobi picks among equal optima non-deterministically) and a +fixed `Seed` — but those make the choice **reproducible**, not **unique**. The chosen +optimum is a property of the solver build, so a different Gurobi version, platform, or +seed lands on a different one. Downstream that surfaces as gene-essentiality predictions +that drift for reasons unrelated to the model or the data. + +The two flags attack that directly: + +* **`strict_gap`** — one solve per step proven to a fixed *absolute* gap (0.05, below the + 0.1 reaction-score granularity), replacing RAVEN's loose relative-gap escalation, whose + final near-zero-objective run otherwise accepts an arbitrary within-gap incumbent. +* **`canonical`** — a lexicographic phase 2 over the removable (binary) reactions: hold the + score objective at its optimum, minimise the count of kept reactions, then their summed + id rank. Applied to both each extraction step and the task gap-fill. + +Both default to off, so the shipped behaviour is unchanged. + +## Method + +* **Model / data.** Human-GEM v2.0.0, Hart2015 DLD1 cell line, the same prep and scoring + as the [Human-GEM validation](humangem_validation.md) run (series `1+1`, tasks on). +* **Solver.** Gurobi 13, `Threads=1`, single machine. +* **Determinism probe.** The extraction seed (`ftinit._EXTRACT_SEED`, RAVEN's 1234) is + changed to 7 and the model rebuilt. Everything else — inputs, code, solver build, + machine — is held fixed, so any difference between the two models is *purely* the + solver's tie-break among equal optima. This is a lower bound on cross-version drift: a + Gurobi upgrade perturbs the search far more than a seed change does. +* **Metrics.** (1) *reaction seed-swing* — the size of the symmetric difference between the + two kept-reaction sets; (2) *essential-gene seed-swing* — the symmetric difference + between the two single-gene-deletion essential sets. The second is the metric that + matters for users; the first is the one the flags act on. + +Each cell below is one build per seed (two builds per row). This is an n=1 probe on one +cell line with one seed pair, not a replicated benchmark — see *Limitations*. + +## Results + +| Configuration | Reaction seed-swing | Essential-gene seed-swing | Build time | +|---|---:|---:|---:| +| Baseline (no flags) | 125 | **5** | ~12 min | +| `canonical` only | 64 | not measured | ~25 min | +| `canonical` + `strict_gap` | **34** | 19 | ~34–80 min | + +Out of ~7766 kept reactions and ~281 essential genes. The five baseline essentiality flips +are PC, ACADVL, DUT, ALDH3A2 and RDH5. Essential-gene counts: 281 / 278 for the two +baseline seeds, 283 / 296 for the two `canonical`+`strict_gap` seeds. + +## Findings + +**The flags do what they claim at the reaction level.** The seed-swing drops 125 → 64 with +`canonical` alone and → 34 with both, a 3.7× reduction. The remaining 34 reactions are the +degeneracy the lexicographic phase 2 cannot see: it canonicalises the *removable-reaction +choice*, while a residual flux-distribution degeneracy (which feeds the next step's small +`ess_force` clamp) is left to the solver. + +**But the reaction metric is a poor proxy for what users care about.** All 34 residual +reactions are GPR-less — mostly transport. They move without touching any gene, so driving +that number down does not, by itself, stabilise anything gene-level. + +**And gene-essentiality determinism got *worse*, not better: 5 → 19 flips.** Two effects +compound: + +1. `canonical` minimises the kept set, so it produces **sparser** models (283 / 296 + essential genes vs the baseline's 281 / 278). A sparser network has fewer alternative + routes, so more genes sit on a unique path and become essential — and more of them are + sensitive to whichever residual alternative pathway survives. The transport-level + degeneracy that the reaction metric dismisses as "GPR-less" propagates into enzymatic + genes' essentiality through connectivity. +2. `strict_gap` **cannot prove the genome-scale optimum in reasonable time.** One build in + this probe accumulated ~67 min of per-step timeouts and fell back to arbitrary + incumbents — exactly the behaviour the flag exists to eliminate, and it adds noise of + its own. The 34–80 min build-time range in the table is that fallback showing up as + variance. + +The runtime cost is real regardless: 2× for `canonical`, 3–7× for both. + +## Conclusions + +* These are a correct, tested, opt-in tool for a **more parsimonious and more reproducible + extraction**. They pin *which* of many equally data-consistent optima is returned; they + do not make the model biologically more accurate, and they reduce the MILP's fragility + rather than its error. +* They are **not** a fix for gene-essentiality determinism, and on this benchmark they made + it worse. Do not reach for them to stabilise a downstream essentiality analysis. +* **For run-to-run essentiality reproducibility the reliable lever is pinning the solver + stack** — raven-toolbox commit plus `gurobipy` version — which gives bit-identical + results without any of the runtime cost. +* Reasonable use: producing one or a few stable, parsimonious model artifacts where the + extra build time is affordable and the parsimony is wanted for its own sake. For most + work the baseline is fine. +* Worth testing if the flags are pursued further: `canonical` **without** `strict_gap` + (untested at gene level here) isolates parsimony from the timeout noise, and is the + cheaper of the two. Fixing the essentiality drift likely needs the residual transport + degeneracy canonicalised too, not a tighter primary gap. + +## Limitations + +One cell line, one seed pair, one build per configuration, one machine — enough to +establish the direction and rough magnitude of each effect, not to put error bars on any +number. The `canonical`-only gene-essentiality figure was never measured, so the +attribution of the 5 → 19 regression between the two flags is inferred from the mechanism, +not measured. Cross-Gurobi-version drift — the failure mode the flags actually target — was +not measured at all; the seed probe stands in for it. + +## Reproducing + +Requires Gurobi and the Human-GEM work directory from the +[Human-GEM validation](humangem_validation.md) run. + +```python +import cobra +from raven_toolbox.init import ftinit +from raven_toolbox.init import ftinit as ftinit_mod + +def build(seed, **flags): + ftinit_mod._EXTRACT_SEED = seed # determinism probe; not a public parameter + model = ftinit(prep, rxn_scores=scores, time_limit=900, **flags) + return {r.id for r in model.reactions}, model + +for label, flags in [("baseline", {}), + ("canonical", {"canonical": True}), + ("canonical+strict", {"canonical": True, "strict_gap": True})]: + a, ma = build(1234, **flags) + b, mb = build(7, **flags) + print(label, "reaction swing:", len(a ^ b)) + ea = {g.id for g in cobra.flux_analysis.find_essential_genes(ma)} + eb = {g.id for g in cobra.flux_analysis.find_essential_genes(mb)} + print(label, "essential-gene swing:", len(ea ^ eb), f"({len(ea)} vs {len(eb)})") +``` + +Budget several hours: the `canonical`+`strict_gap` pair alone is 1–3 h of solve time. diff --git a/docs/studies/homology_cutoff_calibration.md b/docs/studies/homology_cutoff_calibration.md new file mode 100644 index 00000000..bf0ecf60 --- /dev/null +++ b/docs/studies/homology_cutoff_calibration.md @@ -0,0 +1,136 @@ +# Homology cut-off calibration + +Three settings decide which template reactions transfer to a new organism. None +of them had ever been measured, in either toolbox. This is what they are worth. + +| Setting | Was | Now | Why | +|---|---|---|---| +| `min_identity` | 40 | **40** | The one that matters. 40 is right. | +| `min_align_len` | 200 | **100** | 200 was throwing away real matches for nothing. | +| `max_evalue` | 1e-30 | **1e-30** | Makes no difference anywhere between 1e-4 and 1e-50. | + +`min_align_len` now differs from MATLAB RAVEN, which still uses 200. Worth +back-porting. + +## How it was tested + +Reconstruct four organisms from an *S. cerevisiae* template, at a range of +settings, and ask of every match that survives: are these two genes really +counterparts? + +Two independent sources answer that — KEGG's gene annotations and OMA's ortholog +assignments — and wrong matches count double, because a wrong reaction is worse +than a missing one: gap-filling can add what is absent, while something wrong is +hard to notice and harder to remove. + +The organisms span a range of relatedness, because a setting that suits a close +relative need not suit a distant one: + +| | Organism | Relation to yeast | +|---|---|---| +| `kla` | *K. lactis* | close | +| `yli` | *Y. lipolytica* | moderate | +| `ani` | *A. nidulans* | distant | +| `eco` | *E. coli* | very distant | + +Proteomes come from UniProt, relabelled with KEGG gene names so both references +can be applied to the same matches. + +## Sequence identity: keep 40 + +Scored against KEGG annotations (higher is better): + +| identity | `kla` | `yli` | `ani` | `eco` | +|---|---|---|---|---| +| 25 | 0.861 | 0.833 | 0.771 | 0.460 | +| 30 | 0.896 | 0.851 | 0.796 | 0.518 | +| 35 | 0.917 | **0.868** | **0.814** | 0.558 | +| **40** | **0.923** | 0.860 | 0.804 | **0.598** | +| 45 | 0.907 | 0.813 | 0.735 | 0.520 | +| 50 | 0.882 | 0.736 | 0.640 | 0.323 | + +Against OMA the best value is 45 for the three fungi and 35 for *E. coli*. So the +two sources put the answer between 35 and 45, and 40 sits comfortably inside +that. Neither supports anything looser. + +That last point is worth stating plainly, because counting a missing match as +equally bad as a wrong one moves the recommendation to 25 — a completely +different answer from the same measurements. Any recommendation about these +settings is meaningless unless it says how it weighed the two kinds of mistake. + +## Alignment length: 200 was too strict + +| length | `kla` | `yli` | `ani` | `eco` | +|---|---|---|---|---| +| 50 | 0.933 | 0.871 | 0.815 | 0.618 | +| **100** | **0.933** | **0.871** | **0.815** | **0.618** | +| 150 | 0.932 | 0.869 | 0.813 | 0.615 | +| 200 *(old)* | 0.923 | 0.860 | 0.804 | 0.598 | +| 300 | 0.885 | 0.816 | 0.764 | 0.571 | + +Anything at or below 150 performs the same; the loss appears between 150 and +200. Dropping to 100 recovers 3–4% more real matches on every organism while the +wrong-match rate moves by at most 0.6 of a percentage point. 50 and 100 are +identical, so 100 was chosen as the less permissive of the two. + +## E-value: not a setting worth touching + +| `max_evalue` | `kla` | `ani` | +|---|---|---| +| 1e-4 … 1e-50 | 0.923 | 0.804 (identical throughout) | +| 1e-100 | 0.877 | 0.703 | + +Five orders of magnitude, one answer. Identity and length have already removed +whatever a looser e-value would have removed, so it has nothing left to decide. +Only at 1e-100 does it start discarding good matches. Kept at 1e-30 to match +RAVEN. + +## DIAMOND behaves the same as BLAST + +Both toolboxes offer DIAMOND as the fast alternative and apply these same three +settings to its output, which is only safe if the two aligners behave alike. +They do: + +| Organism | best with BLAST | best with DIAMOND | score at that point | +|---|---|---|---| +| `kla` | 40 | 40 | 0.923 vs 0.921 | +| `yli` | 35 | 35 | 0.868 vs 0.865 | +| `ani` | 35 | 35 | 0.814 vs 0.811 | +| `eco` | 40 | 40 | 0.598 vs 0.593 | + +DIAMOND finds about half as many matches — 29,785 against 54,478 on `kla` — but +almost all the missing ones are weak ones these settings discard anyway. After +filtering, the two agree on 87–92% of what survives, and DIAMOND runs 10–20× +faster (34 s against 358 s on `kla`; 40 s against 788 s on `ani`). + +So one shared set of defaults is justified, and DIAMOND is a reasonable choice +for large jobs. + +## What the numbers do not cover + +- **KEGG's annotations lean on BLAST comparisons**, so on their own they would + partly measure agreement with the method being tested. OMA infers counterparts + independently, which is why both were used; they agree. +- **The two sources are not directly comparable in absolute terms.** KEGG only + covers genes it has annotated — a few hundred well-studied ones per organism — + while OMA covers whole proteomes and lists strict counterparts, so ordinary + gene duplicates count against us. Compare the shape of each column, not the + heights between them. +- **The *Y. lipolytica* OMA row is the weakest**: 38% of its OMA pairs could not + be matched to KEGG gene names, so that reference is incomplete. +- **One template.** Everything was measured outward from *S. cerevisiae*. +- **Measured on gene matches, not finished models.** A reaction transfers if any + one of its genes matches, so the effect on a finished model is smaller than + these numbers suggest. + +## Reproducing + +```bash +python scripts/homology_cutoff_kegg.py fetch --out work/ +python scripts/homology_cutoff_kegg.py align --out work/ +python scripts/homology_cutoff_kegg.py score --out work/ --beta 0.5 \ + --gene-ko kegg118_organism_gene_ko.tsv.gz +``` + +Aligning is the only slow part and is saved, so re-scoring — under a different +weighting, or against a different reference — takes about two minutes. diff --git a/docs/studies/index.md b/docs/studies/index.md index a230d040..a73d6225 100644 --- a/docs/studies/index.md +++ b/docs/studies/index.md @@ -10,6 +10,9 @@ its equivalence claims against MATLAB RAVEN. scaling sweeps; dropout / noise / downsample robustness). - **[(f)tINIT solver benchmark](init_solver_benchmark.md)** — Gurobi vs HiGHS vs GLPK on genome-scale ftINIT. +- **[ftINIT extraction determinism](ftinit_determinism.md)** — what the opt-in `strict_gap` / + `canonical` flags buy on Human-GEM/DLD1: a 3.7× smaller reaction seed-swing, but *worse* + gene-essentiality determinism (5 → 19 flips) and a 3–7× build-time cost. - **[Yeast localization benchmark](yeast_localization_benchmark.md)** — `predict_localization` against curated yeast-GEM, with a predictor-noise sweep. - **[Compartment-assignment redesign](localization_redesign.md)** — the design of @@ -51,6 +54,10 @@ its equivalence claims against MATLAB RAVEN. accuracy-neutral, so normalisation stays the default and `normalise=False` is opt-in. - **[KEGG HMM cut-off calibration](kegg_hmm_cutoff_calibration.md)** — HMM E-value / score-ratio sensitivity for the KEGG HMM-query reconstruction path. +- **[Homology cut-off calibration](homology_cutoff_calibration.md)** — what the three + homology filters are worth, measured against two independent references across four + organisms. `min_identity` 40 confirmed, `min_align_len` lowered 200 → 100, `max_evalue` + shown to make no difference at all, and DIAMOND shown to need the same settings as BLAST. ```{toctree} :hidden: @@ -58,6 +65,7 @@ its equivalence claims against MATLAB RAVEN. humangem_validation init_param_calibration init_solver_benchmark +ftinit_determinism yeast_localization_benchmark deeploc_yeast_benchmark deeploc_aracore_benchmark @@ -69,6 +77,7 @@ carvefungi_analysis carvefungi_milp_benchmark deeploc_normalisation_benchmark kegg_hmm_cutoff_calibration +homology_cutoff_calibration localization_redesign curation_priority_signals yeast_validation diff --git a/docs/studies/localization_redesign.md b/docs/studies/localization_redesign.md index 966caa6c..a2a5e751 100644 --- a/docs/studies/localization_redesign.md +++ b/docs/studies/localization_redesign.md @@ -182,6 +182,13 @@ Reading: - **The honest agreement-vs-transport frontier is now measurable.** 72.5 % agreement costs ~1261 reconnecting transports (983 FVA-usable), against curated yeast-GEM's own 1467 inter-compartment reactions — high agreement honestly requires many transports (there is no free lunch). +- **Reaction placement is deterministic and score-aligned, not an arbitrary co-optimum.** The master's + objective is over gene-localisation variables only, so each reaction's compartment is a free + co-optimum — a pinned solver returns it reproducibly, but *arbitrarily* (an un-tie-broken solve scored + only 52.8 %). A lexicographic second pass fixes the gene layout to the primary optimum, then places + each reaction in the compartment its own enzymes are predicted to occupy (summed DeepLoc score of the + reaction's genes, small default-compartment prior for genes-free and tied reactions). The gene layout + is untouched, so the 72.5 % rests on the localisation evidence itself, not on a solver's tie-break. Every number is authoritative because the growth column *is* the certificate: there is no separate MILP floor to disagree with it. Raw JSON in `.research_tmp/certify_yeast_gem.json`. diff --git a/docs/studies/multiorganism_validation.md b/docs/studies/multiorganism_validation.md index 9e725010..f72d0196 100644 --- a/docs/studies/multiorganism_validation.md +++ b/docs/studies/multiorganism_validation.md @@ -14,10 +14,20 @@ only a compartment-label map and a metabolite base-key per organism. | organism | kingdom | model | reactions | compartments | chloroplast | wall | certified | reaction agr | gene agr | transports | blocked | |---|---|---|--:|--:|:--:|--:|:--:|--:|--:|--:|--:| -| *S. cerevisiae* | fungal | yeast-GEM | 2569 | 14 | no | ~140 s | **yes** | 72.0 % | **88.7 %** | 1001 | 29.7 % | -| *H. sapiens* | mammalian | Human-GEM | 12854 | 9 | no | 190–464 s | **yes** | 52.8 % | 79.4 % | 3701 | 10.7 % | -| *A. thaliana* | plant | AraCore v2.1 | 585 | 6 | **yes** | 35 s | **yes** | **81.8 %** | **93.7 %** | 215 | 0.0 % | -| *C. reinhardtii* | algal | iCre1355 | 2394 | 11 | **yes** | 93 s | **yes** | 53.4 % | 70.5 % | 1108 | 19.0 % | +| *S. cerevisiae* | fungal | yeast-GEM | 2569 | 14 | no | ~140 s | **yes** | 72.5 % | **88.7 %** | 967 | 29.7 % | +| *H. sapiens* | mammalian | Human-GEM | 12877 | 9 | no | 190–464 s | **yes** | 56.9 % | 79.4 % | 3608 | 10.7 % | +| *A. thaliana* | plant | AraCore v2.1 | 585 | 6 | **yes** | 35 s | **yes** | **82.1 %** | **93.7 %** | 200 | 0.0 % | +| *C. reinhardtii* | algal | iCre1355 | 2394 | 11 | **yes** | 93 s | **yes** | 54.1 % | 70.5 % | 1083 | 32.9 % | + +All rows are measured on the **deterministic score-aligned placement** (each reaction placed by its own +enzymes' DeepLoc scores; see [yeast_validation.md](yeast_validation.md)), not the earlier arbitrary +co-optimum. **Gene** agreement is unchanged from the arbitrary-placement runs — the tie-break fixes only +the reaction placement, leaving the gene layout untouched — which is why every gene-agreement figure +above matches its pre-tie-break value. Reaction agreement rose or held (Human-GEM 52.8 % → 56.9 % the +largest move; AraCore and iCre1355 were already near their ceilings), and coherent placement needs fewer +transports across the board. The one non-obvious shift is iCre1355's blocked fraction (19.0 % → 32.9 %): +tighter, less transport-heavy placement leaves more reactions dead-ended, though the model still +certifies. Placement is reproducible run-to-run. Agreement is scored against each model's **own** compartment annotation (a *circular* truth — the model being re-placed is also the reference). This measures self-consistency and functionality, not accuracy; @@ -26,22 +36,22 @@ Chloroplast Protein Atlas, UniProt experimental across all) replaces it for the ## What P1 establishes -1. **It scales past 4 compartments at genome size.** Human-GEM — 12854 reactions, **9 compartments**, - 5206 reactions re-placed — certifies (materialised biomass above the floor by a real FBA) in +1. **It scales past 4 compartments at genome size.** Human-GEM — 12877 reactions, **9 compartments**, + 5221 reactions re-placed — certifies (materialised biomass above the floor by a real FBA) in ~3–8 min. iCre1355 reaches **11 compartments** (the richest organelle set here) in ~1.5 min. The flux-free master + FBA-certification design does not degrade as compartments and reactions grow; this was the open scaling risk, now closed. 2. **It handles the chloroplast** — the generality hook CarveFungi structurally cannot represent. DeepLoc's "Plastid" label maps to the chloroplast code (`h` in both plant models), and both AraCore - (plant) and iCre1355 (alga) place and certify with a chloroplast, AraCore at **81.8 % reaction / + (plant) and iCre1355 (alga) place and certify with a chloroplast, AraCore at **82.1 % reaction / 93.7 % gene** self-agreement. 3. **It is organism- and predictor-agnostic in practice, not just in principle.** The same function, the same DeepLoc-2 evidence, no algorithm changes — only a `{DeepLoc-label -> compartment-code}` map and a metabolite base-key (name-based for yeast/Human-GEM; bracket-stripped `Glc[c]->Glc` for AraCore; suffix-stripped `10fthf_c->10fthf` for iCre1355) — carry it from a 585-reaction plant core - model to a 12854-reaction human GEM. + model to a 12877-reaction human GEM. 4. **Every model stays functional.** All four materialise a growing model (blocked-reaction fractions - 0–30 %, mostly the models' intrinsic dead-ends), the property that distinguishes this method. + 0–33 %, mostly the models' intrinsic dead-ends), the property that distinguishes this method. ## Honest caveats diff --git a/docs/studies/yeast_validation.md b/docs/studies/yeast_validation.md index a5cc6729..044fa6e1 100644 --- a/docs/studies/yeast_validation.md +++ b/docs/studies/yeast_validation.md @@ -21,19 +21,29 @@ parent organelle, matching DeepLoc's resolution). | metric | `assign_compartments` | |---|--:| -| reaction-level agreement | **72.0 %** (1400/1943) | +| reaction-level agreement | **72.5 %** (1408/1943) | | gene-level agreement | **88.7 %** (716/807) | -| transports added | 1001 (curated 1467) | +| transports added | 967 (curated 1467) | | blocked re-placed reactions | 29.7 % | | materialised growth | 0.1426 /h (floor 0.040) | | certified (real FBA) | **yes** | Gene agreement (88.7 %) sits well above DeepLoc's ~64 % raw ceiling because functionality corrects -placements; the model stays functional because the method provisions the ~1000 transports the placement +placements; the model stays functional because the method provisions the ~970 transports the placement actually needs, and the blocked 29.7 % is yeast-GEM's *intrinsic* dead-end floor, not a placement artefact. Every number is authoritative because the growth column is a real FBA on the materialised model — the certificate, not a separate solver floor. +Reaction placement is **deterministic and score-aligned.** The placement master maximises a +gene-localisation objective, which never mentions the per-reaction placement variable — so each +reaction's compartment was a free co-optimum: reproducible once the solver is pinned, but an *arbitrary* +vertex (which on its own scored only 52.8 %). A lexicographic second pass fixes the gene layout to the +primary optimum, then places each reaction in the compartment its own enzymes are predicted to occupy +(the summed DeepLoc score of the reaction's genes, with a small default-compartment prior for genes-free +and score-tied reactions). The gene layout — hence gene agreement — is untouched; the 72.5 % reaction +agreement now rests on the localisation evidence itself, not on a solver's arbitrary tie-break, and the +coherent placement needs fewer transports (967 vs 1001). + ## 2. Head-to-head with CarveFungi (Comparison 2) CarveFungi's own `minmax_reduction` carve (CPLEX, unmodified) on its real universal-DB candidate set + diff --git a/pyproject.toml b/pyproject.toml index 3545bacb..c2b6f07b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "raven-toolbox" -version = "0.3.0" +version = "0.4.0" description = "Reconstruction, Analysis and Visualization of Metabolic Networks in Python, a port of the RAVEN Toolbox built on cobrapy" readme = "README.md" requires-python = ">=3.11" @@ -71,6 +71,10 @@ packages = ["src/raven_toolbox"] [tool.pytest.ini_options] testpaths = ["tests"] +markers = [ + "parity: cross-language checks against MATLAB RAVEN (see tests/parity/README.md)", + "slow: needs a genome-scale model and a solver licence that accepts it; run nightly, not in the pull-request suite", +] [tool.ruff] line-length = 100 diff --git a/scripts/homology_cutoff_kegg.py b/scripts/homology_cutoff_kegg.py new file mode 100644 index 00000000..784e3687 --- /dev/null +++ b/scripts/homology_cutoff_kegg.py @@ -0,0 +1,345 @@ +#!/usr/bin/env python +"""Driver for the KEGG arm of the homology cut-off study. + +Reproduces docs/studies/homology_cutoff_calibration.md: fetch proteomes keyed by +KEGG gene id, align the template against each target once, then score every +threshold combination against KEGG's own orthology (do the two genes share a KO). + + python scripts/homology_cutoff_kegg.py fetch --out work/ + python scripts/homology_cutoff_kegg.py align --out work/ + python scripts/homology_cutoff_kegg.py score --out work/ \\ + --gene-ko kegg118_organism_gene_ko.tsv.gz + +The alignment is the only slow step (~6-13 min per pair) and is cached, so +re-scoring with a different loss function costs nothing. +""" +from __future__ import annotations + +import argparse +import collections +import gzip +import json +import pathlib +import sys +import time +import urllib.request + +TEMPLATE = "sce" + +# KEGG organism code -> UniProt reference proteome. The code names a *strain*, so +# these pairings are checked by the KEGG cross-reference prefix, not assumed. +PROTEOMES = { + "sce": "UP000002311", # S. cerevisiae S288C -- template + "kla": "UP000000598", # close + "yli": "UP000001300", # medium + "ani": "UP000000560", # distant + "eco": "UP000000625", # very distant +} + +# OMA is keyed by taxon id, except where only its own genome code resolves. +OMA_TEMPLATE = 559292 +OMA_TARGETS = {"kla": 284590, "yli": 284591, "ani": 227321, "eco": "ECOLI"} + +BASE = {"max_evalue": 1e-30, "min_align_len": 200, "min_identity": 40} +GRID = { + "min_identity": [20, 25, 30, 35, 40, 45, 50, 60], + "min_align_len": [50, 100, 150, 200, 300], + "max_evalue": [1e-100, 1e-50, 1e-30, 1e-10, 1e-4], +} + + +def _stream(url: str) -> str: + with urllib.request.urlopen(url, timeout=900) as fh: + return fh.read().decode("utf-8") + + +def cmd_fetch(args: argparse.Namespace) -> int: + """Download each proteome and relabel its headers with KEGG gene ids.""" + out = args.out / "proteomes" + out.mkdir(parents=True, exist_ok=True) + + for code, upid in PROTEOMES.items(): + target = out / f"{code}.faa" + if target.exists(): + print(f"{code}: present, skipping") + continue + + mapping = {} + rows = _stream( + f"https://rest.uniprot.org/uniprotkb/stream?query=proteome:{upid}" + f"&format=tsv&fields=accession,xref_kegg" + ).splitlines()[1:] + for line in rows: + parts = line.split("\t") + if len(parts) < 2: + continue + for entry in filter(None, (p.strip() for p in parts[1].split(";"))): + prefix, _, gene = entry.partition(":") + # Exact code only: K-12 entries carry both eco: and ecj:, and + # "whichever is commonest" would pick a different genome. + if prefix == code: + mapping[parts[0]] = gene + break + + fasta = _stream( + f"https://rest.uniprot.org/uniprotkb/stream?query=proteome:{upid}&format=fasta" + ) + kept, dropped, lines, emit = 0, 0, [], False + for line in fasta.splitlines(): + if line.startswith(">"): + parts = line[1:].split("|") + accession = parts[1] if len(parts) > 2 else parts[0] + gene = mapping.get(accession) + emit = gene is not None + if emit: + lines.append(f">{gene}") + kept += 1 + else: + dropped += 1 + elif emit: + lines.append(line) + + target.write_text("\n".join(lines) + "\n", encoding="utf-8") + coverage = kept / (kept + dropped) if kept + dropped else 0.0 + print(f"{code}: {kept} kept, {dropped} unmapped (coverage {coverage:.1%})") + return 0 + + +def cmd_align(args: argparse.Namespace) -> int: + """One bidirectional alignment per (template, target) pair, cached.""" + from raven_toolbox.reconstruction.homology import run_blast, run_diamond + + prot = args.out / "proteomes" + prefix = "hits" if args.aligner == "blast" else f"hits_{args.aligner}" + for code in PROTEOMES: + if code == TEMPLATE: + continue + target = args.out / f"{prefix}_{code}.csv" + if target.exists(): + print(f"{code}: present, skipping") + continue + started = time.perf_counter() + if args.aligner == "blast": + hits = run_blast( + code, prot / f"{code}.faa", [TEMPLATE], [prot / f"{TEMPLATE}.faa"], + evalue=args.collection_evalue, threads=args.threads, + ) + else: + hits = run_diamond( + code, prot / f"{code}.faa", [TEMPLATE], [prot / f"{TEMPLATE}.faa"], + threads=args.threads, + ) + hits.to_csv(target, index=False) + print(f"{code}: {len(hits)} hits in {time.perf_counter() - started:.0f}s") + return 0 + + +def _oma_truth(code: str, target_map: dict, source_map: dict, cache_dir: pathlib.Path): + """Ortholog pairs from OMA, as KEGG gene names. + + OMA infers orthologs independently of BLAST, so it answers the objection that + KEGG's annotations partly reflect the method under test. Its identifiers are + UniProt, hence the mapping through both sides. + """ + import re + + cache = cache_dir / f"oma_pairs_{code}.json" + if cache.is_file(): + raw = json.loads(cache.read_text(encoding="utf-8")) + else: + raw, url = [], ( + f"https://omabrowser.org/api/pairs/{OMA_TARGETS[code]}/{OMA_TEMPLATE}/?page_size=1000" + ) + while url: + request = urllib.request.Request( + url, headers={"Accept": "application/json", "User-Agent": "raven-toolbox/1.0"} + ) + with urllib.request.urlopen(request, timeout=300) as fh: + page, headers = json.load(fh), fh.headers + raw += [ + (e["entry_1"]["canonicalid"], e["entry_2"]["canonicalid"]) for e in page + ] + match = re.search(r'<([^>]+)>;\s*rel="next"', headers.get("Link", "")) + url = match.group(1) if match else None + time.sleep(0.5) + cache.write_text(json.dumps(raw), encoding="utf-8") + + truth, unmapped = set(), 0 + for target_id, template_id in raw: + tg, sg = target_map.get(target_id), source_map.get(template_id) + if tg and sg: + truth.add((sg, tg)) + else: + unmapped += 1 + return truth, len(raw), unmapped + + +def _load_gene_ko(path: pathlib.Path, wanted: set[str]) -> dict: + opener = gzip.open if path.suffix == ".gz" else open + per: dict[str, dict[str, set]] = collections.defaultdict(lambda: collections.defaultdict(set)) + with opener(path, "rt", encoding="utf-8") as fh: + next(fh) + for line in fh: + org, gene, ko = line.rstrip("\n").split("\t") + if org in wanted: + per[org][gene].add(ko) + return per + + +def _uniprot_keys(code: str) -> dict[str, str]: + """Every UniProt identifier for this organism -> its KEGG gene name. + + Both the accession and the entry name are included, because OMA uses one or + the other depending on the genome. + """ + url = ( + f"https://rest.uniprot.org/uniprotkb/stream?query=proteome:{PROTEOMES[code]}" + f"&format=tsv&fields=accession,id,xref_kegg" + ) + mapping = {} + for line in _stream(url).splitlines()[1:]: + parts = line.split("\t") + if len(parts) < 3: + continue + for entry in filter(None, (p.strip() for p in parts[2].split(";"))): + prefix, _, gene = entry.partition(":") + if prefix == code: + mapping[parts[0]] = gene + mapping[parts[1]] = gene + break + return mapping + + +def cmd_score(args: argparse.Namespace) -> int: + """Score every threshold combination against the chosen reference.""" + import pandas as pd + + gene_ko = _load_gene_ko(args.gene_ko, set(PROTEOMES)) + source_ko = gene_ko[TEMPLATE] + source_keys = _uniprot_keys(TEMPLATE) if args.reference == "oma" else {} + prefix = "hits" if args.aligner == "blast" else f"hits_{args.aligner}" + + results = {} + for code in PROTEOMES: + if code == TEMPLATE: + continue + path = args.out / f"{prefix}_{code}.csv" + if not path.is_file(): + print(f"{code}: no hit table, run 'align' first") + continue + + hits = pd.read_csv(path) + target_ko = gene_ko[code] + + by_ko: dict[str, list[str]] = {} + for gene, kos in source_ko.items(): + for ko in kos: + by_ko.setdefault(ko, []).append(gene) + truth = { + (source_gene, gene) + for gene, kos in target_ko.items() + for ko in kos + for source_gene in by_ko.get(ko, ()) + } + judged_source, judged_target = set(source_ko), set(target_ko) + + if args.reference == "oma": + target_keys = _uniprot_keys(code) + truth, n_pairs, unmapped = _oma_truth( + code, target_keys, source_keys, args.out + ) + # OMA covers whole proteomes, so every gene with a KEGG name counts. + judged_source = set(source_keys.values()) + judged_target = set(target_keys.values()) + print(f"{code}: {n_pairs} OMA pairs, {len(truth)} usable " + f"({unmapped} unmapped on one side)") + + combos = [dict(BASE)] + for name, values in GRID.items(): + for value in values: + combo = {**BASE, name: value} + if combo not in combos: + combos.append(combo) + + rows = [] + for combo in combos: + kept = hits[ + (hits.evalue <= combo["max_evalue"]) + & (hits.align_len >= combo["min_align_len"]) + & (hits.identity >= combo["min_identity"]) + ] + called, unjudgeable = set(), 0 + for a, b in zip(kept.from_gene, kept.to_gene, strict=True): + if a in judged_source and b in judged_target: + called.add((a, b)) + elif b in judged_source and a in judged_target: + called.add((b, a)) + else: + # The reference says nothing about one of these genes: + # unjudgeable, not wrong. + unjudgeable += 1 + tp, fp = len(called & truth), len(called - truth) + fn = len(truth - called) + precision = tp / (tp + fp) if tp + fp else 0.0 + recall = tp / (tp + fn) if tp + fn else 0.0 + # beta < 1 weights precision above recall. For reconstruction that is + # the honest weighting: a missed reaction can be recovered by + # gap-filling, while a wrongly transferred one pollutes the model and + # its gene associations silently. + b2 = args.beta ** 2 + score = ( + (1 + b2) * precision * recall / (b2 * precision + recall) + if precision and recall else 0.0 + ) + rows.append({**combo, "judged": len(called), "unjudgeable": unjudgeable, + "tp": tp, "fp": fp, "fn": fn, "precision": round(precision, 4), + "recall": round(recall, 4), "beta": args.beta, + "fbeta": round(score, 4)}) + + results[code] = {"truth_pairs": len(truth), "n_hits": len(hits), "rows": rows} + base = next(r for r in rows if all(r[k] == v for k, v in BASE.items())) + best = max(rows, key=lambda r: r["fbeta"]) + label = f"F{args.beta:g}" + print(f"{code}: default {label}={base['fbeta']:.3f} (P={base['precision']:.3f} " + f"R={base['recall']:.3f}) best {label}={best['fbeta']:.3f} at ide=" + f"{best['min_identity']} len={best['min_align_len']}") + + (args.out / "hitlevel_results.json").write_text( + json.dumps(results, indent=2), encoding="utf-8" + ) + print(f"wrote {args.out / 'hitlevel_results.json'}") + return 0 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out", type=pathlib.Path, default=pathlib.Path("work"), + help="working directory for proteomes, hit tables and results") + sub = parser.add_subparsers(dest="command", required=True) + + sub.add_parser("fetch", help="download proteomes, relabelled with KEGG gene ids") + + align = sub.add_parser("align", help="one bidirectional alignment per pair") + align.add_argument("--aligner", choices=("blast", "diamond"), default="blast", + help="DIAMOND is 10-20x faster and reaches the same optimum") + align.add_argument("--threads", type=int, default=4) + align.add_argument("--collection-evalue", type=float, default=1e-4, + help="hit-collection cutoff (RAVEN's getBlast hardcodes 1e-4)") + + score = sub.add_parser("score", help="score thresholds against KO sharing") + score.add_argument("--gene-ko", type=pathlib.Path, required=True, + help="KEGG organism_gene_ko table (.tsv or .tsv.gz)") + score.add_argument("--aligner", choices=("blast", "diamond"), default="blast") + score.add_argument("--reference", choices=("kegg", "oma"), default="kegg", + help="which source decides whether two genes are counterparts") + score.add_argument("--beta", type=float, default=0.5, + help="F-beta weighting; below 1 favours precision, which is the " + "agreed weighting for reconstruction (default: 0.5)") + + args = parser.parse_args(argv) + args.out.mkdir(parents=True, exist_ok=True) + return {"fetch": cmd_fetch, "align": cmd_align, "score": cmd_score}[args.command](args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/measure_confidence_facets.py b/scripts/measure_confidence_facets.py index 8bdc323e..442352d0 100644 --- a/scripts/measure_confidence_facets.py +++ b/scripts/measure_confidence_facets.py @@ -40,6 +40,16 @@ def _recorded(rxn) -> int | None: return None +def _load_model(path: Path): + """Load a model by extension: SBML (.xml/.sbml), cobra YAML (.yml/.yaml) or cobra JSON (.json).""" + suffix = path.suffix.lower() + if suffix in (".yml", ".yaml"): + return cobra.io.load_yaml_model(str(path)) + if suffix == ".json": + return cobra.io.load_json_model(str(path)) + return cobra.io.read_sbml_model(str(path)) + + def main(argv=None): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) @@ -48,7 +58,7 @@ def main(argv=None): ap.add_argument("--head", type=int, default=12, help="rows of the review queue to print") args = ap.parse_args(argv) - model = cobra.io.read_sbml_model(str(args.model)) + model = _load_model(args.model) total = len(model.reactions) print(f"=== {model.id or args.model.name}: {total} reactions, {len(model.metabolites)} metabolites ===") diff --git a/scripts/parity/check_license.py b/scripts/parity/check_license.py new file mode 100644 index 00000000..a4020d07 --- /dev/null +++ b/scripts/parity/check_license.py @@ -0,0 +1,42 @@ +"""Fail unless the Gurobi licence in effect will accept a genome-scale model. + +The licence bundled with pip-installed gurobipy stops at 2000 variables and +2000 constraints -- ample for the small fixtures, useless for a real GEM. A +nightly job that silently fell back to it would skip the genome-scale checks +and still report success, which is exactly the kind of empty green this harness +exists to prevent. So the licence is proven before the tests run. +""" +from __future__ import annotations + +import sys + +SIZE = 20_000 # comfortably past the limited licence, comfortably under a GEM + + +def main() -> int: + try: + import gurobipy as gp + except ImportError as exc: + print(f"gurobipy is not installed: {exc}", file=sys.stderr) + return 1 + + try: + with gp.Env() as env, gp.Model(env=env) as model: + model.setParam("OutputFlag", 0) + variables = model.addVars(SIZE, vtype=gp.GRB.BINARY) + model.addConstr(variables.sum() >= 1) + model.optimize() + print(f"licence OK: solved a {model.NumVars}-variable MIP") + except Exception as exc: # noqa: BLE001 - the message is the whole point + print( + f"Gurobi refused a {SIZE}-variable problem: {exc}\n" + f"The size-limited licence bundled with gurobipy cannot run the " + f"genome-scale checks; a WLS licence must be in effect.", + file=sys.stderr, + ) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/parity/generate_oracles.m b/scripts/parity/generate_oracles.m new file mode 100644 index 00000000..ac13c76b --- /dev/null +++ b/scripts/parity/generate_oracles.m @@ -0,0 +1,129 @@ +function generate_oracles() +% generate_oracles Record MATLAB RAVEN's answers for the parity checks. +% +% Runs a set of RAVEN functions on the fixtures in tests/data/parity/ and +% writes what they return as JSON into tests/data/parity/oracles/. The Python +% side (tests/parity/test_oracles.py) then asserts that raven-toolbox returns +% the same, without needing MATLAB at test time. +% +% Only tier-1 behaviour belongs here: deterministic answers that both +% implementations should agree on exactly. Anything solved by MILP has +% alternate optima and is compared as an overlap band instead -- see +% tests/parity/README.md. +% +% Usage, with RAVEN on the MATLAB path: +% +% cd scripts/parity +% generate_oracles +% +% Commit the resulting JSON together with the RAVEN commit it came from, which +% is recorded in oracles/PROVENANCE.md by this function. + +here = fileparts(mfilename('fullpath')); +repoRoot = fileparts(fileparts(here)); +fixtureDir = fullfile(repoRoot, 'tests', 'data', 'parity'); +oracleDir = fullfile(fixtureDir, 'oracles'); +if ~exist(oracleDir, 'dir') + mkdir(oracleDir); +end + +model = readYAMLmodel(fullfile(fixtureDir, 'tiny.yml')); + +writeOracle(oracleDir, 'model_structure', modelStructure(model)); +writeOracle(oracleDir, 'elemental_balance', elementalBalance(model)); +writeOracle(oracleDir, 'gpr_dnf', gprDnf(model)); + +writeProvenance(oracleDir); +fprintf('Oracles written to %s\n', oracleDir); +end + + +function out = modelStructure(model) +% What the reader saw: ids, bounds, stoichiometry, gene rules. +out = struct(); +out.rxns = reshape(model.rxns, 1, []); +out.mets = reshape(model.mets, 1, []); +out.genes = reshape(model.genes, 1, []); +out.lb = reshape(model.lb, 1, []); +out.ub = reshape(model.ub, 1, []); +out.grRules = reshape(model.grRules, 1, []); + +% Stoichiometry as a list of {rxn, met, coefficient}, so the comparison does +% not depend on either side's matrix layout. +[metIdx, rxnIdx, coeff] = find(model.S); +entries = cell(1, numel(coeff)); +for k = 1:numel(coeff) + entries{k} = struct( ... + 'rxn', model.rxns{rxnIdx(k)}, ... + 'met', model.mets{metIdx(k)}, ... + 'coefficient', coeff(k)); +end +out.stoichiometry = entries; +end + + +function out = elementalBalance(model) +% getElementalBalance grades each reaction: 1 balanced, 0 unbalanced, +% -1 undecidable (a participating metabolite has no formula). +balance = getElementalBalance(model); +out = struct(); +out.rxns = reshape(model.rxns, 1, []); +out.balanceStatus = reshape(balance.balanceStatus, 1, []); +end + + +function out = gprDnf(model) +% Gene rules normalised to disjunctive normal form. +rules = cell(1, numel(model.grRules)); +for i = 1:numel(model.grRules) + if isempty(model.grRules{i}) + rules{i} = ''; + else + rules{i} = grRuleToDNF(model.grRules{i}); + end +end +out = struct(); +out.rxns = reshape(model.rxns, 1, []); +out.grRules = reshape(model.grRules, 1, []); +out.dnf = rules; +end + + +function writeOracle(oracleDir, name, data) +target = fullfile(oracleDir, [name '.json']); +fid = fopen(target, 'w'); +if fid == -1 + error('generate_oracles:cannotWrite', 'Could not open %s', target); +end +cleanup = onCleanup(@() fclose(fid)); +fprintf(fid, '%s', jsonencode(data, 'PrettyPrint', true)); +end + + +function writeProvenance(oracleDir) +% Record which RAVEN produced these answers, so a stale oracle is visible. +commit = 'unknown'; +try + ravenDir = fileparts(which('readYAMLmodel')); + [status, out] = system(sprintf('git -C "%s" rev-parse HEAD', ravenDir)); + if status == 0 + commit = strtrim(out); + end +catch + % Leave as unknown -- provenance is best-effort, not a hard requirement. +end + +text = sprintf([ ... + '# Oracle provenance\n\n' ... + 'Generated by `scripts/parity/generate_oracles.m`.\n\n' ... + '- RAVEN commit: `%s`\n' ... + '- MATLAB: %s\n' ... + '- Generated: %s\n\n' ... + 'Regenerate after changing a fixture or after a RAVEN change that could\n' ... + 'affect these functions, and commit the result.\n'], ... + commit, version, datestr(now, 'yyyy-mm-dd')); + +fid = fopen(fullfile(oracleDir, 'PROVENANCE.md'), 'w'); +cleanup = onCleanup(@() fclose(fid)); +fprintf(fid, '%s', text); +end diff --git a/scripts/parity/record_baseline.py b/scripts/parity/record_baseline.py new file mode 100644 index 00000000..609e586d --- /dev/null +++ b/scripts/parity/record_baseline.py @@ -0,0 +1,112 @@ +"""Record the set-level baseline the parity suite compares against. + +Set-level checks (tier 2 in ``tests/parity/README.md``) cannot assert a single +right answer: extraction is a MILP and equally-good optima exist. What they can +do is detect *drift* -- today's result differing from the result that was +inspected and accepted. + +This script records that accepted result. Run it when a change is expected to +move the extraction, inspect the diff the test reports, and commit the new +baseline with a note in the PR saying why it moved. + + python scripts/parity/record_baseline.py # uses $RAVEN_ROOT + python scripts/parity/record_baseline.py --raven-root /path/to/RAVEN + +The baseline records which implementation produced it. It is currently seeded +from raven-toolbox itself, which makes the check a regression guard rather than +a cross-language one; when ``scripts/parity/generate_oracles.m`` grows an +extraction oracle, the same comparison runs against MATLAB RAVEN's answer and +the ``source`` field changes to say so. +""" +from __future__ import annotations + +import argparse +import datetime as dt +import json +import os +import platform +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent +BASELINE_DIR = REPO_ROOT / "tests" / "data" / "parity" / "baselines" + +# The fixture is RAVEN's own small yeast model, read from a RAVEN checkout +# rather than vendored: RAVEN is GPL and this package is MIT. +FIXTURE = ("tutorial", "smallYeast.yml") + + +def reaction_scores(model) -> dict[str, float]: + """Deterministic scores that force a real decision. + + A third of the reactions are clearly worth keeping, a third clearly not, + and a third marginal -- so the extraction has to choose rather than keep + everything. + """ + return { + rxn.id: (10.0 if i % 3 == 0 else (-5.0 if i % 3 == 1 else 1.0)) + for i, rxn in enumerate(model.reactions) + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--raven-root", + default=os.environ.get("RAVEN_ROOT"), + help="a MATLAB RAVEN checkout (default: $RAVEN_ROOT)", + ) + args = parser.parse_args(argv) + + if not args.raven_root: + parser.error( + "no RAVEN checkout given. Clone RAVEN and pass --raven-root, or set " + "RAVEN_ROOT: git clone --depth 1 -b develop3 " + "https://github.com/SysBioChalmers/RAVEN" + ) + + import cobra + + import raven_toolbox + from raven_toolbox.init import run_init + from raven_toolbox.io import read_yaml_model + + fixture = Path(args.raven_root).joinpath(*FIXTURE) + if not fixture.is_file(): + parser.error(f"fixture not found: {fixture}") + + model = read_yaml_model(fixture) + scores = reaction_scores(model) + result = run_init(model, scores) + kept = sorted(r.id for r in result.model.reactions) + + solver = cobra.Configuration().solver.__name__.split(".")[-1] + baseline = { + "source": "raven-toolbox", + "description": ( + "Reactions kept by run_init on RAVEN's tutorial/smallYeast.yml with " + "the scores from scripts/parity/record_baseline.py. Seeded from " + "raven-toolbox, so this detects drift in this package rather than " + "disagreement with MATLAB RAVEN." + ), + "fixture": "/".join(FIXTURE), + "model_reactions": len(model.reactions), + "kept_reactions": kept, + "recorded": { + "date": dt.date.today().isoformat(), + "raven_toolbox": getattr(raven_toolbox, "__version__", "unknown"), + "cobra": cobra.__version__, + "solver": solver, + "python": platform.python_version(), + }, + } + + BASELINE_DIR.mkdir(parents=True, exist_ok=True) + target = BASELINE_DIR / "init_smallyeast.json" + target.write_text(json.dumps(baseline, indent=2) + "\n", encoding="utf-8") + print(f"recorded {len(kept)}/{len(model.reactions)} reactions to {target}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/raven_toolbox/annotation/delta_g.py b/src/raven_toolbox/annotation/delta_g.py index e0a90e0e..bba16f50 100644 --- a/src/raven_toolbox/annotation/delta_g.py +++ b/src/raven_toolbox/annotation/delta_g.py @@ -18,6 +18,27 @@ import cobra import pandas as pd +#: The "no valid ΔG" sentinel used by the ΔG side-car tables (e.g. yeast-GEM's +#: ``model_rxnDeltaG.csv``). yeast-GEM's ``checkrxnDirection.m`` gates on it verbatim: +#: ``if ~isequal(seed_rxnInfo{...},'10000000.0') %check if database contains valid deltaG +#: value``. Stamping it would present a physically impossible 10⁷ kJ/mol as a measurement. +#: Pass this as ``load_delta_g_csv``'s ``missing_value`` to have it treated as missing +#: instead of stored verbatim — RAVEN's own ``deltaGCSV`` has no sentinel concept by +#: default either, so this is opt-in on both sides, not automatic on either. +DELTA_G_MISSING = 1e7 + + +def _is_missing(value, sentinel: float) -> bool: + """True when ``value`` is the sentinel, whether it arrived as a number or as text. + + The CSV round-trips through MATLAB and pandas, so the same sentinel shows up as ``10000000``, + ``10000000.0`` or ``"10000000.0"`` depending on the writer and the column's inferred dtype. + """ + try: + return math.isclose(float(value), sentinel, rel_tol=1e-9) + except (TypeError, ValueError): + return False + def load_delta_g_csv( entities: Iterable, @@ -26,6 +47,7 @@ def load_delta_g_csv( id_column: str = "Var1", value_column: str = "Var2", note_key: str = "deltaG", + missing_value: float | None = None, verbose: bool = False, ) -> int: """Stamp ``note_key`` on each entity from a CSV of ``id → value``. @@ -42,12 +64,20 @@ def load_delta_g_csv( note_key Key under which the value is stored on ``entity.notes``. Default ``"deltaG"``. + missing_value + A sentinel standing for "no value", left unstamped rather than + recorded as a measurement. Default ``None``: every matched value + is stamped verbatim, matching RAVEN's own ``deltaGCSV`` (which has + no sentinel concept of its own). Pass :data:`DELTA_G_MISSING` + (10⁷) to opt into yeast-GEM's own convention, which covers 777 of + yeast-GEM's 4102 reaction rows. verbose Print a summary of unmatched entity ids. Returns ------- - The number of entities that were stamped (i.e. matched the CSV). + The number of entities that were stamped (i.e. matched the CSV and + carried a real value). """ df = pd.read_csv(path) if id_column not in df.columns or value_column not in df.columns: @@ -64,6 +94,9 @@ def load_delta_g_csv( if value is None or (isinstance(value, float) and math.isnan(value)): missing.append(entity.id) continue + if missing_value is not None and _is_missing(value, missing_value): + missing.append(entity.id) + continue entity.notes[note_key] = str(value) stamped += 1 @@ -111,5 +144,5 @@ def save_delta_g_csv( # Re-export the cobra Model type for type-checker friendliness; helps # IDEs surface the right hints to callers that hand us model.metabolites # / model.reactions directly. -__all__ = ["load_delta_g_csv", "save_delta_g_csv"] +__all__ = ["DELTA_G_MISSING", "load_delta_g_csv", "save_delta_g_csv"] _ = cobra # silence "imported but unused" — used for typing context above diff --git a/src/raven_toolbox/binaries.py b/src/raven_toolbox/binaries.py index 5eebdab8..7b5ecc28 100644 --- a/src/raven_toolbox/binaries.py +++ b/src/raven_toolbox/binaries.py @@ -183,6 +183,18 @@ def _safe_extract_zip(zf: zipfile.ZipFile, dest_dir: Path) -> None: zf.extractall(dest_dir) +def _make_executable(path: Path) -> None: + """Add the execute bit for everyone who can already read the file. + + ZIP archives carry Unix permissions in a field ``zipfile`` does not restore, + so every extracted file arrives non-executable on Linux and macOS. Derived + from the read bits rather than set to a fixed 0o755 so a restrictive umask + is not silently widened. + """ + mode = path.stat().st_mode + path.chmod(mode | ((mode & 0o444) >> 2)) + + def ensure_binary(executable: str, *, registry: dict | None = None) -> Path: """Download (if needed) and return the path to a bundled ``executable``. @@ -217,6 +229,10 @@ def _find_exe() -> Path | None: cached = _find_exe() if cached is not None: + # Re-applied on the cached path too, not only after extraction: a bundle + # left behind by an older version of this function has only the one + # executable it was asked for marked, and the rest still unusable. + _make_executable(cached) return cached dest_dir.mkdir(parents=True, exist_ok=True) @@ -245,7 +261,15 @@ def _find_exe() -> Path | None: raise FileNotFoundError( f"None of {candidates} found in the extracted bundle at {dest_dir}." ) - exe.chmod(0o755) + # Every executable the bundle provides, not just the one asked for. The BLAST + # bundle ships blastp *and* makeblastdb; marking only the requested one left + # the other non-executable, and the failure surfaced much later as a + # PermissionError from whichever tool happened to be called second. + for provided in bundle.get("provides", []): + for name in ([f"{provided}.exe", provided] if key.startswith("windows-") else [provided]): + sibling = dest_dir / name + if sibling.is_file(): + _make_executable(sibling) return exe diff --git a/src/raven_toolbox/comparison/diff.py b/src/raven_toolbox/comparison/diff.py index 44dee758..e4326883 100644 --- a/src/raven_toolbox/comparison/diff.py +++ b/src/raven_toolbox/comparison/diff.py @@ -10,8 +10,8 @@ Diff scope: reaction / metabolite / gene id sets, stoichiometry (within tolerance), bounds, objective coefficients, GPR rules, metabolite formula/charge/compartment, and a configurable set of annotation keys. -Formatting differences (key ordering, whitespace, float repr) are -explicitly **not** failures. +Formatting differences (key ordering, whitespace, float repr, and GPR +operand order — ``a and b`` == ``b and a``) are explicitly **not** failures. """ from __future__ import annotations @@ -20,6 +20,8 @@ import cobra +from raven_toolbox.manipulation.expand import gpr_to_dnf + # Annotation keys checked by default. Add via ``extra_annotations`` or # remove via ``ignore_annotations`` in :func:`diff_models`. DEFAULT_ANNOTATION_KEYS: tuple[str, ...] = ( @@ -152,10 +154,9 @@ def _diff_reactions( f"{rxn_id}: objective A={ra.objective_coefficient} B={rb.objective_coefficient}", ) - ga = _normalise_gpr(ra.gene_reaction_rule) - gb = _normalise_gpr(rb.gene_reaction_rule) - if ga != gb: - _push(diffs, counters, "gpr", cap, f"{rxn_id}: GPR A={ga!r} B={gb!r}") + if _canonical_gpr(ra) != _canonical_gpr(rb): + _push(diffs, counters, "gpr", cap, + f"{rxn_id}: GPR A={ra.gene_reaction_rule!r} B={rb.gene_reaction_rule!r}") _diff_annotations( f"rxn {rxn_id}", ra.annotation, rb.annotation, @@ -218,16 +219,30 @@ def _diff_annotations( _push(diffs, counters, "anno", cap, f"{label}.annotation[{k!r}]: A={va} B={vb}") -def _normalise_gpr(rule: str) -> str: - """Lowercase + collapse whitespace. +def _canonical_gpr(reaction: cobra.Reaction) -> str: + """Canonical, order-insensitive form of a reaction's GPR, for logic-level comparison. + + DNF-expand the parsed GPR, then sort the genes within each isozyme clause and sort the clauses, so + operand order never registers as a difference: ``a and b`` == ``b and a`` and ``a or b`` == + ``b or a``. Mirrors MATLAB RAVEN's ``diffModels`` (`RAVEN #686 + `_), which compares grRules as logic rather than + text. Gene ids are lowercased and duplicate genes/clauses collapse, preserving the formatting-drift + tolerance the previous string heuristic gave. - A more robust comparator would parse to a GPR AST and compare - structures; this is the cheap heuristic that catches the formatting - drift we see between different SBML writers. + Falls back to that whitespace/lowercase heuristic for a rule cobra could not parse (``gpr.body`` is + ``None`` while the rule string is non-empty), so a malformed rule is still compared rather than + silently equated to the empty rule. """ - if not rule: - return "" - return " ".join(rule.lower().split()) + gpr = reaction.gpr + if gpr is not None and gpr.body is not None: + try: + clauses = gpr_to_dnf(gpr) + except ValueError: + clauses = None + if clauses is not None: + canon = sorted({tuple(sorted({g.lower() for g in clause})) for clause in clauses}) + return " | ".join(" & ".join(clause) for clause in canon) + return " ".join((reaction.gene_reaction_rule or "").lower().split()) def _normalise_annotation_value(v): diff --git a/src/raven_toolbox/conditions/apply.py b/src/raven_toolbox/conditions/apply.py index 85c72a86..a19eb6ff 100644 --- a/src/raven_toolbox/conditions/apply.py +++ b/src/raven_toolbox/conditions/apply.py @@ -7,9 +7,13 @@ name: my_condition description: free-form text. - # Set every exchange reaction to (0, ub) before any per-rxn override. + # Set exchange reactions to (0, ub) before any per-rxn override. + # "in" / "out" (RAVEN's getExchangeRxns direction: the boundary metabolite + # is the reaction's substrate / product respectively) reset only that + # direction; "all" / "both", or any other truthy value, reset every + # exchange in either direction. prelude: - reset_exchanges: out # str | bool truthy + reset_exchanges: out # "in" | "out" | "all" | "both" | bool truthy # Remove metabolites from a pseudoreaction and rebalance H+ # (or any other "balance" met) so total charge stays zero. @@ -107,14 +111,44 @@ def set_reaction_bounds(rxn: cobra.Reaction, lb: float, ub: float) -> None: # --- step implementations --------------------------------------------- +def _exchange_direction(rxn: cobra.Reaction) -> str | None: + """RAVEN ``getExchangeRxns``' in/out rule, direction-of-boundary-metabolite based. + + "out": nothing is produced within the model at all (``sum(S > 0) == 0``) --- the + boundary metabolite is implicitly the reaction's product. "in": nothing is + consumed (``sum(S < 0) == 0``) --- the boundary metabolite is implicitly the + substrate. Neither (an ordinary internal reaction with both) returns ``None``. + Not restricted to single-metabolite reactions, matching RAVEN's own rule exactly + rather than cobra's narrower ``Reaction.boundary``. + """ + has_product = any(c > 0 for c in rxn.metabolites.values()) + has_reactant = any(c < 0 for c in rxn.metabolites.values()) + if not has_product: + return "out" + if not has_reactant: + return "in" + return None + + def _apply_prelude(model: cobra.Model, cfg: dict[str, Any]) -> None: prelude = cfg.get("prelude") or {} - if prelude.get("reset_exchanges"): - # cobrapy doesn't distinguish RAVEN's in / out directions; we - # reset every exchange reaction. - for rxn in model.exchanges: - rxn.lower_bound = 0 - rxn.upper_bound = DEFAULT_RESET_EXCHANGES_UPPER_BOUND + value = prelude.get("reset_exchanges") + if not value: + return + direction = value.lower() if isinstance(value, str) else None + if direction in ("in", "out"): + # A named direction resets only that direction, matching RAVEN's + # applyCondition, which forwards this same value straight through to + # getExchangeRxns as its direction filter. + targets = [r for r in model.reactions if _exchange_direction(r) == direction] + else: + # "all" / "both", or any other truthy value that isn't a direction + # keyword (e.g. a bare `true`): reset every reaction RAVEN's + # getExchangeRxns would call an exchange in either direction. + targets = [r for r in model.reactions if _exchange_direction(r) is not None] + for rxn in targets: + rxn.lower_bound = 0 + rxn.upper_bound = DEFAULT_RESET_EXCHANGES_UPPER_BOUND def _apply_cofactor_pseudoreaction(model: cobra.Model, cfg: dict[str, Any]) -> None: diff --git a/src/raven_toolbox/confidence.py b/src/raven_toolbox/confidence.py index f544c0b0..ca6fc40a 100644 --- a/src/raven_toolbox/confidence.py +++ b/src/raven_toolbox/confidence.py @@ -1,10 +1,10 @@ """Per-reaction, multi-facet confidence — persisted in the model, ignored by plain cobra. -Attaches a small structured record to a reaction scoring how well-supported each *facet* of it is -(``localization``, ``equation``, ``gene_association``; ``reversibility`` follows the same shape). Each -facet is a :class:`ConfidenceEntry` — a continuous 0-1 ``score`` plus optional provenance (a categorical -``level``, the ``basis`` evidence, ``method``/``source``/``note``). A reaction carries a -:class:`ReactionConfidence` (facet → entry) whose ``overall`` is the weakest facet. +Attaches a small structured record to a reaction scoring how well-supported each *facet* of it is: +``localization``, ``equation`` and ``gene_association``. Each facet is a :class:`ConfidenceEntry` — a +continuous 0-1 ``score`` plus optional provenance (a categorical ``level``, the ``basis`` evidence, +``method``/``source``/``note``). A reaction carries a :class:`ReactionConfidence` (facet → entry) whose +``overall`` is the weakest facet. **Two rules govern every score**, because ``overall = min(facets)`` and :func:`_write` drops the record when no facet remains: @@ -30,9 +30,9 @@ defect. Detecting biomass by name instead is deliberately *not* done: ``\\bgrowth\\b`` matches "non-growth associated maintenance reaction", and a name regex must never silence a chemistry check. -The design and roadmap (the ``reversibility`` facet, ECO/SBO and Thiele-Palsson mapping) are in -``docs/studies/confidence_tracking.md``. Wire it in by calling :func:`score_localization_confidence` on -an :class:`~raven_toolbox.localization.AssignmentProposal`, :func:`score_equation_confidence` and +The design and the measured yeast-GEM distributions are in ``docs/studies/confidence_tracking.md``; the +facet set above is closed. Wire it in by calling :func:`score_localization_confidence` on an +:class:`~raven_toolbox.localization.AssignmentProposal`, :func:`score_equation_confidence` and :func:`score_gene_association_confidence` on any model, and :func:`mark_curated` when a curator firmly fixes a facet (e.g. after :func:`~raven_toolbox.localization.relocate_reactions`). """ @@ -43,6 +43,7 @@ import json import math import warnings +from collections.abc import Iterable from dataclasses import dataclass, field from typing import Any @@ -54,6 +55,7 @@ __all__ = [ "ConfidenceEntry", "ReactionConfidence", + "annotate_confidence", "clear_confidence", "confidence_report", "equation_exempt", @@ -66,6 +68,7 @@ "score_gene_association_confidence", "score_localization_confidence", "set_confidence", + "thiele_palsson_score", ] _KEY = "raven_confidence" @@ -390,7 +393,8 @@ def _warn_if_no_sbo(model: cobra.Model) -> None: if model.reactions and not any(_sbo(r) for r in model.reactions): warnings.warn( "no reaction carries an SBO term, so biomass and pool pseudo-reactions cannot be told from " - "chemistry defects and will be scored as defects. Annotate SBO terms first.", + "chemistry defects and will be scored as defects. Annotate SBO terms first with " + "raven_toolbox.annotation.add_sbo_terms(model).", stacklevel=3, ) @@ -594,6 +598,67 @@ def score_gene_association_confidence(model, *, overwrite_curated: bool = False, return n +#: ``gene_association`` facet ``basis`` -> Thiele & Palsson reconstruction confidence score (0-4). A +#: homology-derived GPR is sequence evidence (2); a GPR with a literature citation is +#: experimental/genetic (3); a reaction that should have a catalyst but has none is a modelling +#: inference (1). ``curated`` is deliberately absent: a curator's assertion does not, on its own, name +#: the *evidence class* it rests on (the score's ``basis`` is ``"curator"``), so its Thiele-Palsson +#: class must be set from the evidence the curator used, not inferred here (see the study doc §9). +_THIELE_PALSSON_FROM_GA_BASIS = {"gpr+literature": 3, "gpr": 2, "no-gpr": 1} + + +def thiele_palsson_score(reaction: cobra.Reaction) -> int | None: + """The reaction's Thiele & Palsson reconstruction confidence score (0-4), or ``None``. + + Derived from the ``gene_association`` facet's ``basis`` (:func:`score_gene_association_confidence`), + the facet that captures reaction-inclusion evidence: ``gpr+literature`` -> 3 (experimental/genetic), + ``gpr`` -> 2 (sequence), ``no-gpr`` -> 1 (modelling). Returns ``None`` when the facet is absent (the + reaction was not scored, or ``gene_association`` does not apply to it) or ``curated`` (whose evidence + class the curator must name — see the module note and the study doc §9). The ``localization`` and + ``equation`` facets are quality checks, not Thiele-Palsson evidence classes, and are not consulted. + """ + entry = get_confidence(reaction).facets.get("gene_association") + if entry is None: + return None + return _THIELE_PALSSON_FROM_GA_BASIS.get(entry.basis) + + +def annotate_confidence( + model: cobra.Model, + *, + proposal: Any = None, + scores: Any = None, + facets: Iterable[str] | None = None, + overwrite_curated: bool = False, + updated: str | None = None, +) -> dict[str, int]: + """Run every applicable confidence scorer in one call; return ``{facet: reactions_scored}``. + + ``equation`` and ``gene_association`` need only the model and always run; ``localization`` runs + only when both ``proposal`` (an :class:`~raven_toolbox.localization.AssignmentProposal`) and + ``scores`` (a :class:`~raven_toolbox.localization.LocalizationScores`) are given, and is otherwise + **skipped rather than failing** — the same abstain-rather-than-guess rule the scores follow, so a + caller without a localisation proposal still gets the other two facets. ``facets=[...]`` restricts + to a subset (names outside the facet set are ignored). ``overwrite_curated`` and ``updated`` pass + through to each scorer. + """ + requested = {"localization", "equation", "gene_association"} if facets is None else set(facets) + counts: dict[str, int] = {} + if "localization" in requested and proposal is not None and scores is not None: + counts["localization"] = score_localization_confidence( + model, proposal, scores, overwrite_curated=overwrite_curated, updated=updated + ) + if "equation" in requested: + counts["equation"] = score_equation_confidence( + model, overwrite_curated=overwrite_curated, updated=updated + ) + if "gene_association" in requested: + counts["gene_association"] = score_gene_association_confidence( + model, overwrite_curated=overwrite_curated, updated=updated + ) + return counts + + def _score(df, g: str, c: str) -> float: if g not in df.index or c not in df.columns: return 0.0 diff --git a/src/raven_toolbox/init/ftinit.py b/src/raven_toolbox/init/ftinit.py index 5d45b5e0..09c43587 100644 --- a/src/raven_toolbox/init/ftinit.py +++ b/src/raven_toolbox/init/ftinit.py @@ -62,6 +62,9 @@ _FORCE_ON = 0.1 # min flux for a reaction to count as "on" (RAVEN forceOnLim) _BIG_M = 100.0 # indicator/direction big-M cap on a *scored* reaction's flux (RAVEN's 100) +_STRICT_ABS_GAP = 0.05 # absolute gap for the opt-in strict mode (below the 0.1 score granularity) +_EXTRACT_SEED = 1234 # RAVEN optimizeProb Seed; a module constant so a determinism probe + # can vary it (see docs/studies/ftinit_determinism.md) without editing code def _dbg(msg: str) -> None: @@ -101,6 +104,8 @@ def run_ftinit( mip_gap: float | None = None, mip_gap_abs: float | None = None, time_limit: float | None = None, + strict_abs_gap: float | None = None, + canonical: bool = False, ) -> FtInitResult: """Run the single-step ftINIT MILP and return the extracted model. @@ -112,6 +117,12 @@ def run_ftinit( per-step "simple metabolite" removal, e.g. H2O/H+). See the module docstring for the formulation. This is the single-step variant; the staged schedule (:func:`raven_toolbox.init.ftinit`) calls it per step. + + ``canonical`` (opt-in, default off to preserve RAVEN parity) resolves the MILP's + degeneracy deterministically: after the score optimum is found, a lexicographic + phase 2 holds the objective and minimises the id-ordered count of "on" reactions, so + the kept set is the unique sparsest optimum, independent of solver seed/version, + instead of an arbitrary tie-break. See :func:`_canonicalize`. """ scores = dict(rxn_scores or {}) essential = set(essential_rxns or []) @@ -217,9 +228,8 @@ def add_constraint(expr, **kw): add_constraint(add(termlist), lb=0.0, ub=None if allow_excretion else 0.0) opt.add(variables + constraints) - opt.objective = prob.Objective( - add([mul([Real(score), ind]) for ind, score in indicators.values()]), direction="max" - ) + obj_expr = add([mul([Real(score), ind]) for ind, score in indicators.values()]) + opt.objective = prob.Objective(obj_expr, direction="max") try: # Gurobi-specific; harmless if the backend differs. Match RAVEN's optimizeProb # defaults exactly, because the ftINIT MILP is highly degenerate and the chosen # incumbent (hence which reactions are kept) depends on these: @@ -229,22 +239,32 @@ def add_constraint(expr, **kw): # * Presolve=2, FeasibilityTol/OptimalityTol/IntFeasTol=1e-9 — RAVEN's # optimizeProb defaults; they steer which optimal vertex a degenerate MILP # lands on and how binaries round at the 0.5 on/off cut. - # * Seed=1234 — fixed seed so tie-breaking is reproducible. + # * Seed (``_EXTRACT_SEED``=1234) — fixed seed so tie-breaking is reproducible. opt.problem.Params.Threads = 1 opt.problem.Params.Presolve = 2 opt.problem.Params.FeasibilityTol = 1e-9 opt.problem.Params.OptimalityTol = 1e-9 opt.problem.Params.IntFeasTol = 1e-9 - opt.problem.Params.Seed = 1234 + opt.problem.Params.Seed = _EXTRACT_SEED except Exception: # noqa: BLE001 pass - if mip_gap is not None: + if strict_abs_gap is not None: + # Strict mode: prove the optimum to a fixed *absolute* gap below the reaction-score + # granularity (scores are nudged to |score| ≥ 0.1), so the kept-set objective is + # proven optimal at any objective scale. A relative gap is meaningless on the + # near-zero-objective final step; this also bypasses the loose escalation below. + try: # Gurobi-specific; harmless if the backend differs + opt.problem.Params.MIPGap = 0.0 + opt.problem.Params.MIPGapAbs = strict_abs_gap + except Exception: # noqa: BLE001 + pass + elif mip_gap is not None: try: # Gurobi-specific; harmless if the backend differs opt.problem.Params.MIPGap = mip_gap except Exception: # noqa: BLE001 pass - if mip_gap_abs is not None: + if strict_abs_gap is None and mip_gap_abs is not None: # RAVEN's multi-run gap strategy (ftINIT.m). The final staged step has a # near-zero objective (it mostly removes small negative-score reactions), so a # fixed *relative* gap becomes an almost-zero absolute gap the solver cannot @@ -276,27 +296,115 @@ def add_constraint(expr, **kw): _achieved = None _dbg(f"[ftinit] final solve: obj={opt.objective.value} status={opt.status} " f"achieved_gap={_achieved}") - # Accept a near-optimal incumbent (when a MIP gap / time limit is set), as RAVEN does. - if opt.status not in ("optimal", "feasible", "suboptimal", "time_limit"): - raise OptimizationError(f"ftINIT MILP did not solve (status: {opt.status}).") + # Accept a near-optimal incumbent (when a MIP gap / time limit is set), as RAVEN does, + # but only if the solver actually holds one to read. + if opt.status not in ("optimal", "feasible", "suboptimal", "time_limit") \ + or not _has_solution(opt): + raise OptimizationError( + f"ftINIT MILP produced no usable solution (status: {opt.status}); " + "increase time_limit or disable strict_gap." + ) + + # Report the *primary* (score) objective; a canonical phase 2 replaces the objective + # in place, so capture it before that. + primary_obj = float(opt.objective.value) if opt.objective.value is not None else 0.0 # RAVEN: a reaction is "on" iff its indicator ≥ 0.5 (positive indicators are # continuous and can land fractionally when a reaction can carry only tiny flux). - on = {rid for rid, (ind, _) in indicators.items() if (ind.primal or 0.0) >= 0.5} + def _read_solution(): + on = {rid for rid, (ind, _) in indicators.items() if (ind.primal or 0.0) >= 0.5} + fluxes = {rid: sum(sign * (var.primal or 0.0) for var, sign in terms) + for rid, terms in flux_terms.items()} + return on, fluxes + + on, fluxes = _read_solution() # the primary optimum + # canonical is best-effort: keep its result only if phase 2 actually converged. + if canonical and indicators and _canonicalize(opt, prob, obj_expr, indicators, + primary_obj, time_limit): + on, fluxes = _read_solution() + kept = free_or_essential | on deleted = [r.id for r in model.reactions if r.id not in kept] - fluxes: dict[str, float] = { - rid: sum(sign * (var.primal or 0.0) for var, sign in terms) - for rid, terms in flux_terms.items() - } out = model.copy() out.remove_reactions(deleted, remove_orphans=True) return FtInitResult(out, sorted(kept), sorted(deleted), fluxes, - float(opt.objective.value), on_reactions=on, + primary_obj, on_reactions=on, achieved_gap=_achieved) +def _has_solution(opt) -> bool: + """Whether the solver currently holds a readable primal solution. + + A MILP can finish with an accepted status (notably ``time_limit``) yet no incumbent, + so reading ``.primal`` would raise. On Gurobi this is the solution count; on other + backends the status is taken as authoritative. + """ + try: + return opt.problem.SolCount > 0 + except Exception: # noqa: BLE001 - non-Gurobi backend + return True + + +def _canonicalize(opt, prob, obj_expr, indicators, primary, time_limit) -> bool: + """Pin ftINIT's degenerate optimum to a single canonical solution (in place on ``opt``). + + The ftINIT MILP is highly degenerate: many reaction subsets reach the same score + optimum and the solver returns an arbitrary one — reproducible for a fixed solver + build + seed, but fragile to the Gurobi version or platform (a changed tie-break + moves ~1-2% of the kept reactions, which flips a handful of gene-essentiality calls). + + This runs a lexicographic phase 2, holding the score objective at its optimum with a + floor constraint: first minimise the count of kept removable reactions (the sparsest + optimum), then, among the sparsest, minimise their summed id rank (prefer lower ids). + The result is a stable, near-unique optimum independent of seed/solver version. ``opt`` + is left holding the phase-2 solution, which the caller reads for the "on" set and + fluxes; the reported objective stays the phase-1 ``primary`` value. + + Only the negative-score reactions carry a true 0/1 "keep" binary (the positive + indicators are continuous and pinned near 1 by the score objective), so the two phases + run over those: their count and id-sum are integers, provable with a cheap absolute gap + below 1 rather than the near-full proof a tiny relative gap would need. This + canonicalises the removable-reaction choices — the genuine seed-fragile degeneracy; a + residual flux-distribution degeneracy (which only feeds the next step's small + ``ess_force`` clamp) is left to the solver. + + Returns ``True`` if phase 2 produced a usable solution (the caller then reads the + canonical "on" set and fluxes), ``False`` if it did not converge to an incumbent (the + caller keeps the phase-1 optimum). Canonicalisation is best-effort: it never fails the + extraction. + """ + binaries = {rid: ind for rid, (ind, score) in indicators.items() if score < 0} + if not binaries: # only positive / free reactions: nothing removable to canonicalise + return _has_solution(opt) + tol = max(abs(primary) * 1e-7, 1e-7) + opt.add(prob.Constraint(obj_expr, lb=primary - tol, name="_canon_obj_floor")) + count = add([mul([Real(1.0), ind]) for ind in binaries.values()]) + + def _phase(objective) -> bool: + opt.objective = objective + try: # integer objective: an absolute gap < 1 proves the optimum cheaply. + opt.problem.Params.MIPGap = 0.0 + opt.problem.Params.MIPGapAbs = 0.4 + except Exception: # noqa: BLE001 - GLPK solves exactly; harmless + pass + if time_limit is not None: + opt.configuration.timeout = int(time_limit) + opt.optimize() + return (opt.status in ("optimal", "feasible", "suboptimal", "time_limit") + and _has_solution(opt)) + + # Phase 2a — parsimony: the fewest kept removable reactions. + if not _phase(prob.Objective(count, direction="min")): + return False + kmin = opt.objective.value or 0.0 + # Phase 2b — among the sparsest, prefer lower reaction ids (deterministic tie-break). + opt.add(prob.Constraint(count, ub=kmin + 0.5, name="_canon_count_cap")) + ranks = {rid: i for i, rid in enumerate(sorted(binaries))} + idsum = add([mul([Real(float(1 + ranks[rid])), ind]) for rid, ind in binaries.items()]) + return _phase(prob.Objective(idsum, direction="min")) + + def _nudge_scores(rxn_scores: Mapping[str, float]) -> dict[str, float]: """Push tiny reaction scores off zero (RAVEN ``ftINIT.m:160-161``). @@ -317,7 +425,7 @@ def _nudge_scores(rxn_scores: Mapping[str, float]) -> dict[str, float]: def _solve_step( min_model, scores, step, *, essential, directions, ess_force, force_on, big_m, - mip_gap, mip_gap_abs, time_limit, + mip_gap, mip_gap_abs, time_limit, strict_gap=False, canonical=False, ) -> FtInitResult: """Solve one ftINIT step, following RAVEN's multi-run gap-escalation schedule. @@ -328,15 +436,23 @@ def _solve_step( caller ``time_limit`` caps each run's own limit. With no schedule (``step.milp_runs`` empty, e.g. the ``'full'`` series) this is a single solve at the caller's gap. """ - def _run(mg, mga, tl): + def _run(mg, mga, tl, strict=None): return run_ftinit( min_model, scores, essential_rxns=essential, essential_directions=directions, essential_force=ess_force, allow_excretion=step.allow_met_secr, rem_pos_rev=step.pos_rev_off, ignore_mets=step.mets_to_ignore, force_on=force_on, force_on_ess=force_on, big_m=big_m, mip_gap=mg, mip_gap_abs=mga, time_limit=tl, + strict_abs_gap=strict, canonical=canonical, ) + # Strict mode: one solve per step proven to a fixed absolute gap (below the 0.1 score + # granularity), bypassing RAVEN's loose relative escalation — whose final + # near-zero-objective run otherwise accepts an arbitrary within-gap incumbent. Trades + # runtime for a stable, well-defined optimum (pairs naturally with ``canonical``). + if strict_gap: + return _run(None, None, time_limit, strict=_STRICT_ABS_GAP) + if not step.milp_runs: return _run(mip_gap, mip_gap_abs, time_limit) @@ -372,6 +488,8 @@ def ftinit( mip_gap: float | None = None, mip_gap_abs: float | None = 10.0, time_limit: float | None = None, + strict_gap: bool = False, + canonical: bool = False, ) -> cobra.Model: """Run the full ftINIT pipeline on prepData and return the context-specific model. @@ -402,6 +520,32 @@ def ftinit( ``mip_gap``/``time_limit`` are forwarded to each :func:`run_ftinit` solve. On genome-scale models they are essential for tractability — see ``docs/init_param_calibration.md`` for the calibration table. + + ``strict_gap`` and ``canonical`` (both opt-in, default off → exact RAVEN behaviour) + reduce the arbitrariness of the degenerate MILP's tie-break, yielding a more + parsimonious and more *reproducible* extracted model. They do **not** make the model + biologically more accurate: the alternative optima they choose among are equally + consistent with the expression data, so this only pins *which* optimum is returned. It + reduces the fragility of the MILP, not its correctness: + + * ``strict_gap`` replaces the loose relative-gap escalation with a single solve per + step proven to a fixed *absolute* gap (below the 0.1 reaction-score granularity), + so the kept-set objective is optimal at any scale — not an arbitrary within-gap + incumbent. Slower, but removes the largest source of tie-break drift. + * ``canonical`` adds a lexicographic phase 2 that selects the unique sparsest (then + lowest-id) optimum, so the degenerate choice is pinned rather than left to the + solver — applied both to each extraction step and to the task gap-fill. Best used + together with ``strict_gap`` (a well-defined primary optimum to canonicalise). + + Caveats worth carrying into any workflow that uses these: they reduce run-to-run and + platform fragility but do *not* guarantee reproducibility across Gurobi versions + (proving the genome-scale optimum is intractable, so ``strict_gap`` may fall back to an + incumbent), and a downstream metric such as gene essentiality can even shift or worsen, + because the sparser ``canonical`` model is more sensitive to the residual (mostly + transport) degeneracy. For reproducible gene essentiality specifically, pin the solver + stack (raven-toolbox commit + ``gurobipy`` version) rather than relying on these flags. + Rule of thumb: for many models the baseline is fine; reach for these when one or a few + stable, parsimonious model artifacts are wanted. """ if metabolomics: raise NotImplementedError( @@ -436,6 +580,7 @@ def ftinit( min_model, scores, step, essential=essential, directions=directions, ess_force=ess_force, force_on=force_on, big_m=big_m, mip_gap=mip_gap, mip_gap_abs=mip_gap_abs, time_limit=time_limit, + strict_gap=strict_gap, canonical=canonical, ) for rid in res.on_reactions: turned_on[rid] = res.fluxes[rid] @@ -460,7 +605,8 @@ def ftinit( if fill_gaps and prep.tasks: # add reactions back so every task is feasible # The gap-fill MILP is its own problem (RAVEN ftINITFillGaps); it uses RAVEN's # fixed per-task 300 s limit and seed, not the main extraction's time_limit. - out = fill_tasks(out, prep.ref_model, prep.tasks, rxn_scores=rxn_scores).model + out = fill_tasks(out, prep.ref_model, prep.tasks, rxn_scores=rxn_scores, + canonical=canonical).model if gene_scores is not None: # prune negative-scoring genes from the GPRs out, _ = remove_low_score_genes(out, gene_scores) return out diff --git a/src/raven_toolbox/init/taskfill.py b/src/raven_toolbox/init/taskfill.py index ba9bab28..f40b4ee0 100644 --- a/src/raven_toolbox/init/taskfill.py +++ b/src/raven_toolbox/init/taskfill.py @@ -125,9 +125,53 @@ def _set_fill_solver(model: cobra.Model, time_limit: float | None, seed: int) -> model.solver.configuration.timeout = int(time_limit) +def _canonicalize_fill(work, prob, candidates, cost_expr, time_limit) -> list[str] | None: + """Pin the degenerate min-cost gap-fill to a single canonical set (in place on ``work``). + + Like the extraction MILP, the fill has many equal-cost solutions and the solver returns + an arbitrary one (seed/version dependent). Hold the cost at its optimum, then lexicographically + minimise the number of added reactions and then their summed id rank — both integer + objectives, provable with a cheap absolute gap below 1. Returns the chosen reaction ids, + or ``None`` if a phase did not converge (the caller then keeps the arbitrary min-cost fill). + """ + yvars = {cid: work.variables[f"_fill_{cid}"] for cid in candidates} + primary_cost = work.objective.value or 0.0 + tol = max(abs(primary_cost) * 1e-7, 1e-7) + work.add_cons_vars([prob.Constraint(cost_expr, ub=primary_cost + tol, name="_fill_cost_floor")]) + count = add([mul([Real(1.0), y]) for y in yvars.values()]) + + def _phase(objective) -> bool: + work.objective = objective + try: # integer objective → an absolute gap < 1 proves the optimum cheaply. + work.solver.problem.Params.MIPGap = 0.0 + work.solver.problem.Params.MIPGapAbs = 0.4 + except Exception: # noqa: BLE001 - harmless on other backends + pass + work.slim_optimize() + if work.solver.status not in ("optimal", "feasible", "suboptimal", "time_limit"): + return False + try: + return work.solver.problem.SolCount > 0 + except Exception: # noqa: BLE001 - non-Gurobi backend: status is authoritative + return True + + # fewest added reactions (parsimony) ... + if not _phase(prob.Objective(count, direction="min")): + return None + kmin = work.objective.value or 0.0 + # ... then, among the sparsest, the unique lowest-id set. + work.add_cons_vars([prob.Constraint(count, ub=kmin + 0.5, name="_fill_count_cap")]) + ranks = {cid: i for i, cid in enumerate(sorted(candidates))} + idsum = add([mul([Real(float(1 + ranks[cid])), yvars[cid]]) for cid in candidates]) + if not _phase(prob.Objective(idsum, direction="min")): + return None + return [cid for cid in candidates if (yvars[cid].primal or 0.0) > 0.5] + + def _gap_fill_task( reference_model: cobra.Model, present_ids: set[str], task: Task, costs: dict[str, float], *, time_limit: float | None, seed: int, + canonical: bool = False, ) -> list[str]: """Min-cost reference reactions that make ``task`` feasible (RAVEN ``ftINITFillGaps``). @@ -169,7 +213,8 @@ def _gap_fill_task( work.add_cons_vars(extras) # add() over a flat list, not Python sum() — the latter is O(n²) in sympy and with # thousands of candidates dominates gap-fill runtime (see ftINIT/tINIT, same fix). - work.objective = prob.Objective(add(objective_terms), direction="min") + cost_expr = add(objective_terms) + work.objective = prob.Objective(cost_expr, direction="min") _set_fill_solver(work, time_limit, seed) work.slim_optimize() # Accept a near-optimal incumbent (time_limit); only a truly infeasible fill (no @@ -177,8 +222,16 @@ def _gap_fill_task( if work.solver.status not in ("optimal", "feasible", "suboptimal", "time_limit") or \ work.variables[f"_fill_{candidates[0]}"].primal is None: raise OptimizationError(f"gap-filling found no way to make task {task.id!r} feasible.") - return [cid for cid in candidates - if (work.variables[f"_fill_{cid}"].primal or 0.0) > 0.5] + + chosen = [cid for cid in candidates + if (work.variables[f"_fill_{cid}"].primal or 0.0) > 0.5] + # canonical (best-effort): pin the degenerate min-cost fill to the fewest, lowest-id + # reactions so the added set does not depend on the solver seed/version. + if canonical: + canon = _canonicalize_fill(work, prob, candidates, cost_expr, time_limit) + if canon is not None: + chosen = canon + return chosen def fill_tasks( @@ -189,6 +242,7 @@ def fill_tasks( rxn_scores: Mapping[str, float] | None = None, time_limit: float | None = _FILL_TIME_LIMIT, seed: int = _FILL_SEED, + canonical: bool = False, ) -> TaskFillResult: """Add minimum-cost reference reactions so every task is feasible in ``model``. @@ -199,7 +253,9 @@ def fill_tasks( the model, excluding exchange/boundary reactions); ``rxn_scores`` (original reaction id → score) sets each candidate's cost as ``−min(score, −0.1)`` (missing → cost 1). ``should_fail`` tasks are ignored. Each gap-fill MILP is single-threaded with a fixed - ``seed`` and bounded by ``time_limit`` (RAVEN's 300 s). + ``seed`` and bounded by ``time_limit`` (RAVEN's 300 s). ``canonical`` (opt-in) pins the + degenerate min-cost fill to the fewest, lowest-id reactions so the added set does not + depend on the solver seed/version — see :func:`_canonicalize_fill`. Boundary reactions are closed while testing/solving each task, so task inputs and outputs come solely from the task's ranged metabolite bounds (RAVEN gap-fills the exchange-free @@ -225,7 +281,7 @@ def fill_tasks( for r in reference_model.reactions if r.id not in present and not r.boundary} try: chosen = _gap_fill_task(reference_model, present, task, costs, - time_limit=time_limit, seed=seed) + time_limit=time_limit, seed=seed, canonical=canonical) except OptimizationError: failed.append(task.id) continue diff --git a/src/raven_toolbox/io/excel.py b/src/raven_toolbox/io/excel.py index a154a535..5a251905 100644 --- a/src/raven_toolbox/io/excel.py +++ b/src/raven_toolbox/io/excel.py @@ -63,6 +63,30 @@ def _blank_if_empty(value: str) -> str | None: return value or None +def _rxn_bound_cells( + rxn: cobra.Reaction, default_lb: float | None, default_ub: float | None +) -> tuple[float | None, float | None]: + """(LOWER BOUND, UPPER BOUND) cell values for the RXNS sheet. + + Blank (``None``) when a bound matches the model's own declared default, + matching ``exportToExcelFormat`` exactly rather than always writing the + literal number: a reversible reaction (``lb < 0``) hides its lower bound + only when it equals the declared default; an irreversible one hides it + whenever it is exactly 0, regardless of the declared default. Neither rule + applies when the model declares no default at all (both bounds are then + always shown literally, matching exportToExcelFormat's own fallback). + """ + lb, ub = rxn.lower_bound, rxn.upper_bound + if default_lb is None: + lb_cell = lb + elif lb < 0: + lb_cell = None if lb == default_lb else lb + else: + lb_cell = None if lb == 0 else lb + ub_cell = ub if default_ub is None or ub != default_ub else None + return lb_cell, ub_cell + + def _fmt_count(value: float) -> str: """Subunit count as an integer string when integral, else as-is.""" value = float(value) @@ -111,6 +135,8 @@ def export_to_excel( ) genes = sorted(model.genes, key=lambda g: g.id) if sort_ids else list(model.genes) metadata = dict(model.notes.get("metaData", {})) if model.notes else {} + default_lb = metadata.get("defaultLB") + default_ub = metadata.get("defaultUB") wb = Workbook() wb.remove(wb.active) # drop the default empty sheet @@ -124,9 +150,10 @@ def export_to_excel( ) for r in reactions: subsystem = subsystem_to_str(r.subsystem) + lb_cell, ub_cell = _rxn_bound_cells(r, default_lb, default_ub) ws.append([ None, r.id, r.name, _equation(r), _ec_codes(r), r.gene_reaction_rule, - r.lower_bound, r.upper_bound, + lb_cell, ub_cell, r.objective_coefficient or None, None, _miriam_string(r.annotation, exclude=("ec-code",)), subsystem, None, r.notes.get("note"), r.notes.get("references"), r.notes.get("confidence_score"), diff --git a/src/raven_toolbox/io/yaml.py b/src/raven_toolbox/io/yaml.py index eed439c3..37154c56 100644 --- a/src/raven_toolbox/io/yaml.py +++ b/src/raven_toolbox/io/yaml.py @@ -42,13 +42,17 @@ from __future__ import annotations import gzip +import re import warnings from collections import OrderedDict +from datetime import date from pathlib import Path import cobra from cobra.io.dict import model_from_dict, model_to_dict from cobra.io.yaml import yaml as _cobra_yaml # ruamel round-trip YAML (handles !!omap) +from ruamel.yaml import YAML +from ruamel.yaml.scalarstring import DoubleQuotedScalarString from raven_toolbox.io.ec_data import ( EcData, @@ -56,6 +60,29 @@ ec_data_to_yaml_sections, ) +# A dedicated instance for *writing* only (reading still goes through +# _cobra_yaml, whose parsing is unaffected by any of this). Mutating +# _cobra_yaml's own width would also change cobra.io.save_yaml_model for +# the rest of the process, so this module gets its own ruamel round-trip +# instance instead. `width` is set far past any real line length so +# ruamel never folds a long scalar onto a continuation line, matching +# writeYAMLmodel.m. `indent` is pinned explicitly (to the values ruamel +# already used implicitly, verified byte-identical) rather than left for +# ruamel to pick, so a future ruamel version can't silently change the +# on-disk layout out from under either writer. Every list in this format +# is itself a block sequence of single-key `!!omap` entries (reactions, +# a reaction's metabolites, ...), so `sequence=2, offset=0` still nests +# each entry two spaces past its parent key, matching writeYAMLmodel.m's +# own hand-indented style — it would only read as flush-with-the-key for +# a plain `key: [list]` mapping, which this format never emits at any +# level. Quoting still only happens when YAML requires it; the quote +# *character* is double, matching Prettier's YAML default and +# writeYAMLmodel.m (see _needs_quote below for why this needs its own +# check rather than ruamel's default single-quote-when-needed behaviour). +_write_yaml = YAML(typ="rt") +_write_yaml.width = 1_000_000 +_write_yaml.indent(mapping=2, sequence=2, offset=0) + def _open_text(path: str | Path, mode: str): """Open ``path`` as a text handle, transparently gzipping when it ends ``.gz``.""" @@ -198,6 +225,20 @@ def model_from_yaml_data(raw: dict) -> cobra.Model: # current cobra-shaped files. _lift_eccodes_to_annotation(raw.get("reactions")) + # A bare scalar annotation value (writeYAMLmodel.m/write_yaml_model both + # collapse a single-value MIRIAM entry to one) is wrapped into a + # single-item list here, before model_from_dict, so cobra/geckopy always + # see annotation[key] as list[str] regardless of whether the source file + # used a scalar or a list — cobra's own model_from_dict does not do this + # normalisation itself. + for section in ("metabolites", "reactions", "genes"): + _normalize_annotation_values(raw.get(section)) + + # A metabolite with no explicit compartment defaults to the first one, + # matching readYAMLmodel.m's own convention — cobra's model_from_dict + # would otherwise leave met.compartment as None. + _default_missing_compartment(raw.get("metabolites"), raw.get("compartments")) + # Normalise legacy reaction-side YAML keys (e.g. RAVEN MATLAB's # ``rxnNotes`` -> the canonical ``notes``) before any field capture so # the capture step sees a single key per concept. @@ -215,6 +256,13 @@ def model_from_yaml_data(raw: dict) -> cobra.Model: rxn_notes = _capture_entry_fields(raw.get("reactions", []), _RXN_FIELDS) gene_notes = _capture_entry_fields(raw.get("genes", []), _GENE_FIELDS) + # RAVEN MATLAB omits a section entirely when it is empty -- a model with no + # genes has no ``genes:`` block at all. cobra's ``model_from_dict`` indexes + # these keys directly and raises ``KeyError``, so a valid RAVEN file would + # not load. Supply the empty lists it expects. + for section in ("metabolites", "reactions", "genes"): + raw.setdefault(section, []) + model = model_from_dict(raw) for met, notes in zip(model.metabolites, met_notes, strict=False): @@ -269,6 +317,51 @@ def model_from_yaml_data(raw: dict) -> cobra.Model: # Legacy quirk normalisers # --------------------------------------------------------------------------- # +def _normalize_annotation_values(entries) -> None: + """Wrap a bare scalar annotation value into a single-item list. + + Both writers collapse a singleton MIRIAM value to a scalar (see + :func:`_collapse_singleton_annotations`), so a file can carry either + form for the same key. Normalising here — rather than leaving cobra to + store whatever shape the source used — means ``annotation[key]`` is + always ``list[str]`` however the file spelled it, matching what cobra + and geckopy expect and keeping a scalar-collapsed value from coming + back as a bare string after a write/read round trip. Normalises in + place; a no-op when ``entries`` is falsy. + """ + if not entries: + return + for entry in entries: + if not isinstance(entry, dict): + continue + annotation = entry.get("annotation") + if not isinstance(annotation, dict): + continue + for key, value in annotation.items(): + if not isinstance(value, list): + annotation[key] = [value] + + +def _default_missing_compartment(metabolites, compartments) -> None: + """Default a metabolite's missing compartment to the first compartment. + + Matches readYAMLmodel.m's own convention (a metabolite with no + explicit compartment is assigned index 1, i.e. the first entry of + ``compartments:``) — cobra's ``model_from_dict`` has no equivalent + default and would otherwise leave ``met.compartment`` as ``None``. + "First" is the first key of the ``compartments`` mapping, in file + order (preserved by the round-trip YAML loader), matching + ``model.comps{1}`` on the MATLAB side. Normalises in place; a no-op + when there is no compartments section to default to. + """ + if not compartments or not metabolites: + return + first = next(iter(compartments)) + for met in metabolites: + if isinstance(met, dict) and not met.get("compartment"): + met["compartment"] = first + + def _lift_smiles_to_annotation(metabolites) -> None: """Move per-metabolite top-level ``smiles`` into ``annotation['smiles']``. @@ -391,6 +484,208 @@ def _emit_entry_fields(entries, fields): entry["notes"] = notes +def _coerce_floats(obj): + """Recursively coerce every plain ``int`` in ``obj`` to ``float``. + + Every number is written as an explicit float (``2.0``, not ``2``), + matching writeYAMLmodel.m. ``bool`` is checked first since it is an + ``int`` subclass in Python and must not be turned into ``0.0``/``1.0``. + """ + if isinstance(obj, bool): + return obj + if isinstance(obj, int): + return float(obj) + if isinstance(obj, dict): + return type(obj)((k, _coerce_floats(v)) for k, v in obj.items()) + if isinstance(obj, list): + return [_coerce_floats(v) for v in obj] + return obj + + +_QUOTE_RESOLVER_PATTERN = re.compile( + r"^(true|True|TRUE|false|False|FALSE)$" + r"|^[-+]?[0-9][0-9_]*\.[0-9_]*([eE][-+]?[0-9]+)?$" + r"|^[-+]?[0-9][0-9_]*[eE][-+]?[0-9]+$" + r"|^[-+]?\.[0-9_]+([eE][-+][0-9]+)?$" + r"|^[-+]?\.(inf|Inf|INF)$" + r"|^\.(nan|NaN|NAN)$" + r"|^[-+]?0b[01_]+$" + r"|^[-+]?0o?[0-7_]+$" + r"|^[-+]?[0-9_]+$" + r"|^[-+]?0x[0-9a-fA-F_]+$" + r"|^(~|[Nn]ull|NULL)$" + r"|^[0-9]{4}-[0-9]{2}-[0-9]{2}$" +) + + +def _is_ws(ch: str) -> bool: + """The four characters writeYAMLmodel.m's needsQuote treats as whitespace. + + Deliberately narrower than str.isspace(), which also matches \\v, \\f and + other Unicode whitespace the MATLAB port never considered. + """ + return ch in (" ", "\t", "\n", "\r") + + +def _needs_quote(s: str) -> bool: + """Whether this scalar needs quoting when written as YAML. + + Ports writeYAMLmodel.m's needsQuote so both writers quote exactly the + same set of values, for the same two reasons: a plain (bare) reading + would resolve to a non-string YAML type (bool/int/float/null/ + timestamp — neither writer emits explicit type tags), or the text + itself is not valid as a plain block scalar (leading indicator + character, a mid-string ": "/" #", leading/trailing whitespace, or an + embedded line break). + """ + if not s: + return True + if _QUOTE_RESOLVER_PATTERN.match(s): + return True + first = s[0] + if first in "#,[]{}&*!|>'\"%@`": + return True + followed_by_ws = len(s) == 1 or _is_ws(s[1]) + if first in "?:" and followed_by_ws: + return True + if first == "-" and followed_by_ws: + return True + if _is_ws(first) or _is_ws(s[-1]): + return True + if "\n" in s or "\r" in s: + return True + for i in range(1, len(s)): + ch = s[i] + if ch == ":" and (i == len(s) - 1 or _is_ws(s[i + 1])): + return True + if ch == "#" and _is_ws(s[i - 1]): + return True + return False + + +def _prefer_double_quotes(obj): + """Recursively mark every scalar string that needs quoting as double-quoted. + + ruamel's round-trip dumper single-quotes a plain ``str`` when quoting + is syntactically required; there is no global "quote with this + character when needed" switch, only a per-value style override. Wrap + exactly the values :func:`_needs_quote` says need quoting in + ``DoubleQuotedScalarString`` — never unconditionally, or every scalar + would end up quoted regardless of whether YAML requires it. + """ + if isinstance(obj, str): + return DoubleQuotedScalarString(obj) if _needs_quote(obj) else obj + if isinstance(obj, dict): + return type(obj)((k, _prefer_double_quotes(v)) for k, v in obj.items()) + if isinstance(obj, list): + return [_prefer_double_quotes(v) for v in obj] + return obj + + +_LIST_ONLY_ANNOTATION_KEYS = frozenset({"ec-code", "smiles"}) + + +def _collapse_singleton_annotations(entries) -> None: + """Collapse a one-item annotation list to a bare scalar. + + Matches writeYAMLmodel.m's MIRIAM handling: every annotation entry + collapses to a scalar when it carries a single value, except + ``ec-code``/``smiles`` — RAVEN's own writer marks exactly those two + ``forceList`` — which stay a list even for one value, since cobra and + geckopy read them as ``list[str]``. Normalises in place. + """ + for entry in entries: + if not isinstance(entry, dict): + continue + annotation = entry.get("annotation") + if not isinstance(annotation, dict): + continue + for key, value in annotation.items(): + if ( + key not in _LIST_ONLY_ANNOTATION_KEYS + and isinstance(value, list) + and len(value) == 1 + ): + annotation[key] = value[0] + + +_META_ANNOTATION_FIELDS = ( + "givenName", + "familyName", + "authors", + "email", + "organization", + "taxonomy", + "note", + "sourceUrl", +) + + +def _default_bounds(model: cobra.Model) -> tuple[float, float] | None: + """The model's ``(min lower_bound, max upper_bound)`` across all reactions. + + Mirrors readYAMLmodel.m's ``defaultLB``/``defaultUB`` + (``min(model.lb)``/``max(model.ub)``), recomputed fresh from the + model's current bounds — like the MATLAB reader recomputes them from + the just-parsed bounds — rather than echoed from a stored value, so a + model whose bounds changed after loading still gets a correct + default. ``None`` for a model with no reactions (nothing to derive + from). + """ + if not model.reactions: + return None + lbs = [rxn.lower_bound for rxn in model.reactions] + ubs = [rxn.upper_bound for rxn in model.reactions] + return min(lbs), max(ubs) + + +def _build_metadata(model: cobra.Model, stored_meta: dict, version) -> OrderedDict: + """Assemble the ``metaData`` block in writeYAMLmodel.m's fixed field order. + + ``id``/``name`` are always present (``"blankID"``/``"blankName"`` when + unset, matching writeMetadata's own ``valueOrDefault``), as is + ``date`` (today's date when the model carries none). ``defaultLB``/ + ``defaultUB`` are recomputed from the model's current bounds (see + :func:`_default_bounds`) rather than merely echoed from a stored + value. The remaining fields follow in the same order + writeYAMLmodel.m's ``annoFields`` emits them, each only when present + and non-empty. + """ + metadata: OrderedDict = OrderedDict() + metadata["id"] = model.id or "blankID" + metadata["name"] = model.name or "blankName" + if version is not None: + metadata["version"] = version + metadata["date"] = stored_meta.get("date") or date.today().isoformat() + bounds = _default_bounds(model) + if bounds is not None: + metadata["defaultLB"], metadata["defaultUB"] = bounds + for key in _META_ANNOTATION_FIELDS: + value = stored_meta.get(key) + if value: + metadata[key] = value + return metadata + + +def _normalize_subsystems(reactions) -> None: + """Drop a reaction's ``subsystem`` key when it carries no subsystem. + + A reaction that does carry one or more subsystems is normalised to a + plain list of non-empty strings, even for a single subsystem, matching + writeYAMLmodel.m. Normalises in place. + """ + for rxn in reactions: + if not isinstance(rxn, dict) or "subsystem" not in rxn: + continue + value = rxn["subsystem"] + items = value if isinstance(value, list) else [value] + cleaned = [str(s) for s in items if s] + if cleaned: + rxn["subsystem"] = cleaned + else: + del rxn["subsystem"] + + def write_yaml_model( model: cobra.Model, path: str | Path, *, sort_ids: bool = False ) -> None: @@ -433,6 +728,18 @@ def write_yaml_model( _emit_entry_fields(doc.get("metabolites", []), _MET_FIELDS) _emit_entry_fields(doc.get("reactions", []), _RXN_FIELDS) _emit_entry_fields(doc.get("genes", []), _GENE_FIELDS) + _normalize_subsystems(doc.get("reactions", []) or ()) + # An unset charge is left omitted (cobra's own default), not defaulted to + # 0 — readYAMLmodel.m's own fill-missing-fields step is a positional + # backfill shared by several columns, not a "charge defaults to 0" rule: + # a gap before some later metabolite that does carry a charge parses as + # NaN (omitted on write, like deltaG); only a gap at the very end of the + # metabolite list — after the last one with any charge — hits the + # different, unrelated tail-padding branch and becomes a real 0. That is + # a position-dependent side effect of shared fill code, not a + # convention worth reproducing here. + for section in ("metabolites", "reactions", "genes"): + _collapse_singleton_annotations(doc.get(section, []) or ()) # cobra's _gene_to_dict always emits `name: ''` because name is a # required attribute; RAVEN MATLAB skips empty names. Drop the @@ -460,13 +767,7 @@ def write_yaml_model( # are NOT emitted at root level. Cobra reading such a file recovers # the lists and compartments and leaves model.id empty (consistent # with how RAVEN MATLAB has always laid these files out). - metadata = OrderedDict(stored_meta) - if model.id: - metadata.setdefault("id", model.id) - if model.name: - metadata.setdefault("name", model.name) - if version is not None: - metadata.setdefault("version", version) + metadata = _build_metadata(model, stored_meta, version) # cobra's model_to_dict put id / name at root level; drop them so they # don't duplicate the metaData copy. @@ -474,8 +775,7 @@ def write_yaml_model( doc.pop("name", None) ordered = OrderedDict() - if metadata: - ordered["metaData"] = metadata + ordered["metaData"] = metadata for key in ("metabolites", "reactions", "genes", "compartments", "notes"): if key in doc: ordered[key] = doc.pop(key) @@ -491,4 +791,4 @@ def write_yaml_model( ordered.setdefault(key, value) with _open_text(path, "w") as handle: - _cobra_yaml.dump(ordered, handle) + _write_yaml.dump(_prefer_double_quotes(_coerce_floats(ordered)), handle) diff --git a/src/raven_toolbox/localization/certify.py b/src/raven_toolbox/localization/certify.py index b84df744..9205f7e4 100644 --- a/src/raven_toolbox/localization/certify.py +++ b/src/raven_toolbox/localization/certify.py @@ -29,6 +29,7 @@ from __future__ import annotations import math +import warnings from collections.abc import Callable, Iterable, Mapping from dataclasses import dataclass @@ -126,6 +127,31 @@ def _prepare_scope(model, scores, reactions_to_relocate, *, transportable, base_ # --------------------------------------------------------------------------- master +def _pin_deterministic(prob, opt) -> None: + """Pin solver parameters so a degenerate placement objective is resolved the + same way every run and across solvers/machines. On a benchmark of the yeast + master (identical model, one parameter varied at a time) exactly three + settings changed which co-optimal placement Gurobi returned -- thread count, + seed, and presolve level -- while the MIP gap only mattered once loosened + past ~1e-2 (which also loses optimality) and every tolerance was irrelevant. + So: single thread and fixed seed remove thread/seed nondeterminism, a fixed + presolve level (2, matching the MATLAB port's optimizeProb default) removes + the presolve divergence, and a zero MIP gap forces the exact optimum. Only + Gurobi exposes these by the names used here; other backends keep defaults.""" + if "gurobi" not in getattr(prob, "__name__", ""): + return + try: + gp_model = opt.problem + gp_model.update() + gp_model.setParam("Threads", 1) + gp_model.setParam("Seed", 0) + gp_model.setParam("Presolve", 2) + gp_model.setParam("MIPGap", 0.0) + gp_model.setParam("IntFeasTol", 1e-9) + except Exception: # noqa: BLE001 — reproducibility hint, never fatal + pass + + def _score(score_df, g: str, c: str) -> float: if c not in score_df.columns or g not in score_df.index: return 0.0 @@ -133,8 +159,14 @@ def _score(score_df, g: str, c: str) -> float: return 0.0 if v is None or (isinstance(v, float) and math.isnan(v)) else float(v) +# Tie-break weight for the placement master's second pass: small enough never to override a real +# per-reaction score difference (DeepLoc scores are ~[0, 1] summed over a reaction's genes), just enough +# to send a genes-free or score-tied reaction to the default compartment deterministically. +_DEFAULT_COMPARTMENT_PRIOR = 1e-3 + + def _solve_placement_master( - model, movable, genes_in_scope, gene_rxns, score_df, compartments, *, + model, movable, genes_in_scope, gene_rxns, score_df, compartments, default_compartment, *, multi_compartment_penalty, forced, colocation_groups, time_limit, ): """Flux-free score-maximising placement MILP (mono-localisation). @@ -144,24 +176,36 @@ def _solve_placement_master( compartment (``x[a, c] = x[b, c]``), letting the score objective pick which one. No flux and no growth floor at all in the master — the placement can never harvest a compartment's score through leaked flux. + + Solved in two lexicographic passes: the primary objective places genes by score (and penalises + spread), which leaves the per-reaction placement ``x`` a free co-optimum; the second pass then fixes + the gene layout and places each reaction in the compartment its own enzymes score highest for, so the + reaction placement is deterministic and evidence-aligned rather than an arbitrary co-optimal vertex. """ model.solver # noqa: B018 — initialise the solver so model.problem is usable prob = model.problem opt = prob.Model() + # Build the MILP in a canonical order so the solver is handed the same + # problem every run and across the MATLAB port: movable is already sorted by + # id, but genes_in_scope is a set whose iteration order is randomised per + # process (string hash randomisation), which would otherwise reorder the + # variables/constraints and let a degenerate objective pick a different + # co-optimal placement each run. Iterate it sorted everywhere. + genes_sorted = sorted(genes_in_scope) x = {(r.id, c): prob.Variable(f"x_{r.id}_{c}", type="binary") for r in movable for c in compartments} y = {(g, c): prob.Variable(f"y_{g}_{c}", type="binary") - for g in genes_in_scope for c in compartments} + for g in genes_sorted for c in compartments} cons: list = [] for r in movable: cons.append(prob.Constraint(add([x[r.id, c] for c in compartments]), lb=1.0, ub=1.0, name=f"place_{r.id}")) - for g in {gg.id for gg in r.genes} & genes_in_scope: + for g in sorted({gg.id for gg in r.genes} & genes_in_scope): for c in compartments: cons.append(prob.Constraint(x[r.id, c] - y[g, c], ub=0.0, name=f"couple_{r.id}_{g}_{c}")) - for g in genes_in_scope: + for g in genes_sorted: cons.append(prob.Constraint(add([y[g, c] for c in compartments]), lb=1.0, name=f"gene1_{g}")) for c in compartments: @@ -180,20 +224,52 @@ def _solve_placement_master( opt.add(list(x.values()) + list(y.values()) + cons) - obj = [] - for g in genes_in_scope: + # Primary objective: place each gene in the compartment(s) its DeepLoc score favours, penalising + # spread across compartments. This fixes the gene layout but leaves the *reaction* placement x a free + # co-optimum (the objective never mentions x), which a deterministic solver resolves to an arbitrary + # vertex -- yeast-GEM reaction agreement 52.8%. A lexicographically-lower second pass then chooses x. + primary = [] + for g in genes_sorted: for c in compartments: s = _score(score_df, g, c) if s: - obj.append(mul([Real(s), y[g, c]])) + primary.append(mul([Real(s), y[g, c]])) if multi_compartment_penalty: for v in y.values(): - obj.append(mul([Real(-multi_compartment_penalty), v])) - opt.objective = prob.Objective(add(obj) if obj else Real(0.0), direction="max") + primary.append(mul([Real(-multi_compartment_penalty), v])) + opt.objective = prob.Objective(add(primary) if primary else Real(0.0), direction="max") if time_limit is not None: opt.configuration.timeout = int(time_limit) + _pin_deterministic(prob, opt) opt.optimize() + if opt.status not in ("optimal", "feasible", "suboptimal", "time_limit"): + return opt.status, {}, {} + # Lexicographic second pass. Fix the gene layout to the primary optimum -- fixing the *solution* (each + # y binary), not the objective value, so there is no near-optimal tolerance to tune -- then place each + # reaction in the compartment its own enzymes are predicted to occupy: reward x[r, c] by the summed + # gene score of r's genes for c, with a small prior for `default_compartment` so genes-free and + # score-tied reactions fall there deterministically rather than to an arbitrary co-optimal vertex. + # The gene layout (agreement, multi-compartment consolidation) is untouched; only reaction placement, + # which was free, is now meaningful. + try: # read every primal before touching any bound -- the first bound change discards the solution + y_star = {k: 1.0 if (v.primal or 0.0) >= 0.5 else 0.0 for k, v in y.items()} + except (AttributeError, ValueError): + return "no_incumbent", {}, {} + for k, v in y.items(): + v.lb = v.ub = y_star[k] + secondary = [] + for r in movable: + r_genes = sorted({gg.id for gg in r.genes} & genes_in_scope) + for c in compartments: + w = sum(_score(score_df, g, c) for g in r_genes) + if c == default_compartment: + w += _DEFAULT_COMPARTMENT_PRIOR + if w: + secondary.append(mul([Real(w), x[r.id, c]])) + opt.objective = prob.Objective(add(secondary) if secondary else Real(0.0), direction="max") + _pin_deterministic(prob, opt) + opt.optimize() if opt.status not in ("optimal", "feasible", "suboptimal", "time_limit"): return opt.status, {}, {} try: @@ -588,7 +664,7 @@ def assign_compartments( seen.add(signature) status, placements, gene_comps = _solve_placement_master( - model, movable, genes_in_scope, gene_rxns, score_df, compartments, + model, movable, genes_in_scope, gene_rxns, score_df, compartments, default_compartment, multi_compartment_penalty=multi_compartment_penalty, forced=forced, colocation_groups=groups, time_limit=time_limit) if not placements: @@ -664,19 +740,63 @@ def assign_compartments( def _gapfill(applied, universal, biomass_reaction, min_growth) -> list[str]: - """Minimal gap-fill reactions (from ``universal``) that restore the primary growth floor. - - Uses ``cobra.flux_analysis.gapfill``; the result is validated by the caller's certification FBA, - so any tolerance quirk in cobra's own MILP cannot produce a false certificate here. + """The ``universal`` reactions whose addition restores the primary growth floor, via a flux-based fill. + + On a working copy: add every ``universal`` candidate not already present in one batch, check the floor + is even reachable, hold biomass at the floor, run pFBA (maximise biomass, then minimise total flux), + and return -- sorted -- the added reactions that carry flux. pFBA makes the set flux-parsimonious (not + guaranteed reaction-count-minimal); the caller re-certifies with a real FBA regardless, so no false + certificate is possible. + + An LP, not cobra's indicator MILP — ``cobra.flux_analysis.gapfill`` at genome scale fails to find a + valid fill in the *majority* of cases even when the exact restoring reaction is present in the + universal (its own validation then rejects the broken incumbent and raises); on single-reaction + knockout-recovery this restores every case where that MILP restores under half. + + **Namespace.** Candidates are matched to the model by metabolite id (as cobra's gapfill required): the + universal must share the draft's metabolite namespace. A candidate whose metabolites do not resolve + becomes a dead-end that cannot carry flux and is left out — and a warning fires when most candidates + fail to resolve, so a silent empty result is distinguishable from a namespace mismatch. """ + if biomass_reaction not in applied.reactions: + return [] + floor = max(min_growth, 1e-4) try: - from cobra.flux_analysis import gapfill as cobra_gapfill - with applied: - applied.objective = biomass_reaction - solutions = cobra_gapfill(applied, universal, lower_bound=max(min_growth, 1e-4), - demand_reactions=False, iterations=1) - return [r.id for r in solutions[0]] if solutions else [] - except Exception: # noqa: BLE001 — infeasible gap-fill / backend quirk: report none added + from cobra.flux_analysis import pfba + # Work on a copy: no context-manager rollback (whose failure on some optlang backends would + # otherwise discard a valid result), and the caller rebuilds `applied` from the proposal anyway. + work = applied.copy() + candidates = [u for u in universal.reactions if u.id not in work.reactions] + if not candidates: + return [] + fresh = [cobra.Reaction(u.id, name=u.name, lower_bound=u.lower_bound, upper_bound=u.upper_bound) + for u in candidates] + work.add_reactions(fresh) # one batch — per-reaction adds are super-linear at scale + fully_unresolved = 0 + for nr, urxn in zip(fresh, candidates, strict=True): + stoich, n_unres = {}, 0 + for met, coeff in urxn.metabolites.items(): + if met.id in work.metabolites: + stoich[work.metabolites.get_by_id(met.id)] = coeff + else: + n_unres += 1 + stoich[cobra.Metabolite(met.id, name=met.name, formula=met.formula, + charge=met.charge, compartment=met.compartment)] = coeff + nr.add_metabolites(stoich) + fully_unresolved += n_unres == len(urxn.metabolites) and len(urxn.metabolites) > 0 + if fully_unresolved > 0.5 * len(candidates): + warnings.warn( + f"gap-fill: {fully_unresolved}/{len(candidates)} universal candidates share no metabolite " + "id with the model — a likely namespace mismatch; gap-fill will find little or nothing.", + stacklevel=2) + + work.objective = biomass_reaction + if (work.slim_optimize(error_value=0.0) or 0.0) < floor - 1e-9: + return [] # unfixable even with every candidate present + work.reactions.get_by_id(biomass_reaction).lower_bound = floor + fluxes = pfba(work).fluxes + return sorted(nr.id for nr in fresh if abs(fluxes.get(nr.id, 0.0)) > 1e-9) + except Exception: # noqa: BLE001 — infeasible / backend quirk: report none added return [] diff --git a/src/raven_toolbox/localization/curation.py b/src/raven_toolbox/localization/curation.py index 7f708397..abeceb59 100644 --- a/src/raven_toolbox/localization/curation.py +++ b/src/raven_toolbox/localization/curation.py @@ -130,6 +130,7 @@ def curation_priority( check_essential: bool = True, essential_candidate_threshold: float = 0.2, min_growth: float | None = None, + include_curated: bool = False, ): """Rank reaction placements and added transports by how likely each needs manual curation. @@ -155,6 +156,11 @@ def curation_priority( (heuristic) ``min_growth`` floor, and is skipped entirely if the materialised model cannot beat that floor (where the test could not discriminate). Set ``check_essential=False`` to skip the only non-cheap signal. + + A reaction whose ``localization`` confidence has been marked ``curated`` (via + :func:`~raven_toolbox.confidence.mark_curated`, e.g. after a curator settled its placement) is + dropped from the queue, closing the score -> review -> curate loop so a settled reaction stops + resurfacing. Pass ``include_curated=True`` to score it anyway. """ import pandas as pd @@ -165,6 +171,18 @@ def curation_priority( # gap-fills that are also (somehow) in placements are scored as placements, not duplicated as gapfill added_reactions = [r for r in proposal.added_reactions if r not in proposal.placements] + # Reactions a curator has settled (localization facet marked "curated"): drop them from the queue so + # they stop resurfacing, closing the score -> review -> curate loop. include_curated keeps them. + curated_localization: set[str] = set() + if not include_curated: + from raven_toolbox.confidence import get_confidence + for rid in {*proposal.placements, *added_reactions}: + if rid not in model.reactions: + continue + entry = get_confidence(model.reactions.get_by_id(rid)).facets.get("localization") + if entry is not None and entry.level == "curated": + curated_localization.add(rid) + def _cur(b): return _is_currency(b, extra_currency) @@ -207,6 +225,8 @@ def _gene_consensus(rid): # ---- placement signals: one row per (reaction, compartment) ---- for rid, cs in placements_of.items(): + if rid in curated_localization: + continue r = model.reactions.get_by_id(rid) noncur = [(base(m), coeff) for m, coeff in r.metabolites.items() if not _cur(base(m))] subs = [b for b, coeff in noncur if coeff < 0] @@ -259,6 +279,8 @@ def _gene_consensus(rid): # ---- H3 gap-fill: reactions pulled from the universal model (not in the draft's placements) ---- for rid in added_reactions: + if rid in curated_localization: + continue comp = default_compartment if universal is not None and rid in universal.reactions: csx = {mm.compartment for mm in universal.reactions.get_by_id(rid).metabolites} diff --git a/src/raven_toolbox/localization/transport_evidence.py b/src/raven_toolbox/localization/transport_evidence.py index 2c142907..6b887fa8 100644 --- a/src/raven_toolbox/localization/transport_evidence.py +++ b/src/raven_toolbox/localization/transport_evidence.py @@ -167,6 +167,12 @@ def evidence_aware_transport_cost( Parameters ---------- + model: + the model to cost; every compartment-agnostic metabolite base in it gets an entry in the + result. + annotation: + gene -> transporter evidence, as returned by :func:`annotate_transporters` or + :func:`annotate_proteome`. gene_compartments: gene -> predicted compartment ids (the reliable DeepLoc *compartment* calls). A carrier at compartment *X* is taken to support transports across *X*'s boundary. @@ -186,9 +192,15 @@ def evidence_aware_transport_cost( base_cost: the flat cost for an unsupported transport (recovers today's constant behaviour when no gene supports the metabolite). - - Returns every metabolite base -> cost (unsupported metabolites map to ``base_cost``), so the result - is a self-contained ``transport_cost`` mapping. + base_metabolite: + metabolite -> the compartment-agnostic key to group it under (default: strip a trailing + ``_`` from the id). + + Returns + ------- + dict[str, float] + every metabolite base -> cost (unsupported metabolites map to ``base_cost``), so the result + is a self-contained ``transport_cost`` mapping. """ base_of = base_metabolite or _default_base chebi_of = metabolite_chebi or default_metabolite_chebi diff --git a/src/raven_toolbox/manipulation/__init__.py b/src/raven_toolbox/manipulation/__init__.py index ea0ec962..dfaac77b 100644 --- a/src/raven_toolbox/manipulation/__init__.py +++ b/src/raven_toolbox/manipulation/__init__.py @@ -4,10 +4,11 @@ from .add import add_reactions_from_equations from .boundary import close_model from .change import change_gene_reaction_rules, change_reaction_equations +from .compartments import copy_to_compartment, merge_compartments from .expand import expand_model, gpr_to_dnf from .irreversible import convert_to_irreversible from .merge import merge_models -from .parameters import set_variance_bounds +from .parameters import set_exchange_bounds, set_variance_bounds from .remove import remove_genes, remove_metabolites from .simplify import ( constrain_reversible_reactions, @@ -31,10 +32,12 @@ "close_model", "constrain_reversible_reactions", "convert_to_irreversible", + "copy_to_compartment", "expand_model", "find_duplicate_reactions", "gpr_to_dnf", "group_linear_reactions", + "merge_compartments", "merge_models", "remove_dead_end_reactions", "remove_duplicate_reactions", @@ -42,6 +45,7 @@ "remove_metabolites", "remove_no_flux_reactions", "remove_zero_interval_reactions", + "set_exchange_bounds", "set_variance_bounds", "simplify_model", ] diff --git a/src/raven_toolbox/manipulation/add.py b/src/raven_toolbox/manipulation/add.py index ef8f7ed1..3ed37abd 100644 --- a/src/raven_toolbox/manipulation/add.py +++ b/src/raven_toolbox/manipulation/add.py @@ -158,7 +158,10 @@ def _resolve_metabolite( f"Cannot create metabolite {token!r}: no compartment given." ) _warn_unknown_compartment(model, compartment, token) - met = Metabolite(token, compartment=compartment) + # name defaults to the id, matching RAVEN's addRxns --- an unnamed + # metabolite is invisible to anything that keys on name (merge_compartments, + # add_reactions_from_model, mergeCompartments on the MATLAB side). + met = Metabolite(token, name=token, compartment=compartment) model.add_metabolites([met]) met_index.setdefault((met.name, met.compartment), met) return met diff --git a/src/raven_toolbox/manipulation/compartments.py b/src/raven_toolbox/manipulation/compartments.py index e7c4c67d..ab11148a 100644 --- a/src/raven_toolbox/manipulation/compartments.py +++ b/src/raven_toolbox/manipulation/compartments.py @@ -9,6 +9,7 @@ """ from __future__ import annotations +import warnings from collections.abc import Callable, Iterable import cobra @@ -55,10 +56,13 @@ def merge_compartments( with; this transform only ever inherits them.) Metabolites that share a key collapse into one entity in the merged compartment; their - stoichiometric contributions are summed per reaction. Reactions that end up with - only one metabolite (e.g. ``A[c] → A[m]`` becomes ``A → A`` = nothing) are deleted - by default (RAVEN's ``deleteRxnsWithOneMet``). Reactions that become identical - after merging are deduplicated (one survives). + stoichiometric contributions are summed per reaction. Reactions that *become* trivial + through the merge (e.g. ``A[c] → A[m]`` becomes ``A → A`` = nothing) are deleted by + default (RAVEN's ``deleteRxnsWithOneMet``). A reaction that already had a single + metabolite before merging is an exchange (``=> glucose``) and is always kept — + dropping those would silently remove the model's medium and its biomass reaction. + Reactions that become identical after merging are deduplicated (one survives). The + objective is carried over to the returned model. """ out = model.copy() key = base_metabolite if base_metabolite is not None else _base_id @@ -112,7 +116,12 @@ def merge_compartments( keep_reactions: list[cobra.Reaction] = [] for r in out.reactions: stoich = rewritten[r.id] - if drop_single_metabolite_reactions and len(stoich) <= 1: + # Only reactions that *became* trivial through merging are dropped. One + # that already had a single metabolite is an exchange/boundary reaction + # (``=> glucose``), and deleting those silently strips the model of its + # medium and its biomass reaction. RAVEN reserves them the same way. + was_single = len(r.metabolites) <= 1 + if drop_single_metabolite_reactions and len(stoich) <= 1 and not was_single: deleted_single.append(r.id) continue if not stoich: # everything cancelled @@ -132,6 +141,25 @@ def merge_compartments( new_r.notes = dict(r.notes or {}) keep_reactions.append(new_r) merged.add_reactions(keep_reactions) + + # The merged model is built from scratch, so the objective has to be carried + # over explicitly -- without this the caller gets a model that optimises to + # 0.0 with no indication why. + surviving = {r.id for r in keep_reactions} + objective = { + merged.reactions.get_by_id(r.id): r.objective_coefficient + for r in out.reactions + if r.objective_coefficient and r.id in surviving + } + if objective: + merged.objective = objective + merged.objective_direction = out.objective_direction + elif any(r.objective_coefficient for r in out.reactions): + warnings.warn( + "merge_compartments: every reaction carrying an objective coefficient was " + "removed by the merge; the returned model has no objective.", + stacklevel=2, + ) return merged, deleted_single, deduplicated diff --git a/src/raven_toolbox/manipulation/irreversible.py b/src/raven_toolbox/manipulation/irreversible.py index 5c6c3dcf..be6ae9e4 100644 --- a/src/raven_toolbox/manipulation/irreversible.py +++ b/src/raven_toolbox/manipulation/irreversible.py @@ -10,6 +10,8 @@ """ from __future__ import annotations +import copy + import cobra @@ -22,8 +24,13 @@ def convert_to_irreversible(model: cobra.Model) -> list[str]: - A new reaction with the same ID plus a ``_REV`` suffix is added, representing the reverse direction. Its stoichiometry is the negation of the original, its bounds are ``(0, -original_lb)``, - and it inherits the name (with " (reversible)" appended) and the - gene-protein rule of the original. + and it inherits the name (with " (reversible)" appended), the + gene-protein rule, the subsystem, the annotations and the notes + of the original. MATLAB's ``convertToIrrev`` copies the same + per-reaction fields across (``eccodes``, ``rxnMiriams``, + ``subSystems``, ``rxnNotes``, ...); dropping them would, among + other things, leave the reverse reaction without an EC code and + so without a kcat downstream. Exchange reactions (boundary reactions) are never split, regardless of their bounds, matching MATLAB behavior where exchange reactions @@ -59,6 +66,9 @@ def convert_to_irreversible(model: cobra.Model) -> list[str]: rev_rxn.upper_bound = -original_lb rev_rxn.add_metabolites({m: -c for m, c in rxn.metabolites.items()}) rev_rxn.gene_reaction_rule = rxn.gene_reaction_rule + rev_rxn.subsystem = rxn.subsystem + rev_rxn.annotation = copy.deepcopy(rxn.annotation) + rev_rxn.notes = copy.deepcopy(rxn.notes) reverse_rxns_to_add.append(rev_rxn) forward_updates.append(rxn) diff --git a/src/raven_toolbox/manipulation/parameters.py b/src/raven_toolbox/manipulation/parameters.py index f3498044..5f6684bf 100644 --- a/src/raven_toolbox/manipulation/parameters.py +++ b/src/raven_toolbox/manipulation/parameters.py @@ -1,4 +1,5 @@ -"""Set reaction bounds to a sign-aware ±% variance band around measured values. +"""Set reaction bounds to a sign-aware ±% variance band around measured values, and +set exchange-flux bounds for a chosen set of metabolites. Cobra has no idiom for the *variance band* case (e.g. "5 ± 20 %"); the other common bound-setting cases are cobra one-liners: @@ -7,13 +8,19 @@ * equality → ``reaction.bounds = (v, v)`` * objective → ``model.objective = {reaction: coeff}`` * unconstrained → ``reaction.bounds = cobra.Configuration().bounds`` +* a medium → ``model.medium = {ex_id: uptake}`` (reaction-id-keyed, one bound only) + +``model.medium`` doesn't cover ``set_exchange_bounds``'s case: independent lb and ub +per metabolite, a media-only compartment filter, and a check that every exchange +reaction agrees on which sign of flux is import before touching any of them. """ from __future__ import annotations +import warnings from collections.abc import Iterable, Sequence import cobra -from cobra import Reaction +from cobra import Metabolite, Reaction Number = int | float @@ -76,3 +83,151 @@ def set_variance_bounds( lo, hi = v * (1 - half), v * (1 + half) rxn.bounds = (hi, lo) if v < 0 else (lo, hi) return rxns + + +def _exchange_reactions(model: cobra.Model, media_only: bool) -> list[Reaction]: + rxns = [r for r in model.reactions if r.boundary] + if not media_only: + return rxns + extracellular_ids = { + cid for cid, name in model.compartments.items() if name.lower() == "extracellular" + } + if not extracellular_ids: + raise ValueError('Could not find any compartments named "extracellular".') + return [r for r in rxns if next(iter(r.metabolites)).compartment in extracellular_ids] + + +def _resolve_met(model: cobra.Model, exchanged: list[Metabolite], token) -> Metabolite | None: + """Match ``token`` (a Metabolite, id, or name) against the *exchanged* set only — + a metabolite that exists in the model but isn't exchanged is unused here, same as + a metabolite that doesn't exist at all.""" + if isinstance(token, Metabolite): + return token if token in exchanged else None + by_id = {m.id: m for m in exchanged} + if token in by_id: + return by_id[token] + by_name = {m.name.lower(): m for m in exchanged} + return by_name.get(str(token).lower()) + + +def set_exchange_bounds( + model: cobra.Model, + mets: str | Metabolite | Iterable[str | Metabolite] | None = None, + lb: Number | Sequence[Number] | None = None, + ub: Number | Sequence[Number] | None = None, + *, + close_others: bool = True, + media_only: bool = False, +) -> list[Metabolite]: + """Set exchange-flux bounds for a chosen set of metabolites. + + Port of RAVEN ``setExchangeBounds``. Exchange reactions are cobra's own boundary + reactions (single metabolite); a reaction where the same metabolite is exchanged + more than once is warned about but still handled, matching RAVEN. + + Parameters + ---------- + mets + Metabolite objects, ids, or names (case-insensitive), matched against the + model's exchanged metabolites specifically — a real metabolite that isn't + exchanged counts as unused, same as one that doesn't exist. ``None`` (default) + means every exchanged metabolite, in which case ``lb``/``ub`` must each be a + single value, not a sequence. + lb, ub + A single value applied to every metabolite in ``mets``, or one value per + metabolite. Default to ``cobra.Configuration().bounds`` when not given. + close_others + Close import (only import — export is left alone) on every other exchange + reaction not covered by ``mets``. Forced off, with a warning, if the model's + exchange reactions don't agree on which sign of flux is import: correctly + closing "import" would then require a different bound (lb vs ub) reaction by + reaction, which this function does not attempt. + media_only + Only consider exchange reactions whose metabolite is in the compartment named + "extracellular" (by ``model.compartments``, case-insensitive) — e.g. skips + sink/demand reactions on an intracellular metabolite. Raises if the model has + no compartment with that name. + + Returns + ------- + list of cobra.Metabolite + Metabolites in ``mets`` that were not found among the model's exchanged + metabolites. + """ + exch_rxns = _exchange_reactions(model, media_only) + exchanged = list(dict.fromkeys(next(iter(r.metabolites)) for r in exch_rxns)) + + signs = {next(iter(r.metabolites.values())) >= 0 for r in exch_rxns} + if len(signs) > 1: + warnings.warn( + "Some exchange reactions differ in direction, and therefore have opposite " + 'meanings of lb and ub; forcing close_others=False.', + stacklevel=2, + ) + close_others = False + direction = None + else: + direction = "forward" if True in signs else "backward" + + default_lb, default_ub = cobra.Configuration().bounds + if lb is None: + lb = default_lb + if ub is None: + ub = default_ub + + if mets is None: + if isinstance(lb, Sequence) or isinstance(ub, Sequence): + raise ValueError( + "Only one upper and one lower bound may be provided if metabolites " + "are not specified." + ) + targets = exchanged + lbs = _broadcast(lb, len(targets)) + ubs = _broadcast(ub, len(targets)) + unused: list[Metabolite] = [] + else: + tokens = [mets] if isinstance(mets, (str, Metabolite)) else list(mets) + lbs_in = _broadcast(lb, len(tokens)) + ubs_in = _broadcast(ub, len(tokens)) + targets = [] + lbs, ubs, unused = [], [], [] + for token, lo, hi in zip(tokens, lbs_in, ubs_in, strict=True): + met = _resolve_met(model, exchanged, token) + if met is None: + unused.append(token) + continue + targets.append(met) + lbs.append(lo) + ubs.append(hi) + + if not targets: + return unused + + met_to_rxns: dict[Metabolite, list[Reaction]] = {} + for r in exch_rxns: + met_to_rxns.setdefault(next(iter(r.metabolites)), []).append(r) + repeated = [m for m, rs in met_to_rxns.items() if len(rs) > 1] + if repeated: + warnings.warn( + "The following metabolites are involved in more than one exchange " + f"reaction: {', '.join(m.name for m in repeated[:10])}" + + (f", and {len(repeated) - 10} more." if len(repeated) > 10 else ""), + stacklevel=2, + ) + + touched: set[Reaction] = set() + for met, lo, hi in zip(targets, lbs, ubs, strict=True): + for rxn in met_to_rxns[met]: + rxn.lower_bound, rxn.upper_bound = lo, hi + touched.add(rxn) + + if close_others: + for rxn in exch_rxns: + if rxn in touched: + continue + if direction == "backward": + rxn.lower_bound = 0 + else: + rxn.upper_bound = 0 + + return unused diff --git a/src/raven_toolbox/manipulation/simplify.py b/src/raven_toolbox/manipulation/simplify.py index 1ac3b6dc..57a73370 100644 --- a/src/raven_toolbox/manipulation/simplify.py +++ b/src/raven_toolbox/manipulation/simplify.py @@ -26,7 +26,9 @@ """ from __future__ import annotations +import ast import math +import re from collections.abc import Iterable import cobra @@ -34,6 +36,8 @@ from raven_toolbox.manipulation.irreversible import convert_to_irreversible +_EXP_SUFFIX = re.compile(r"_EXP_\d+$") + def _prune_orphan_metabolites(model: cobra.Model) -> list[str]: orphans = [m for m in model.metabolites if not m.reactions] @@ -182,6 +186,17 @@ def remove_duplicate_reactions( Reactions are duplicates when they have identical stoichiometry, bounds, and objective coefficient. One of each set is kept (reserved reactions are never removed). Returns the removed reaction IDs. + + The survivor is the first-encountered reaction in ``model.reactions`` order + (matching RAVEN's ``contractModel``). Its ``gene_reaction_rule`` becomes the + union of every duplicate's top-level OR-clauses, deduplicated, rather than + just its own — an isozyme relationship recorded on a *different* duplicate + would otherwise be silently dropped, and ``contractModel`` merges this way. + If every reaction in the group shares an ``_EXP_`` suffix + (:func:`~raven_toolbox.manipulation.expand.expand_model`'s naming), the model + is assumed to have gone through ``expand_model``, and the suffix is stripped + from the survivor's id — so an expand-then-remove-duplicates round trip + returns the original reaction id, not an arbitrarily-numbered expansion copy. """ reserved = set(reserved or []) groups: dict = {} @@ -192,14 +207,55 @@ def remove_duplicate_reactions( for rxns in groups.values(): if len(rxns) <= 1: continue - keep = rxns[-1] + keep = rxns[0] to_remove = [r for r in rxns if r is not keep and r.id not in reserved] - if to_remove: - removed += [r.id for r in to_remove] - model.remove_reactions(to_remove) + if not to_remove: + continue + + _merge_gene_reaction_rules(keep, rxns) + if all(_EXP_SUFFIX.search(r.id) for r in rxns): + keep.id = _EXP_SUFFIX.sub("", keep.id) + + removed += [r.id for r in to_remove] + model.remove_reactions(to_remove) return removed +def _top_level_or_clauses(gpr: cobra.core.gene.GPR) -> list[str]: + """The top-level OR-clauses of a GPR, as strings, via cobra's own AST. + + A clause containing "and" is parenthesised for readability when rejoined + with " or " — not required for correctness (Python's own operator + precedence already makes "A and B or C" parse as "(A and B) or C"), but + matches how a human, and RAVEN's own ``contractModel``, would write it. + """ + body = gpr.body + if body is None: + return [] + is_or = isinstance(body, ast.BoolOp) and isinstance(body.op, ast.Or) + clauses = body.values if is_or else [body] + out = [] + for clause in clauses: + text = ast.unparse(clause) + if isinstance(clause, ast.BoolOp) and isinstance(clause.op, ast.And): + text = f"({text})" + out.append(text) + return out + + +def _merge_gene_reaction_rules(keep: cobra.Reaction, group: list[cobra.Reaction]) -> None: + """Set ``keep``'s GPR to the union of every reaction in ``group``'s OR-clauses.""" + clauses: list[str] = [] + seen: set[str] = set() + for rxn in group: + for clause in _top_level_or_clauses(rxn.gpr): + if clause not in seen: + seen.add(clause) + clauses.append(clause) + if clauses: + keep.gene_reaction_rule = clauses[0] if len(clauses) == 1 else " or ".join(clauses) + + def constrain_reversible_reactions( model: cobra.Model, *, eps: float = 1e-10 ) -> list[str]: diff --git a/src/raven_toolbox/reconstruction/homology/blast.py b/src/raven_toolbox/reconstruction/homology/blast.py index e7432c01..c59a708f 100644 --- a/src/raven_toolbox/reconstruction/homology/blast.py +++ b/src/raven_toolbox/reconstruction/homology/blast.py @@ -53,7 +53,7 @@ def run_blast( model_ids: Sequence[str], ref_fastas: Sequence[str | Path], *, - evalue: float = 1e-5, + evalue: float = 1e-4, threads: int = 1, blastp: str | Path | None = None, makeblastdb: str | Path | None = None, @@ -62,6 +62,10 @@ def run_blast( Returns the hits DataFrame (filtered at ``evalue``). Requires BLAST+ (`blastp`, `makeblastdb`). + + The default matches MATLAB RAVEN's ``getBlast``, which hardcodes + ``-evalue 10e-5`` -- that is 1e-4, not 1e-5. Unlike RAVEN's, this one is an + argument, so a stricter search stays available. """ model_ids = list(model_ids) ref_fastas = _as_list(ref_fastas) @@ -97,7 +101,7 @@ def run_diamond( model_ids: Sequence[str], ref_fastas: Sequence[str | Path], *, - evalue: float = 1e-5, + evalue: float = 1e-3, threads: int = 1, sensitivity: str = "--more-sensitive", diamond: str | Path | None = None, @@ -105,6 +109,13 @@ def run_diamond( """Bidirectional DIAMOND between an organism and template organisms. Returns the hits DataFrame. Requires DIAMOND. + + The default matches MATLAB RAVEN's ``getDiamond``, which passes no + ``--evalue`` at all and so inherits DIAMOND's own default of 1e-3. Stated + explicitly here rather than left implicit, since the two aligners disagree: + ``run_blast`` uses 1e-4, following ``getBlast``. Neither cutoff has been + calibrated against data on either side -- unlike the KO-assignment cutoffs, + where measurement did justify diverging from RAVEN. """ model_ids = list(model_ids) ref_fastas = _as_list(ref_fastas) diff --git a/src/raven_toolbox/reconstruction/homology/homology.py b/src/raven_toolbox/reconstruction/homology/homology.py index ad1d017e..fd16189f 100644 --- a/src/raven_toolbox/reconstruction/homology/homology.py +++ b/src/raven_toolbox/reconstruction/homology/homology.py @@ -33,10 +33,16 @@ class HomologyResult: The draft ``cobra.Model``. gene_map ``{model_id: {template_gene: [new_gene, ...]}}`` ortholog mapping used. + candidates + Reactions that just missed the identity threshold, with the best hit + supporting each, sorted strongest first. ``None`` unless + ``review_identity`` was given. These are *not* in ``model``: they are the + near misses, for a curator to accept or reject deliberately. """ model: cobra.Model gene_map: dict = field(default_factory=dict) + candidates: pd.DataFrame | None = None class _Unmapped: @@ -202,13 +208,35 @@ def get_model_from_homology( complex_policy: str = "flag", only_genes_in_models: bool = False, max_evalue: float = 1e-30, - min_align_len: int = 200, + min_align_len: int = 100, min_identity: float = 40, + review_identity: float | None = None, strictness: int | None = None, ) -> HomologyResult: """Build a draft model for ``model_for`` by transferring reactions from templates. ``strictness`` (1/2/3) is a legacy alias for ``bidirectional`` / ``best_hits_only``. + + Defaults for the three filters come from + :doc:`a calibration against KEGG orthology `, + scored with precision weighted above recall (a wrongly transferred reaction + is harder to undo than a missing one): + + * ``min_identity`` 40 is the binding filter and the measured optimum. + * ``min_align_len`` 100 replaces RAVEN's 200, which discarded real orthologs + for no gain in precision. Anything at or below 150 measured the same; 200 + cost 1-2 points on every organism tested. + * ``max_evalue`` makes no difference anywhere between 1e-4 and 1e-50 -- the + other two filters have already excluded whatever it would exclude -- so + 1e-30 is kept for continuity with RAVEN rather than for any measured effect. + + ``review_identity`` (e.g. 25) collects what those thresholds turn away. Any + reaction that *would* have transferred at the looser identity is reported in + ``HomologyResult.candidates`` with its strongest supporting hit, and is not + added to the model. The two error types are not symmetric -- a missing + reaction can be gap-filled, a wrong one is hard to find and harder to remove + -- so the filters stay strict, but the evidence for the near misses is handed + to the curator instead of being discarded silently. """ if isinstance(models, cobra.Model): models = [models] @@ -216,6 +244,11 @@ def get_model_from_homology( raise ValueError(f"complex_policy must be flag/keep/drop, got {complex_policy!r}") if map_direction not in ("new_to_old", "old_to_new"): raise ValueError(f"map_direction must be new_to_old/old_to_new, got {map_direction!r}") + if review_identity is not None and review_identity >= min_identity: + raise ValueError( + f"review_identity ({review_identity}) must be below min_identity " + f"({min_identity}); it exists to catch what min_identity rejects." + ) bidirectional, best_hits_only, complex_policy, map_direction = _strictness_to_params( strictness, bidirectional, best_hits_only, complex_policy, map_direction ) @@ -250,7 +283,36 @@ def get_model_from_homology( if preferred_order and len(models) > 1: ortho = _apply_preferred_order(ortho, order) - # Build a per-template model holding only the transferred reactions with rewritten GPRs. + draft = _transfer(model_by_id, order, ortho, model_for, model_ids, complex_policy) + + candidates = None + if review_identity is not None: + # Reactions that only a looser identity would have transferred: reported + # for review, never added. Rejecting a real ortholog and silently binning + # the evidence are different things, and only the first is intended. + loose_ortho = _ortholog_map( + hits, model_for, model_ids, bidirectional=bidirectional, + best_hits_only=best_hits_only, score=score, map_direction=map_direction, + model_genes=model_genes, max_evalue=max_evalue, + min_align_len=min_align_len, min_identity=review_identity, + ) + if preferred_order and len(models) > 1: + loose_ortho = _apply_preferred_order(loose_ortho, order) + loose = _transfer( + model_by_id, order, loose_ortho, model_for, model_ids, complex_policy + ) + extra = {r.id for r in loose.reactions} - {r.id for r in draft.reactions} + candidates = _candidate_evidence( + extra, model_by_id, order, loose_ortho, hits, + max_evalue=max_evalue, min_align_len=min_align_len, + min_identity=review_identity, + ) + + return HomologyResult(model=draft, gene_map=ortho, candidates=candidates) + + +def _transfer(model_by_id, order, ortho, model_for, model_ids, complex_policy) -> cobra.Model: + """Assemble the draft: per-template reactions whose GPRs survive rewriting.""" transferred = [] for mid in order: model = model_by_id.get(mid) @@ -272,16 +334,79 @@ def get_model_from_homology( if m.reactions: transferred.append(m) - if transferred: - draft = merge_models(transferred, match_by="name") - else: - draft = cobra.Model() + draft = merge_models(transferred, match_by="name") if transferred else cobra.Model() draft.id = model_for draft.name = "Generated by get_model_from_homology using " + ", ".join(model_ids) - # Drop OLD_ placeholder genes that ended up orphaned (none survive in OR branches by construction). - orphan_genes = [g for g in draft.genes if not g.reactions] - for g in orphan_genes: + # Drop OLD_ placeholder genes left orphaned (none survive in OR branches by construction). + for g in [g for g in draft.genes if not g.reactions]: draft.genes.remove(g) + return draft + + +def _candidate_evidence( + reaction_ids, model_by_id, order, ortho, hits, *, + max_evalue, min_align_len, min_identity, +) -> pd.DataFrame: + """One row per candidate reaction, carrying the hit that held it back. + + The reported identity is the *limiting* one. A bidirectional match has to + clear the threshold in both directions, so a pair whose forward hit is 44 % + and reverse hit 30 % is rejected on the 30 -- and reporting the 44 would + leave a curator wondering why a comfortable match was turned away. - return HomologyResult(model=draft, gene_map=ortho) + Sorted strongest first, so the most defensible candidates are read first: the + point is a list somebody will actually skim, not an exhaustive dump. + """ + usable = hits[ + (hits.evalue <= max_evalue) + & (hits.align_len >= min_align_len) + & (hits.identity >= min_identity) + ] + # Weakest hit per ordered (gene, gene) pair, then the weaker of the two + # directions: whichever value the threshold actually acted on. + weakest: dict[tuple[str, str], tuple] = {} + for row in usable.itertuples(): + key = (row.from_gene, row.to_gene) + current = weakest.get(key) + if current is None or row.identity < current[0]: + weakest[key] = (row.identity, row.align_len, row.evalue, row.bitscore) + + def limiting(template_gene: str, target_gene: str): + both = [ + weakest.get((template_gene, target_gene)), + weakest.get((target_gene, template_gene)), + ] + found = [x for x in both if x is not None] + return min(found, key=lambda x: x[0]) if found else None + + rows = [] + for rid in sorted(reaction_ids): + for mid in order: + model = model_by_id.get(mid) + if model is None or rid not in model.reactions: + continue + per_model = ortho.get(mid, {}) + support = [] + for gene in (g.id for g in model.reactions.get_by_id(rid).genes): + for target_gene in per_model.get(gene, ()): + evidence = limiting(gene, target_gene) + if evidence is not None: + support.append((gene, target_gene, *evidence)) + if not support: + continue + support.sort(key=lambda s: s[2], reverse=True) # by identity + gene, target_gene, identity, align_len, evalue, bitscore = support[0] + rows.append({ + "reaction": rid, "template_model": mid, "template_gene": gene, + "target_gene": target_gene, "identity": identity, + "align_len": align_len, "evalue": evalue, "bitscore": bitscore, + "n_support": len(support), + }) + break # first template in preferred order wins, as for the draft + + frame = pd.DataFrame(rows, columns=[ + "reaction", "template_model", "template_gene", "target_gene", + "identity", "align_len", "evalue", "bitscore", "n_support", + ]) + return frame.sort_values("identity", ascending=False).reset_index(drop=True) diff --git a/src/raven_toolbox/reconstruction/kegg/assemble.py b/src/raven_toolbox/reconstruction/kegg/assemble.py index a2b5eb9e..8a0025e1 100644 --- a/src/raven_toolbox/reconstruction/kegg/assemble.py +++ b/src/raven_toolbox/reconstruction/kegg/assemble.py @@ -35,11 +35,17 @@ def assemble_model_from_ko_genes( model_id: str | None = None, model_name: str | None = None, note: str | None = None, + prune_orthology: bool = False, ) -> tuple[cobra.Model, dict[str, list[str]]]: """Build a draft model from a ``{ko: [gene, ...]}`` assignment. Returns ``(model, gpr_map)`` where ``gpr_map`` is the kept reactions' gene lists, so callers can add gene annotations afterwards. + + ``prune_orthology`` restricts each gene-backed reaction's ``kegg.orthology`` + annotation to the KOs that actually contributed a gene, matching RAVEN + ``getKEGGModelForOrganism``'s HMM branch (the organism-annotation path keeps + the full reference KO list, so this is off by default). """ rxn_to_kos: dict[str, set[str]] = {} for ko, rid in zip(ko_reaction["ko"], ko_reaction["reaction"], strict=True): @@ -53,15 +59,19 @@ def assemble_model_from_ko_genes( } gpr_map: dict[str, list[str]] = {} + matched_kos: dict[str, set[str]] = {} spontaneous_kept: set[str] = set() for rxn in reference_model.reactions: rid = rxn.id # Quality filters first: dropped even if it would have genes. if any(not keep_flag and rid in flagged for keep_flag, flagged in drop_if.values()): continue - genes = sorted({g for ko in rxn_to_kos.get(rid, ()) for g in ko_to_genes.get(ko, ())}) + rxn_kos = rxn_to_kos.get(rid, ()) + genes = sorted({g for ko in rxn_kos for g in ko_to_genes.get(ko, ())}) if genes: gpr_map[rid] = genes + # KOs that actually contributed a gene, for optional annotation pruning. + matched_kos[rid] = {ko for ko in rxn_kos if ko_to_genes.get(ko)} elif rid in spontaneous and keep_spontaneous: spontaneous_kept.add(rid) @@ -76,6 +86,20 @@ def assemble_model_from_ko_genes( ) for rid, genes in gpr_map.items(): model.reactions.get_by_id(rid).gene_reaction_rule = " or ".join(genes) + if prune_orthology: + # Restrict each gene-backed reaction's kegg.orthology to the KOs that + # matched a gene, preserving the reference annotation's order. (The + # order-preserving intersection also avoids the index misalignment in + # RAVEN getKEGGModelForOrganism's rxnMiriams pruning.) + for rid, kos in matched_kos.items(): + annotation = model.reactions.get_by_id(rid).annotation + orthology = annotation.get("kegg.orthology") + if orthology is None: + continue + if isinstance(orthology, list): + annotation["kegg.orthology"] = [k for k in orthology if k in kos] + elif orthology not in kos: + annotation["kegg.orthology"] = [] if note is not None: for rid in keep: model.reactions.get_by_id(rid).notes["note"] = note diff --git a/src/raven_toolbox/reconstruction/kegg/query.py b/src/raven_toolbox/reconstruction/kegg/query.py index bbb3ec62..2de4e040 100644 --- a/src/raven_toolbox/reconstruction/kegg/query.py +++ b/src/raven_toolbox/reconstruction/kegg/query.py @@ -31,7 +31,7 @@ from raven_toolbox.reconstruction.kegg.assemble import assemble_model_from_ko_genes from raven_toolbox.reconstruction.kegg.parse import _resolve_artefact, read_kegg_table -_NOTE = "Included by get_kegg_model_from_sequences (using HMMs)" +_NOTE = "Included by KEGG HMM reconstruction" _MIN_EVALUE = 1e-250 # floor for a reported E-value of 0, to keep logs finite @@ -198,6 +198,11 @@ def get_kegg_model_from_sequences( (:func:`assign_kos`), and assembles the model against ``reference_model`` / ``ko_reaction``. Genes are the query proteome's identifiers. """ + if model_id is None: + # RAVEN always sets model.id (to organismID); default it here so the + # draft never inherits the reference model's id. Pass model_id explicitly + # for byte-identical parity with a specific MATLAB organismID. + model_id = Path(fasta).stem hits = parse_hmmsearch_tblout(run_hmmsearch(fasta, library, threads=threads, hmmsearch=hmmsearch)) ko_to_genes = assign_kos( hits, @@ -216,6 +221,7 @@ def get_kegg_model_from_sequences( keep_general=keep_general, model_id=model_id, note=_NOTE, + prune_orthology=True, ) return model diff --git a/src/raven_toolbox/tasks/check.py b/src/raven_toolbox/tasks/check.py index 3800bace..da2f3272 100644 --- a/src/raven_toolbox/tasks/check.py +++ b/src/raven_toolbox/tasks/check.py @@ -8,8 +8,12 @@ Inputs/outputs are encoded as ranges on the per-metabolite mass-balance constraint (``model.constraints[met.id]``): an input allows net consumption (``Sv ∈ [-UB, -LB]``) and an output allows / requires net production (``Sv ≤ UB``, and ``≥ LB`` if -``LB > 0``). Existing boundary reactions are closed first, so inputs/outputs are -defined solely by the task (closed-model semantics). +``LB > 0``). By default, existing boundary reactions are left exactly as they are: a +task's inputs/outputs *add to* whatever the model's own open exchanges already allow, +rather than replacing them (RAVEN's ``checkTasks`` semantics, and what the published +task lists this module reads were curated against). Pass ``close_boundaries=True`` for +the stricter reading, where a task's declared inputs/outputs are the complete boundary +of the system for that check. """ from __future__ import annotations @@ -200,13 +204,16 @@ def check_tasks( model: cobra.Model, tasks: str | Iterable[Task], *, - close_boundaries: bool = True, + close_boundaries: bool = False, ) -> list[TaskResult]: """Run a task list against ``model`` and return a :class:`TaskResult` per task. - ``tasks`` is a parsed list of :class:`Task` or a path to a task-list file. With - ``close_boundaries`` (default), existing exchange/sink/demand reactions are - closed so inputs/outputs are defined purely by the tasks (as RAVEN assumes). + ``tasks`` is a parsed list of :class:`Task` or a path to a task-list file. By + default (``close_boundaries=False``), existing exchange/sink/demand reactions are + left open, so a task's inputs/outputs add to whatever the model's own boundary + already allows — matching RAVEN's ``checkTasks``, which published task lists were + curated against. Pass ``close_boundaries=True`` for the stricter reading, where + inputs/outputs are defined purely by the tasks. """ tasks = _as_tasks(tasks) base, name_to_id, comp_to_ids = _prepare_base(model, close_boundaries) @@ -222,10 +229,11 @@ def _as_tasks(tasks: str | Iterable[Task]) -> list[Task]: def _prepare_base(model: cobra.Model, close_boundaries: bool): base = model.copy() if close_boundaries: - # RAVEN closeModel: close reactions whose |coeff| sum to 1 (unit exchanges), not - # just cobra .boundary (any single-metabolite reaction). Task I/O is then defined - # solely by the task constraints, and the same exchange bypasses are open as in - # RAVEN — required for the essential-reaction discovery to match. + # Opt-in only: RAVEN's own checkTasks does not close boundary reactions before + # applying a task's constraints (confirmed directly — a task's inputs/outputs + # add to whatever the model's own exchanges already allow, not replace them), + # so close_model_in_place is skipped by default to match. Kept as an option for + # callers that want the stricter, task-is-the-complete-boundary reading. close_model_in_place(base) name_to_id, comp_to_ids = task_name_maps(base) return base, name_to_id, comp_to_ids @@ -325,7 +333,7 @@ def find_task_essential_reactions( model: cobra.Model, tasks: str | Iterable[Task], *, - close_boundaries: bool = True, + close_boundaries: bool = False, cache_path: str | Path | None = None, ) -> EssentialReactionsResult: """Find the reactions a model must use to satisfy a task list. @@ -336,7 +344,9 @@ def find_task_essential_reactions( test). This is the ``prepINITModel`` step that feeds (ft)INIT: essential reactions are kept regardless of expression score and made irreversible in their forced direction. When a reaction is essential in several tasks with conflicting directions, the majority - wins (ties → forward), matching RAVEN's ``pos < neg``. + wins (ties → forward), matching RAVEN's ``pos < neg``. ``close_boundaries`` defaults to + ``False``, matching RAVEN: a task's inputs/outputs add to whatever the model's own + open exchanges already allow, rather than replacing them. The solver is configured per task to match RAVEN's ``solveLP`` (``FeasibilityTol = 1e-9``, single-threaded with a fixed seed) so the degenerate min-flux vertex — and thus diff --git a/tests/data/parity/baselines/init_smallyeast.json b/tests/data/parity/baselines/init_smallyeast.json new file mode 100644 index 00000000..5a9768f2 --- /dev/null +++ b/tests/data/parity/baselines/init_smallyeast.json @@ -0,0 +1,28 @@ +{ + "source": "raven-toolbox", + "description": "Reactions kept by run_init on RAVEN's tutorial/smallYeast.yml with the scores from scripts/parity/record_baseline.py. Seeded from raven-toolbox, so this detects drift in this package rather than disagreement with MATLAB RAVEN.", + "fixture": "tutorial/smallYeast.yml", + "model_reactions": 53, + "kept_reactions": [ + "ACO", + "ADH1", + "CO2TRANS", + "ENO", + "FBA", + "FUM1", + "GLD", + "LSC1LSC2", + "PGI", + "PGK", + "RPE", + "TKI1TKI2a", + "TKI1TKI2b" + ], + "recorded": { + "date": "2026-08-23", + "raven_toolbox": "0.3.0", + "cobra": "0.30.0", + "solver": "gurobi_interface", + "python": "3.14.3" + } +} diff --git a/tests/data/parity/tiny.yml b/tests/data/parity/tiny.yml new file mode 100644 index 00000000..8a8abfca --- /dev/null +++ b/tests/data/parity/tiny.yml @@ -0,0 +1,90 @@ +--- +- metaData: !!omap + - id: parityTiny + - name: Parity fixture - a tiny balanced network + - description: >- + Authored for raven-toolbox's cross-language parity checks. Deliberately + small, fully specified (formulas and charges on every metabolite), and + free of any content taken from RAVEN, so the same file can be fed to both + implementations without licence entanglement. + - version: '1' +- metabolites: + - !!omap + - id: glc_c + - name: glucose + - compartment: c + - formula: C6H12O6 + - charge: 0 + - !!omap + - id: g6p_c + - name: glucose 6-phosphate + - compartment: c + - formula: C6H11O9P + - charge: -2 + - !!omap + - id: atp_c + - name: ATP + - compartment: c + - formula: C10H12N5O13P3 + - charge: -4 + - !!omap + - id: adp_c + - name: ADP + - compartment: c + - formula: C10H12N5O10P2 + - charge: -3 + - !!omap + - id: h_c + - name: H+ + - compartment: c + - formula: H + - charge: 1 + - !!omap + - id: unknown_c + - name: unspecified cofactor + - compartment: c +- reactions: + - !!omap + - id: HEX1 + - name: hexokinase + - metabolites: !!omap + - adp_c: 1 + - atp_c: -1 + - g6p_c: 1 + - glc_c: -1 + - h_c: 1 + - lower_bound: 0 + - upper_bound: 1000 + - gene_reaction_rule: (G1 and G2) or G3 + - !!omap + - id: IMBAL + - name: deliberately unbalanced reaction + - metabolites: !!omap + - g6p_c: -1 + - glc_c: 1 + - lower_bound: 0 + - upper_bound: 1000 + - !!omap + - id: UNKN + - name: reaction with an unspecified metabolite + - metabolites: !!omap + - g6p_c: -1 + - unknown_c: 1 + - lower_bound: 0 + - upper_bound: 1000 + - !!omap + - id: EX_glc + - name: glucose exchange + - metabolites: !!omap + - glc_c: -1 + - lower_bound: -1000 + - upper_bound: 1000 +- genes: + - !!omap + - id: G1 + - !!omap + - id: G2 + - !!omap + - id: G3 +- compartments: !!omap + - c: cytoplasm diff --git a/tests/parity/README.md b/tests/parity/README.md new file mode 100644 index 00000000..3dcfbdcd --- /dev/null +++ b/tests/parity/README.md @@ -0,0 +1,130 @@ +# Cross-language parity checks + +These tests answer one question: **does raven-toolbox still agree with MATLAB +RAVEN?** Until now that agreement was reported in study documents and nothing +would fail if it stopped being true. + +Run them with: + +```bash +pytest -m parity +``` + +They are marked so the default suite stays fast and self-contained; nothing here +is required for raven-toolbox to be correct on its own terms. + +## What "agree" means + +"Identical output" is achievable for some functions and meaningless for others, +so the checks are written in three tiers. Putting a function in the wrong tier +produces either a flaky test or a vacuous one. + +### Tier 1 — exact + +Deterministic transformations of a model or a file: I/O round-trips, task-list +parsing, gene-association normalisation, elemental balance, identifier sorting, +model merging, reversibility splitting, KEGG table parsing. Values must match. +A disagreement means one implementation is wrong. + +Note that *exact* means semantically exact, not byte-identical. RAVEN and +raven-toolbox both write valid YAML but differ in key order and quoting style, +which carries no meaning; comparing bytes would test the serialiser's habits +rather than the model. + +### Tier 2 — set-level + +Mixed-integer results: INIT/ftINIT extraction, gap-filling, compartment +assignment. These problems have many optima of equal objective value, so a +different answer is not a wrong answer. What can be checked is *drift*: today's +result against the result that was last inspected and accepted. + +`test_set_level.py` does that against a baseline recorded by +`scripts/parity/record_baseline.py`. When a change is expected to move the +extraction, read the diff the failure prints, re-record, and say in the pull +request why it moved: + +```bash +python scripts/parity/record_baseline.py # uses $RAVEN_ROOT +``` + +That baseline asserts **exact** set equality rather than an overlap band, +because it was measured rather than assumed: on this fixture GLPK and Gurobi +return the same 13 reactions, and each is identical across repeated runs. A +difference therefore means this package changed, not that the solver picked +another optimum. On a fixture where the solvers genuinely disagree, the honest +form is a band with a measured floor — not a loosened threshold on this one. + +The baseline is seeded from raven-toolbox itself, which makes it a regression +guard rather than a cross-language check. Its `source` field records that, and +the test prints it on failure so the two are not confused. Extending +`generate_oracles.m` with an extraction oracle turns the same comparison into a +real parity check. + +### Tier 3 — statistical + +Flux and random sampling. Two runs of the *same* implementation differ. Compare +distributions at a fixed seed, never individual samples. + +## Where the reference values come from + +Two sources, with different trade-offs: + +1. **RAVEN's own artefacts** (`test_yaml_interop.py`). RAVEN's repository + contains models written by MATLAB RAVEN itself. Reading them with + raven-toolbox is a genuine cross-language check that needs no MATLAB + installation — but it does need a RAVEN checkout, so these tests skip unless + `RAVEN_ROOT` points at one: + + ```bash + git clone --depth 1 -b develop3 https://github.com/SysBioChalmers/RAVEN + RAVEN_ROOT=$PWD/RAVEN pytest -m parity + ``` + + RAVEN is GPL and raven-toolbox is MIT, so those files are **read where they + are** and never copied into this repository. + +2. **Recorded oracles** (`test_oracles.py`). For behaviour that cannot be + inferred from a file — what `checkTasks` reports, how `getElementalBalance` + grades a reaction — MATLAB has to be run once and its answers recorded. + `scripts/parity/generate_oracles.m` does that, reading the fixtures in + `tests/data/parity/` (authored here, so no GPL material is involved) and + writing JSON into `tests/data/parity/oracles/`. Those tests skip when the + oracle file is absent, so a checkout without them still passes. + + To regenerate, in MATLAB with RAVEN on the path: + + ```matlab + cd scripts/parity + generate_oracles + ``` + + Commit the resulting JSON, and note the RAVEN commit it came from in + `oracles/PROVENANCE.md`. + +## Determinism + +`test_determinism.py` is not cross-language: it checks that raven-toolbox gives +the *same* answer twice — including in a second process. Python randomises +string hashing per process, so a set iterated to build constraint rows gives a +stable answer within one run and a different one in the next; repeating the call +in-process cannot see that, so one test runs the computation under three +`PYTHONHASHSEED` values through `_hashseed_worker.py` and compares digests. +(That is what `scripts/determinism_probe.py` did by hand while the placement +determinism fixes were being made.) + +It lives here because it protects the same property the parity tiers do — several recent fixes made compartment placement and gap-filling +deterministic, and nothing would have caught a regression. + +## What is enforced today + +- Every RAVEN-authored model loads and round-trips without losing RAVEN's own + fields (tier 1). +- The small-model extraction has not drifted from its recorded baseline + (tier 2). +- The deterministic paths return the same answer twice. + +Still only *reported*, not enforced: the genome-scale Human-GEM, yeast and +multi-organism comparisons in `docs/studies/`. They need Gurobi and models too +large for a free runner, so closing that gap needs a nightly job on a licensed +runner — the next piece of this harness. Tier 3 has a stated contract and no +tests yet for the same reason. diff --git a/tests/parity/_hashseed_worker.py b/tests/parity/_hashseed_worker.py new file mode 100644 index 00000000..389a89f8 --- /dev/null +++ b/tests/parity/_hashseed_worker.py @@ -0,0 +1,85 @@ +"""Run one gap-fill and print a digest of the result. Not a test module. + +Invoked as a subprocess by ``test_determinism.py`` under different +``PYTHONHASHSEED`` values. Python randomises string hashing per process, so a +set iterated to build constraint rows, or a dict deciding a tie-break, produces +a stable answer *within* one process and a different one in the next. An +in-process repetition cannot see that; two processes with different seeds can. + +Prints one JSON object on the last line so the caller can parse it out of +whatever the solver has written to stdout. +""" +from __future__ import annotations + +import hashlib +import json +import sys + + +def build_models(): + import cobra + + def model_with(model_id, reactions, objective=None): + model = cobra.Model(model_id) + mets = { + mid: cobra.Metabolite(mid, compartment="c") + for mid in ("a", "b", "c", "d") + } + model.add_metabolites(list(mets.values())) + for rid, stoich, lb, ub in reactions: + rxn = cobra.Reaction(rid, lower_bound=lb, upper_bound=ub) + rxn.add_metabolites({mets[m]: v for m, v in stoich.items()}) + model.add_reactions([rxn]) + if objective: + model.objective = objective + return model + + draft = model_with( + "draft", + [ + ("EX_a", {"a": 1}, -10.0, 1000.0), + ("R1", {"a": -1, "b": 1}, 0.0, 1000.0), + ("EX_d", {"d": -1}, 0.0, 1000.0), + ], + objective="EX_d", + ) + # Two symmetric routes plus a two-step alternative: nothing in the problem + # distinguishes them, so any hash-order leak shows up as a different pick. + template = model_with( + "template", + [ + ("T_bd_1", {"b": -1, "d": 1}, 0.0, 1000.0), + ("T_bd_2", {"b": -1, "d": 1}, 0.0, 1000.0), + ("T_bc", {"b": -1, "c": 1}, 0.0, 1000.0), + ("T_cd", {"c": -1, "d": 1}, 0.0, 1000.0), + ], + ) + return draft, template + + +def main() -> int: + import cobra + + from raven_toolbox.gapfilling import connect_blocked_reactions + + # Single-process: the gap-filler runs FVA, which otherwise starts a worker + # pool per invocation. On a four-reaction model that costs far more than it + # saves, and three of these workers running pools at once makes the test + # slower than everything else in the suite combined. + cobra.Configuration().processes = 1 + + draft, template = build_models() + result = connect_blocked_reactions(draft, [template]) + payload = { + "added": sorted(result.added_reactions), + "connected": sorted(result.newly_connected), + } + payload["digest"] = hashlib.sha256( + json.dumps(payload, sort_keys=True).encode("utf-8") + ).hexdigest()[:16] + print("HASHSEED_RESULT " + json.dumps(payload)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/parity/conftest.py b/tests/parity/conftest.py new file mode 100644 index 00000000..4f925073 --- /dev/null +++ b/tests/parity/conftest.py @@ -0,0 +1,76 @@ +"""Fixtures for the cross-language parity checks. + +Everything here skips rather than fails when its reference material is absent: +a checkout without a RAVEN clone, or without regenerated oracles, still has a +green suite. What must never happen is a parity test that silently passes +because it compared nothing -- so each fixture skips with a message naming +exactly what is missing and how to supply it. +""" +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest + +ORACLE_DIR = Path(__file__).resolve().parent.parent / "data" / "parity" / "oracles" +FIXTURE_DIR = Path(__file__).resolve().parent.parent / "data" / "parity" + + +@pytest.fixture(scope="session") +def raven_root() -> Path: + """A MATLAB RAVEN checkout, from ``$RAVEN_ROOT``. + + RAVEN is GPL and this package is MIT, so its files are read in place and + never vendored here. + """ + root = os.environ.get("RAVEN_ROOT") + if not root: + pytest.skip( + "RAVEN_ROOT is not set. Clone RAVEN and point at it to run the " + "cross-language checks: git clone --depth 1 -b develop3 " + "https://github.com/SysBioChalmers/RAVEN && RAVEN_ROOT=$PWD/RAVEN " + "pytest -m parity" + ) + path = Path(root) + if not (path / "io").is_dir(): + pytest.skip(f"RAVEN_ROOT={path} does not look like a RAVEN checkout") + return path + + +@pytest.fixture(scope="session") +def raven_models(raven_root: Path) -> dict[str, Path]: + """RAVEN-authored YAML models, by file name. + + These were written by MATLAB RAVEN's own ``writeYAMLmodel``, which makes + them reference data for the Python reader without needing MATLAB. + """ + tutorial = raven_root / "tutorial" + found = {p.name: p for p in sorted(tutorial.glob("*.yml"))} + if not found: + pytest.skip(f"no YAML models under {tutorial}") + return found + + +@pytest.fixture +def oracle(): + """Load a recorded MATLAB answer, or skip if it has not been generated.""" + + def _load(name: str): + path = ORACLE_DIR / f"{name}.json" + if not path.is_file(): + pytest.skip( + f"oracle '{name}' has not been generated. Run " + f"scripts/parity/generate_oracles.m in MATLAB with RAVEN on the " + f"path, then commit tests/data/parity/oracles/{name}.json" + ) + return json.loads(path.read_text(encoding="utf-8")) + + return _load + + +@pytest.fixture(scope="session") +def parity_fixture_dir() -> Path: + """Models authored in this repository, used as input on both sides.""" + return FIXTURE_DIR diff --git a/tests/parity/test_determinism.py b/tests/parity/test_determinism.py new file mode 100644 index 00000000..a64dbf03 --- /dev/null +++ b/tests/parity/test_determinism.py @@ -0,0 +1,178 @@ +"""The same input must give the same answer twice. + +Not a cross-language check, but it protects the property the parity tiers rest +on: a comparison against MATLAB means nothing if raven-toolbox itself returns a +different model each run. + +Compartment placement and gap-filling were made deterministic in #76, #83 and +the assignment gap-fill rework, all without a regression test -- the property +was verified once, by hand, and nothing has guarded it since. Tie-breaking in a +MILP is easy to reintroduce accidentally (an unordered set, a dict iteration +order, an unseeded solver), and the symptom is a benchmark that quietly moves. +""" +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import cobra +import pytest + +from raven_toolbox.gapfilling import connect_blocked_reactions +from raven_toolbox.manipulation import remove_duplicate_reactions + +pytestmark = pytest.mark.parity + +RUNS = 3 + + +def _toy_model() -> cobra.Model: + """A small network with a blocked branch and a symmetric fill choice. + + B is produced but not consumed unless one of two equally-good reactions is + added, so any tie-breaking that is not deterministic shows up here. + """ + model = cobra.Model("toy") + mets = { + mid: cobra.Metabolite(mid, compartment="c") for mid in ("a", "b", "c", "d") + } + model.add_metabolites(list(mets.values())) + + def rxn(rid, stoich, lb=0.0, ub=1000.0): + r = cobra.Reaction(rid, lower_bound=lb, upper_bound=ub) + r.add_metabolites({mets[m]: v for m, v in stoich.items()}) + return r + + model.add_reactions( + [ + rxn("EX_a", {"a": 1}, lb=-10.0), + rxn("R1", {"a": -1, "b": 1}), + rxn("EX_d", {"d": -1}), + ] + ) + model.objective = "EX_d" + return model + + +def _template() -> cobra.Model: + """Two symmetric routes from b to d, plus a decoy.""" + template = cobra.Model("template") + mets = { + mid: cobra.Metabolite(mid, compartment="c") for mid in ("a", "b", "c", "d") + } + template.add_metabolites(list(mets.values())) + + def rxn(rid, stoich): + r = cobra.Reaction(rid, lower_bound=0.0, upper_bound=1000.0) + r.add_metabolites({mets[m]: v for m, v in stoich.items()}) + return r + + template.add_reactions( + [ + rxn("T_bd_1", {"b": -1, "d": 1}), + rxn("T_bd_2", {"b": -1, "d": 1}), + rxn("T_bc", {"b": -1, "c": 1}), + rxn("T_cd", {"c": -1, "d": 1}), + ] + ) + return template + + +def test_gap_fill_returns_the_same_reactions_every_run(): + results = [] + for _ in range(RUNS): + result = connect_blocked_reactions(_toy_model(), [_template()]) + results.append(tuple(sorted(result.added_reactions))) + + assert results[0], ( + "the fixture no longer requires any fill, so this test would pass " + "whatever the tie-breaking does -- fix the fixture, not the assertion" + ) + assert len(set(results)) == 1, ( + f"connect_blocked_reactions returned different reaction sets across " + f"{RUNS} runs: {set(results)}" + ) + + +def test_duplicate_removal_keeps_the_same_reaction_every_run(): + """Which duplicate survives must not depend on iteration order.""" + kept = [] + for _ in range(RUNS): + model = _toy_model() + for suffix in ("dup1", "dup2"): + clone = cobra.Reaction(f"R1_{suffix}", lower_bound=0.0, upper_bound=1000.0) + clone.add_metabolites( + {m: c for m, c in model.reactions.R1.metabolites.items()} + ) + model.add_reactions([clone]) + remove_duplicate_reactions(model) + kept.append(tuple(sorted(r.id for r in model.reactions))) + + assert all("R1_dup1" not in run or "R1_dup2" not in run for run in kept), ( + "no duplicates were removed, so this test would pass whatever the " + "selection does -- fix the fixture, not the assertion" + ) + assert len(set(kept)) == 1, ( + f"remove_duplicate_reactions kept different reactions across {RUNS} " + f"runs: {set(kept)}" + ) + + +def test_result_does_not_depend_on_the_process_hash_seed(tmp_path): + """The same computation, in two processes with different string hashing. + + Python randomises string hashes per process. A set iterated to build + constraint rows, or a dict deciding a tie-break, gives a stable answer + *within* a process and a different one in the next -- so the repetitions + above cannot see it, however many times they run. Two processes with + different PYTHONHASHSEED values can. + + This is what scripts/determinism_probe.py checked by hand while the + placement determinism fixes were being made. Running it as a test means + nobody has to remember to. + """ + worker = Path(__file__).with_name("_hashseed_worker.py") + results = {} + + for seed in ("0", "1", "12345"): + env = {**os.environ, "PYTHONHASHSEED": seed} + # The package under test must win over any installed copy. + src = Path(__file__).resolve().parents[2] / "src" + env["PYTHONPATH"] = os.pathsep.join( + [str(src), env.get("PYTHONPATH", "")] + ).rstrip(os.pathsep) + + proc = subprocess.run( + [sys.executable, str(worker)], + capture_output=True, + text=True, + env=env, + timeout=600, + check=False, + ) + assert proc.returncode == 0, ( + f"worker failed under PYTHONHASHSEED={seed}:\n{proc.stderr[-2000:]}" + ) + marker = [ + line for line in proc.stdout.splitlines() if line.startswith("HASHSEED_RESULT ") + ] + assert marker, ( + f"worker printed no result under PYTHONHASHSEED={seed}; " + f"stdout tail:\n{proc.stdout[-1000:]}" + ) + results[seed] = json.loads(marker[-1].removeprefix("HASHSEED_RESULT ")) + + assert all(r["added"] for r in results.values()), ( + "the worker's fixture no longer requires any fill, so this test would " + "pass whatever the ordering does -- fix the worker, not the assertion" + ) + + digests = {seed: r["digest"] for seed, r in results.items()} + assert len(set(digests.values())) == 1, ( + "the result depends on Python's per-process hash seed, so something is " + "iterating a set or dict whose order is not stable across runs:\n" + + "\n".join(f" PYTHONHASHSEED={s}: {r['added']}" for s, r in results.items()) + ) diff --git a/tests/parity/test_genome_scale.py b/tests/parity/test_genome_scale.py new file mode 100644 index 00000000..435e7527 --- /dev/null +++ b/tests/parity/test_genome_scale.py @@ -0,0 +1,160 @@ +"""Tier 2 at genome scale: the extraction the studies report, re-run nightly. + +The Human-GEM, yeast and multi-organism validations in ``docs/studies/`` are the +strongest evidence raven-toolbox agrees with MATLAB RAVEN, and they are also the +least protected: they were measured once, by hand, on a model too large for a +free runner's solver licence. Nothing re-checks them. + +These tests close that gap on a licensed runner (see +``.github/workflows/parity-nightly.yml``). They need a genome-scale model, given +by ``$HUMAN_GEM_YML``, and a Gurobi licence that will accept it -- the +size-limited licence bundled with pip-installed gurobipy will not. + +Unlike the small fixture, this is where an overlap *band* is the honest +assertion rather than exact equality: at this size the MILP has many optima of +equal value, and a bounded solve (``time_limit``, ``mip_gap``) may legitimately +stop at a different one. The band floor is recorded in the baseline alongside +the run that produced it, so it can be tightened as evidence accumulates rather +than guessed now. +""" +from __future__ import annotations + +import json +import os +import time +from pathlib import Path + +import pytest + +from raven_toolbox.init import run_init +from raven_toolbox.io import read_yaml_model + +pytestmark = [pytest.mark.parity, pytest.mark.slow] + +BASELINE = ( + Path(__file__).resolve().parent.parent + / "data" + / "parity" + / "baselines" + / "init_genome_scale.json" +) + +# Bounded so a nightly job cannot run for hours. A solve that hits the limit is +# still useful -- it is compared as a band, not as an exact answer. +TIME_LIMIT_SECONDS = 1800.0 +MIP_GAP = 0.001 + + +@pytest.fixture(scope="module") +def genome_scale_model(): + path = os.environ.get("HUMAN_GEM_YML") + if not path: + pytest.skip( + "HUMAN_GEM_YML is not set. The nightly workflow downloads " + "Human-GEM.yml and points at it; to run locally, download " + "model/Human-GEM.yml from SysBioChalmers/Human-GEM and set the " + "variable." + ) + model_path = Path(path) + if not model_path.is_file(): + pytest.skip(f"HUMAN_GEM_YML={model_path} does not exist") + return read_yaml_model(model_path) + + +@pytest.fixture(scope="module") +def extraction(genome_scale_model): + """Run the extraction once and share it across the assertions below. + + The result is also written to disk. pytest captures stdout for a *passing* + test, so a print here is invisible in exactly the case that matters most: + the successful nightly run whose numbers are supposed to become the next + baseline. The report file survives capture and is uploaded as an artefact. + """ + model = genome_scale_model + scores = { + rxn.id: (10.0 if i % 3 == 0 else (-5.0 if i % 3 == 1 else 1.0)) + for i, rxn in enumerate(model.reactions) + } + started = time.perf_counter() + result = run_init( + model.copy(), scores, mip_gap=MIP_GAP, time_limit=TIME_LIMIT_SECONDS + ) + elapsed = time.perf_counter() - started + kept = sorted(r.id for r in result.model.reactions) + + summary = ( + f"kept {len(kept)} of {len(model.reactions)} reactions in " + f"{elapsed:.0f}s (mip_gap={MIP_GAP}, " + f"time_limit={TIME_LIMIT_SECONDS:.0f}s)" + ) + print(f"\ngenome-scale extraction: {summary}") + + out_dir = Path(os.environ.get("RAVEN_PARITY_REPORT_DIR", ".")) + out_dir.mkdir(parents=True, exist_ok=True) + (out_dir / "genome_scale_run.json").write_text( + json.dumps( + { + "kept_reactions": kept, + "kept": len(kept), + "total": len(model.reactions), + "seconds": round(elapsed, 1), + "mip_gap": MIP_GAP, + "time_limit": TIME_LIMIT_SECONDS, + }, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + + step_summary = os.environ.get("GITHUB_STEP_SUMMARY") + if step_summary: + with open(step_summary, "a", encoding="utf-8") as fh: + fh.write(f"### Genome-scale extraction\n\n{summary}\n") + + return {"kept": kept, "total": len(model.reactions), "seconds": elapsed} + + +def test_extraction_produces_a_plausible_model(extraction): + """A sanity floor that holds regardless of which optimum was found.""" + kept, total = extraction["kept"], extraction["total"] + + assert kept, "extraction returned an empty model" + assert len(kept) < total, ( + "extraction kept every reaction, so it did not discriminate at all" + ) + fraction = len(kept) / total + assert 0.02 < fraction < 0.98, ( + f"extraction kept {fraction:.1%} of the model, which is outside any " + f"plausible range for a context-specific extraction" + ) + assert len(set(kept)) == len(kept), "extraction returned duplicate reaction ids" + + +def test_extraction_overlaps_the_recorded_baseline(extraction): + """Compare as a band: alternate optima are legitimate at this size.""" + if not BASELINE.is_file(): + pytest.skip( + f"no genome-scale baseline at {BASELINE}. The first successful " + f"nightly run reports the numbers; record them (with the band floor " + f"the evidence supports) and commit the file." + ) + + baseline = json.loads(BASELINE.read_text(encoding="utf-8")) + floor = baseline["jaccard_floor"] + expected = set(baseline["kept_reactions"]) + got = set(extraction["kept"]) + + overlap = len(got & expected) / len(got | expected) + print(f"genome-scale Jaccard vs baseline: {overlap:.4f} (floor {floor})") + + assert overlap >= floor, ( + f"extraction overlap with the baseline fell to {overlap:.4f}, below the " + f"recorded floor of {floor} " + f"(baseline recorded {baseline['recorded']['date']} with " + f"{baseline['recorded']['solver']}).\n" + f" kept now: {len(got)}, baseline: {len(expected)}, " + f"shared: {len(got & expected)}\n" + f"A drop this size is not solver noise -- something in the extraction " + f"changed. If it is intended, re-record the baseline and say why." + ) diff --git a/tests/parity/test_oracles.py b/tests/parity/test_oracles.py new file mode 100644 index 00000000..d2159978 --- /dev/null +++ b/tests/parity/test_oracles.py @@ -0,0 +1,102 @@ +"""Tier 1: compare against answers recorded from MATLAB RAVEN. + +Some behaviour cannot be inferred from a file -- how RAVEN grades a reaction's +elemental balance, what it normalises a gene rule to. For those, MATLAB is run +once by ``scripts/parity/generate_oracles.m`` and its answers are committed as +JSON; these tests assert raven-toolbox agrees. + +Each test skips, with instructions, when its oracle has not been generated, so +a checkout without oracles still has a green suite. The trade-off is deliberate: +a skipped test is visible in the report, a fabricated oracle is not. +""" +from __future__ import annotations + +import pytest + +from raven_toolbox.io import read_yaml_model +from raven_toolbox.manipulation import gpr_to_dnf +from raven_toolbox.utils import get_elemental_balance + +pytestmark = pytest.mark.parity + + +@pytest.fixture +def tiny(parity_fixture_dir): + return read_yaml_model(parity_fixture_dir / "tiny.yml") + + +def test_model_structure_matches_matlab(tiny, oracle): + """The two readers see the same model in the same file.""" + expected = oracle("model_structure") + + assert sorted(r.id for r in tiny.reactions) == sorted(expected["rxns"]) + assert sorted(m.id for m in tiny.metabolites) == sorted(expected["mets"]) + assert sorted(g.id for g in tiny.genes) == sorted(expected["genes"]) + + bounds = {r.id: (r.lower_bound, r.upper_bound) for r in tiny.reactions} + for rid, lb, ub in zip( + expected["rxns"], expected["lb"], expected["ub"], strict=True + ): + assert bounds[rid] == pytest.approx((lb, ub)), f"{rid}: bounds differ" + + ours = { + (r.id, m.id): coeff + for r in tiny.reactions + for m, coeff in r.metabolites.items() + } + theirs = { + (e["rxn"], e["met"]): e["coefficient"] for e in expected["stoichiometry"] + } + assert ours.keys() == theirs.keys(), "different reaction/metabolite pairs" + for key, coeff in theirs.items(): + assert ours[key] == pytest.approx(coeff), f"{key}: coefficient differs" + + +def test_elemental_balance_matches_matlab(tiny, oracle): + """Balanced / unbalanced / undecidable must be graded the same way. + + RAVEN reports 1, 0 and -1; raven-toolbox reports the same three states by + name. The fixture contains one of each, including a reaction whose + metabolite has no formula -- the case plain cobrapy silently miscounts. + """ + expected = oracle("elemental_balance") + matlab = dict(zip(expected["rxns"], expected["balanceStatus"], strict=True)) + + to_status = {1: "balanced", 0: "unbalanced", -1: "unknown"} + report = get_elemental_balance(tiny) + ours = {entry.reaction_id: entry.status for entry in report} + + for rid, code in matlab.items(): + assert rid in ours, f"{rid} missing from the Python report" + assert ours[rid] == to_status[int(code)], ( + f"{rid}: MATLAB says {to_status[int(code)]}, Python says {ours[rid]}" + ) + + +def test_gpr_normalisation_matches_matlab(tiny, oracle): + """Gene rules normalise to the same disjunctive normal form.""" + expected = oracle("gpr_dnf") + matlab = { + rid: dnf + for rid, dnf in zip(expected["rxns"], expected["dnf"], strict=True) + if dnf + } + + for rid, their_dnf in matlab.items(): + ours = gpr_to_dnf(tiny.reactions.get_by_id(rid).gpr) + ours_clauses = {frozenset(clause) for clause in ours} + assert ours_clauses == _clauses(their_dnf), ( + f"{rid}: MATLAB normalised to {their_dnf!r}, Python to {ours!r}" + ) + + +def _clauses(rule: str) -> set[frozenset[str]]: + """A DNF rule as a set of AND-clauses, so ordering carries no weight.""" + if not rule: + return set() + stripped = rule.replace("(", " ").replace(")", " ") + return { + frozenset(part.strip() for part in clause.split(" and ") if part.strip()) + for clause in stripped.split(" or ") + if clause.strip() + } diff --git a/tests/parity/test_set_level.py b/tests/parity/test_set_level.py new file mode 100644 index 00000000..13a6e52d --- /dev/null +++ b/tests/parity/test_set_level.py @@ -0,0 +1,147 @@ +"""Tier 2: the extracted model must not drift unnoticed. + +Extraction is a MILP, so there is no single right answer to assert -- equally +good optima exist and a different set is not automatically a wrong set. What can +be asserted is that today's result matches the result that was last inspected +and accepted, recorded by ``scripts/parity/record_baseline.py``. + +The baseline is currently seeded from raven-toolbox itself, so this is a +regression guard rather than a cross-language check; the ``source`` field says +which, and the test reports it on failure so nobody mistakes one for the other. +When an extraction oracle is generated from MATLAB, the same comparison runs +against RAVEN's answer. + +Exact equality is asserted rather than an overlap band because it was measured: +on this fixture GLPK and Gurobi return the same 13 reactions, and each is +identical across repeated runs. A difference therefore means something in this +package changed, not that the solver picked another optimum. Should that stop +being true on a larger fixture, the right move is a band with a measured floor +-- not a loosened threshold here. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from raven_toolbox.init import run_init +from raven_toolbox.io import read_yaml_model + +pytestmark = pytest.mark.parity + +BASELINE = ( + Path(__file__).resolve().parent.parent + / "data" + / "parity" + / "baselines" + / "init_smallyeast.json" +) + + +def _scores(model) -> dict[str, float]: + """Must match scripts/parity/record_baseline.py, or the comparison is void.""" + return { + rxn.id: (10.0 if i % 3 == 0 else (-5.0 if i % 3 == 1 else 1.0)) + for i, rxn in enumerate(model.reactions) + } + + +@pytest.fixture(scope="module") +def baseline() -> dict: + if not BASELINE.is_file(): + pytest.skip( + f"no baseline at {BASELINE}. Record one with " + f"python scripts/parity/record_baseline.py" + ) + return json.loads(BASELINE.read_text(encoding="utf-8")) + + +def test_extraction_matches_the_recorded_baseline(raven_root, baseline): + fixture = raven_root.joinpath(*baseline["fixture"].split("/")) + if not fixture.is_file(): + pytest.skip(f"fixture {baseline['fixture']} not in this RAVEN checkout") + + model = read_yaml_model(fixture) + assert len(model.reactions) == baseline["model_reactions"], ( + "the fixture itself changed upstream, so the baseline describes a " + "different model -- re-record it rather than adjusting this test" + ) + + kept = sorted(r.id for r in run_init(model, _scores(model)).model.reactions) + expected = baseline["kept_reactions"] + + if kept == expected: + return + + got, want = set(kept), set(expected) + overlap = len(got & want) / len(got | want) + pytest.fail( + f"extraction drifted from the recorded baseline " + f"(source: {baseline['source']}, recorded {baseline['recorded']['date']} " + f"with {baseline['recorded']['solver']}).\n" + f" Jaccard: {overlap:.3f} ({len(got & want)} shared of {len(got | want)})\n" + f" added: {sorted(got - want)}\n" + f" dropped: {sorted(want - got)}\n" + f"If the change is intended, re-record with " + f"python scripts/parity/record_baseline.py and say in the PR why it moved." + ) + + +def test_baseline_describes_a_real_decision(baseline): + """A baseline that kept everything (or nothing) would assert nothing.""" + kept = len(baseline["kept_reactions"]) + total = baseline["model_reactions"] + assert 0 < kept < total, ( + f"the baseline keeps {kept} of {total} reactions, so the extraction is " + f"not choosing -- fix the fixture or the scores, not this assertion" + ) + + +def test_solvers_agree_on_the_extraction(raven_root, baseline): + """The extracted set must not depend on which solver found it. + + This is the measurement the exact-equality assertion above rests on, kept + as a test so it stays true. If a future model makes the solvers disagree, + this fails first and says so -- which is the signal to move that fixture to + an overlap band with a measured floor, rather than to loosen the assertion + above and lose the ability to detect real drift. + + Solvers that are not installed, or that refuse the model (the size-limited + licence bundled with pip-installed gurobipy will refuse a large one), are + skipped individually rather than failing the test. + """ + from cobra.util.solver import solvers as available + + fixture = raven_root.joinpath(*baseline["fixture"].split("/")) + if not fixture.is_file(): + pytest.skip(f"fixture {baseline['fixture']} not in this RAVEN checkout") + + candidates = [name for name in ("glpk", "gurobi", "cplex") if name in available] + results: dict[str, list[str]] = {} + refused: dict[str, str] = {} + + for name in candidates: + model = read_yaml_model(fixture) + try: + model.solver = name + results[name] = sorted( + r.id for r in run_init(model, _scores(model)).model.reactions + ) + except Exception as exc: # noqa: BLE001 - reported, not swallowed + refused[name] = f"{type(exc).__name__}: {exc}" + + if len(results) < 2: + pytest.skip( + f"need two usable solvers to compare; ran {sorted(results)}, " + f"refused {refused or 'none'}" + ) + + distinct = {tuple(v) for v in results.values()} + assert len(distinct) == 1, ( + "solvers disagree on the extracted set, so exact equality is no longer " + "the right assertion for this fixture:\n" + + "\n".join(f" {name}: {len(kept)} reactions" for name, kept in results.items()) + + f"\n symmetric difference: " + f"{sorted(set(results[candidates[0]]) ^ set(results[candidates[1]]))}" + ) diff --git a/tests/parity/test_yaml_interop.py b/tests/parity/test_yaml_interop.py new file mode 100644 index 00000000..a3b76ed4 --- /dev/null +++ b/tests/parity/test_yaml_interop.py @@ -0,0 +1,112 @@ +"""Tier 1: raven-toolbox must read what MATLAB RAVEN writes. + +RAVEN's repository ships models written by its own ``writeYAMLmodel``. Reading +them here is a real cross-language check that needs no MATLAB installation -- +and it is the check that would have caught raven-toolbox failing on any RAVEN +model without genes (``KeyError: 'genes'``), which no unit test covered because +every fixture in this repository happened to have genes. + +Exact here means semantically exact. Both implementations write valid YAML but +differ in key order and quoting; comparing bytes would test serialiser habits +rather than the model. +""" +from __future__ import annotations + +import pytest + +from raven_toolbox.io import read_yaml_model, write_yaml_model + +pytestmark = pytest.mark.parity + + +def test_reads_every_raven_authored_model(raven_models): + """Every YAML model RAVEN ships must load.""" + failures = {} + for name, path in raven_models.items(): + try: + read_yaml_model(path) + except Exception as exc: # noqa: BLE001 - reported per file below + failures[name] = f"{type(exc).__name__}: {exc}" + + assert not failures, "RAVEN-written models that raven-toolbox cannot read: " + repr( + failures + ) + + +def test_models_are_non_trivial(raven_models): + """Guard against the suite passing because it read nothing. + + A reader that silently produced empty models would satisfy the test above. + """ + sizes = {} + for name, path in raven_models.items(): + model = read_yaml_model(path) + sizes[name] = (len(model.reactions), len(model.metabolites)) + + assert any(r > 0 and m > 0 for r, m in sizes.values()), ( + f"no RAVEN model produced any content: {sizes}" + ) + + +def test_round_trip_is_stable(raven_models, tmp_path): + """Writing a RAVEN model and reading it back preserves the model. + + Not byte-for-byte against RAVEN's file -- key order and quoting differ + without meaning -- but the content a model *is* must survive intact. + """ + for name, path in raven_models.items(): + original = read_yaml_model(path) + out = tmp_path / name + write_yaml_model(original, out) + again = read_yaml_model(out) + + assert [r.id for r in again.reactions] == [ + r.id for r in original.reactions + ], f"{name}: reaction ids changed" + assert [m.id for m in again.metabolites] == [ + m.id for m in original.metabolites + ], f"{name}: metabolite ids changed" + assert [g.id for g in again.genes] == [ + g.id for g in original.genes + ], f"{name}: gene ids changed" + + for before, after in zip(original.reactions, again.reactions, strict=True): + assert after.bounds == before.bounds, f"{name}: {before.id} bounds changed" + assert ( + after.gene_reaction_rule == before.gene_reaction_rule + ), f"{name}: {before.id} GPR changed" + assert {m.id: c for m, c in after.metabolites.items()} == { + m.id: c for m, c in before.metabolites.items() + }, f"{name}: {before.id} stoichiometry changed" + + +def test_raven_specific_fields_survive(raven_models, tmp_path): + """RAVEN's own per-entry fields are not dropped on the way through. + + They have no cobra counterpart and live in ``notes``; losing them silently + would make raven-toolbox a lossy step in a MATLAB workflow. + """ + interesting = {"inchis", "deltaG", "metFrom", "rxnFrom", "confidence_score"} + + for name, path in raven_models.items(): + original = read_yaml_model(path) + present = { + key + for entity in (*original.metabolites, *original.reactions) + for key in (entity.notes or {}) + } & interesting + if not present: + continue + + out = tmp_path / name + write_yaml_model(original, out) + again = read_yaml_model(out) + + survived = { + key + for entity in (*again.metabolites, *again.reactions) + for key in (entity.notes or {}) + } & interesting + assert present <= survived, ( + f"{name}: RAVEN fields lost in round-trip: {sorted(present - survived)}" + ) diff --git a/tests/test_annotation.py b/tests/test_annotation.py index d61a6511..24cc2bf3 100644 --- a/tests/test_annotation.py +++ b/tests/test_annotation.py @@ -12,6 +12,7 @@ load_delta_g_csv, save_delta_g_csv, ) +from raven_toolbox.annotation.delta_g import DELTA_G_MISSING from raven_toolbox.annotation.sbo import _default_transport_detector # --- shared tiny model ------------------------------------------------- @@ -185,6 +186,43 @@ def test_load_skips_nan_rows(tmp_path): assert m.metabolites.get_by_id("glc_e").notes["deltaG"] == "1.0" +def test_load_skips_the_missing_sentinel(tmp_path): + """The ΔG side-car tables write 10000000 for "no valid ΔG", and yeast-GEM's side-car carries it on + 777 of its 4102 reaction rows. Stamping it would record a physically impossible 10^7 kJ/mol as a + measurement. yeast-GEM's own checkrxnDirection.m gates on the same value.""" + m = _toy_model() + m.metabolites.get_by_id("atp_c").notes["deltaG"] = "preserved" + + csv = tmp_path / "met_dg.csv" + pd.DataFrame({"Var1": ["atp_c", "glc_e"], "Var2": [10000000.0, 1.0]}).to_csv(csv, index=False) + + stamped = load_delta_g_csv(m.metabolites, csv, missing_value=DELTA_G_MISSING) + assert stamped == 1 # only the real value + assert m.metabolites.get_by_id("atp_c").notes["deltaG"] == "preserved" + assert m.metabolites.get_by_id("glc_e").notes["deltaG"] == "1.0" + + +def test_sentinel_skipping_can_be_opted_out_of(tmp_path): + m = _toy_model() + csv = tmp_path / "met_dg.csv" + pd.DataFrame({"Var1": ["atp_c"], "Var2": [10000000.0]}).to_csv(csv, index=False) + + assert load_delta_g_csv(m.metabolites, csv, missing_value=None) == 1 + assert m.metabolites.get_by_id("atp_c").notes["deltaG"] == "10000000.0" + + +def test_sentinel_recognised_whatever_dtype_the_csv_round_trip_produces(tmp_path): + """The same sentinel arrives as 10000000, 10000000.0 or "10000000.0" depending on the writer and + the column's inferred dtype -- a string column appears as soon as one row holds text.""" + m = _toy_model() + csv = tmp_path / "met_dg.csv" + pd.DataFrame({"Var1": ["atp_c", "glc_e"], "Var2": ["10000000.0", "-2.5"]}).to_csv(csv, index=False) + + assert load_delta_g_csv(m.metabolites, csv, missing_value=DELTA_G_MISSING) == 1 + assert "deltaG" not in m.metabolites.get_by_id("atp_c").notes + assert m.metabolites.get_by_id("glc_e").notes["deltaG"] == "-2.5" + + def test_custom_columns_and_note_key(tmp_path): m = _toy_model() m.metabolites.get_by_id("atp_c").notes["dG_kJ"] = "-30.5" diff --git a/tests/test_binaries.py b/tests/test_binaries.py index 12acca40..c1b5e0c6 100644 --- a/tests/test_binaries.py +++ b/tests/test_binaries.py @@ -1,5 +1,6 @@ """Tests for raven_toolbox.binaries (binary resolution + bundled-ZIP provisioning).""" import hashlib +import os import shutil import stat import zipfile @@ -9,6 +10,8 @@ from raven_toolbox import binaries +_WINDOWS = os.name == "nt" + def test_resolve_explicit_path(): assert binaries.resolve_binary("blastp", binary="/opt/x/blastp") == "/opt/x/blastp" @@ -261,3 +264,52 @@ def spy(url, *args, **kwargs): monkeypatch.setattr(binaries, "urlopen", spy) binaries.ensure_binary("footool", registry=registry) assert isinstance(seen["timeout"], (int, float)) and seen["timeout"] > 0 + + +def _two_tool_bundle(tmp_path): + """A bundle providing two executables, like BLAST's blastp + makeblastdb.""" + archive = tmp_path / "blast.zip" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("blastp", "#!/bin/sh\nexit 0\n") + zf.writestr("makeblastdb", "#!/bin/sh\nexit 0\n") + sha = hashlib.sha256(archive.read_bytes()).hexdigest() + return { + "blast": { + "version": "2.17.0", + "provides": ["blastp", "makeblastdb"], + "platforms": {binaries.platform_key(): {"url": archive.as_uri(), "sha256": sha}}, + } + } + + +@pytest.mark.skipif(_WINDOWS, reason="the execute bit is meaningless on Windows") +def test_every_executable_in_a_bundle_is_made_executable(tmp_path, monkeypatch): + """Not only the one that triggered the download. + + zipfile does not restore Unix permissions, so an extracted file is not + executable. Marking only the requested one left the rest of the bundle + unusable: fetching blastp then calling makeblastdb raised PermissionError, + which is what broke the parity nightly. + """ + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache")) + registry = _two_tool_bundle(tmp_path) + + fetched = binaries.ensure_binary("blastp", registry=registry) + + sibling = Path(fetched).parent / "makeblastdb" + assert sibling.stat().st_mode & stat.S_IXUSR, "the other tool in the bundle is not executable" + + +@pytest.mark.skipif(_WINDOWS, reason="the execute bit is meaningless on Windows") +def test_a_cached_bundle_is_repaired_rather_than_trusted(tmp_path, monkeypatch): + """A cache written by the older code has the bit missing and never re-extracts.""" + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache")) + registry = _two_tool_bundle(tmp_path) + + fetched = Path(binaries.ensure_binary("blastp", registry=registry)) + fetched.chmod(0o644) # simulate what the previous version left behind + + again = Path(binaries.ensure_binary("blastp", registry=registry)) + + assert again == fetched, "expected the cached copy, not a fresh download" + assert again.stat().st_mode & stat.S_IXUSR diff --git a/tests/test_comparison_diff.py b/tests/test_comparison_diff.py index 91ea5f76..e4e21134 100644 --- a/tests/test_comparison_diff.py +++ b/tests/test_comparison_diff.py @@ -5,7 +5,7 @@ import pytest from raven_toolbox.comparison import DiffReport, diff_models -from raven_toolbox.comparison.diff import _normalise_gpr +from raven_toolbox.comparison.diff import _canonical_gpr def _mini_model(model_id: str = "m") -> cobra.Model: @@ -98,16 +98,55 @@ def test_extra_annotations_picked_up(): assert not diff_models(a, b, extra_annotations={"custom-key"}).equal +def _canon(rule: str) -> str: + r = cobra.Reaction("r") + r.gene_reaction_rule = rule + return _canonical_gpr(r) + + +@pytest.mark.parametrize( + "ga, gb", + [ + ("A and B", "a AND b"), # case + AND spelling + ("A and B", "A and B"), # whitespace + ("(A or B) and C", "(a OR b) AND c"), # case, nested + ("A and B", "B and A"), # AND operand order (the back-port) + ("A or B", "B or A"), # OR isozyme order (the back-port) + ("(A or B) and C", "C and (B or A)"), # nested, reordered at both levels + ("A and A", "A"), # duplicate gene collapses + ], +) +def test_canonical_gpr_is_order_insensitive(ga, gb): + assert _canon(ga) == _canon(gb) + + @pytest.mark.parametrize( "ga, gb", [ - ("A and B", "a AND b"), - ("A and B", "A and B"), - ("(A or B) and C", "(a OR b) AND c"), + ("A and B", "A or B"), # AND vs OR is a real logic difference, not formatting + ("A and B", "A and C"), # a different gene really differs ], ) -def test_gpr_normalisation(ga, gb): - assert _normalise_gpr(ga) == _normalise_gpr(gb) +def test_canonical_gpr_keeps_real_differences(ga, gb): + assert _canon(ga) != _canon(gb) + + +def test_diff_models_ignores_gpr_operand_order(): + # Two models identical but for GPR operand order must compare equal — the RAVEN diffModels + # behaviour this back-port adds. The string heuristic used to flag this as a GPR difference. + a = _mini_model("a") # r1 GPR is "g1 AND g2" + b = _mini_model("b") + b.reactions.r1.gene_reaction_rule = "g2 and g1" + assert diff_models(a, b).equal + + +def test_diff_models_flags_real_gpr_difference(): + a = _mini_model("a") # r1 GPR is "g1 AND g2" + b = _mini_model("b") + b.reactions.r1.gene_reaction_rule = "g1 or g2" # AND -> OR is genuine + report = diff_models(a, b) + assert not report.equal + assert any("GPR" in d for d in report.differences) def test_max_per_category_truncates(): diff --git a/tests/test_confidence.py b/tests/test_confidence.py index 5d001c5e..fff58762 100644 --- a/tests/test_confidence.py +++ b/tests/test_confidence.py @@ -7,6 +7,7 @@ from raven_toolbox.confidence import ( ConfidenceEntry, ReactionConfidence, + annotate_confidence, confidence_report, equation_exempt, facet_summary, @@ -18,6 +19,7 @@ score_gene_association_confidence, score_localization_confidence, set_confidence, + thiele_palsson_score, ) from raven_toolbox.localization import AssignmentProposal from raven_toolbox.localization.scores import LocalizationScores @@ -447,3 +449,67 @@ def test_facet_summary_separates_scored_from_exempt(): assert eq["n"].sum() == 3 assert int(eq[eq["basis"] == "balanced"]["n"].iloc[0]) == 2 assert int(eq[eq["basis"] == "mass-imbalanced"]["n"].iloc[0]) == 1 # SPONT: H2O -> H + + +# --------------------------------------------------------------------------- annotate_confidence umbrella + +def _proposal_for(model_reaction_id="r1"): + return AssignmentProposal( + placements={model_reaction_id: ["c"]}, added_transports=[], added_reactions=[], + unplaced_reactions=[], min_growth=1.0, status="optimal", + ) + + +def test_annotate_confidence_runs_model_facets_and_skips_localization_without_inputs(): + m = _model() + counts = annotate_confidence(m) # no proposal/scores + # equation and gene_association need only the model; localization is skipped, not failed + assert set(counts) == {"equation", "gene_association"} + assert counts["equation"] == 1 and counts["gene_association"] == 1 # only r1 is non-boundary + assert "localization" not in get_confidence(m.reactions.r1).facets + + +def test_annotate_confidence_runs_localization_with_proposal_and_scores(): + m = _model() + scores = LocalizationScores(pd.DataFrame({"c": [0.8]}, index=["g1"])) + counts = annotate_confidence(m, proposal=_proposal_for("r1"), scores=scores) + assert set(counts) == {"localization", "equation", "gene_association"} + assert set(get_confidence(m.reactions.r1).facets) == {"localization", "equation", "gene_association"} + + +def test_annotate_confidence_respects_facets_arg(): + m = _model() + counts = annotate_confidence(m, facets=["equation"]) + assert set(counts) == {"equation"} + assert set(get_confidence(m.reactions.r1).facets) == {"equation"} + + +def test_sbo_warning_names_add_sbo_terms(): + m = _model() # carries no SBO terms + with pytest.warns(UserWarning, match=r"add_sbo_terms"): + score_equation_confidence(m) + + +# --------------------------------------------------------------------------- Thiele-Palsson mapping + +def test_thiele_palsson_score_maps_gene_association_basis(): + m = _model() + r = m.reactions.r1 + for basis, expected in (("gpr+literature", 3), ("gpr", 2), ("no-gpr", 1)): + set_confidence(r, "gene_association", ConfidenceEntry(0.6, basis=basis)) + assert thiele_palsson_score(r) == expected + # curated: the evidence class is the curator's to name, not inferred here -> None + mark_curated(r, facet="gene_association") + assert thiele_palsson_score(r) is None + # no gene_association facet at all (an exempt boundary reaction) -> None + assert thiele_palsson_score(m.reactions.EX_A) is None + + +def test_thiele_palsson_score_end_to_end(): + m = _model() + score_gene_association_confidence(m) + assert get_confidence(m.reactions.r1).facets["gene_association"].basis == "gpr" + assert thiele_palsson_score(m.reactions.r1) == 2 # sequence-only GPR + m.reactions.r1.annotation["pubmed"] = ["12345"] + score_gene_association_confidence(m) + assert thiele_palsson_score(m.reactions.r1) == 3 # GPR + literature diff --git a/tests/test_init_ftinit.py b/tests/test_init_ftinit.py index 2a58c436..c8f17196 100644 --- a/tests/test_init_ftinit.py +++ b/tests/test_init_ftinit.py @@ -137,3 +137,50 @@ def test_forced_flux_lower_bound_is_respected(): res = run_ftinit(model, scores) assert res.fluxes["R6"] >= 2.0 - 1e-6 assert "R6" not in res.deleted_reactions + + +# --------------------------------------------------------------------------- # +# canonical (deterministic uniqueness) — Tier 2 item 4. +# --------------------------------------------------------------------------- # +def _degenerate_model(): + """Two interchangeable negative-score reactions (R1, R2) both feed an essential E. + + Exactly one is needed to make E carry flux, so the score optimum (-1) is degenerate + between keeping R1 or R2. Which one a plain solve keeps is an arbitrary tie-break. + """ + m = cobra.Model("degen") + a, mm, p = (cobra.Metabolite(x, name=x, compartment="s") for x in ("a", "m", "p")) + m.add_metabolites([a, mm, p]) + R1 = cobra.Reaction("R1", lower_bound=0, upper_bound=1000) + R1.add_metabolites({a: -1, mm: 1}) + R2 = cobra.Reaction("R2", lower_bound=0, upper_bound=1000) + R2.add_metabolites({a: -1, mm: 1}) + E = cobra.Reaction("E", lower_bound=0, upper_bound=1000) + E.add_metabolites({mm: -1, p: 1}) + EXa = cobra.Reaction("EX_a", lower_bound=-1000, upper_bound=1000) + EXa.add_metabolites({a: -1}) + EXp = cobra.Reaction("EX_p", lower_bound=-1000, upper_bound=1000) + EXp.add_metabolites({p: -1}) + m.add_reactions([R1, R2, E, EXa, EXp]) + m.objective = "E" + return m + + +def test_canonical_breaks_degenerate_tie_by_id(): + """canonical selects the unique sparsest, lowest-id optimum among equal alternatives.""" + m = _degenerate_model() + res = run_ftinit(m, {"R1": -1.0, "R2": -1.0}, essential_rxns=["E"], canonical=True) + # exactly one of the degenerate pair is kept (the tie is resolved, not doubled) ... + assert len({"R1", "R2"} & set(res.kept_reactions)) == 1 + # ... and it is deterministically the lower-id one. + assert "R1" in res.kept_reactions and "R2" not in res.kept_reactions + # the reported objective is the primary (score) optimum, not the phase-2 secondary. + assert res.objective == pytest.approx(-1.0, abs=1e-6) + + +def test_canonical_safe_on_unique_optimum(): + """On a non-degenerate model canonical returns the same optimum (no regression).""" + model = make_test_model() + res = run_ftinit(model, _scores(model), canonical=True) + assert set(res.kept_reactions) == _LOOP + assert res.objective == pytest.approx(8.0, abs=1e-6) diff --git a/tests/test_init_pipeline.py b/tests/test_init_pipeline.py index 910d727f..e5dee74f 100644 --- a/tests/test_init_pipeline.py +++ b/tests/test_init_pipeline.py @@ -159,3 +159,26 @@ def test_essential_merged_away_is_skipped(): prep = prep_init_model(m, [task], ext_comp="s") # must not raise assert "REV" not in prep.essential_rxns # merged into a collapsed group + + +# --------------------------------------------------------------------------- # +# strict_gap / canonical (deterministic extraction) — Tier 2 items 4 & 5. +# These are opt-in; the default path stays exact-RAVEN. On the toy oracle they must +# reproduce T0001 (no regression) and be run-to-run identical. +# --------------------------------------------------------------------------- # +def test_ftinit_strict_gap_matches_oracle(): + """strict_gap: one near-proven-optimal solve per step, still gives T0001.""" + model = make_test_model() + prep = prep_init_model(model, ext_comp="s") + out = ftinit(prep, _scores(model), strict_gap=True) + assert {r.id for r in out.reactions} == set(TEST_MODEL_FTINIT_NO_TASKS) + + +def test_ftinit_canonical_matches_oracle_and_is_stable(): + """canonical (+ strict_gap): preserves T0001 and is identical across repeated runs.""" + model = make_test_model() + prep = prep_init_model(model, ext_comp="s") + out1 = ftinit(prep, _scores(model), strict_gap=True, canonical=True) + out2 = ftinit(prep, _scores(model), strict_gap=True, canonical=True) + assert {r.id for r in out1.reactions} == set(TEST_MODEL_FTINIT_NO_TASKS) + assert {r.id for r in out1.reactions} == {r.id for r in out2.reactions} diff --git a/tests/test_init_taskfill.py b/tests/test_init_taskfill.py index add56937..2db517f5 100644 --- a/tests/test_init_taskfill.py +++ b/tests/test_init_taskfill.py @@ -157,3 +157,29 @@ def test_additions_carry_forward_to_later_tasks(): res = fill_tasks(gapped, ref, [first, second]) assert res.added_reactions == ["R7"] # added once; the second task saw R7 already there assert not res.failed_tasks + + +def test_canonical_gap_fill_breaks_tie_by_id(): + """Two equal-cost candidate fills: canonical adds the lower-id one deterministically.""" + import cobra + + from raven_toolbox.tasks import Task + + ref = cobra.Model("ref") + A, M, P = (cobra.Metabolite(x, name=x, compartment="s") for x in ("A", "M", "P")) + ref.add_metabolites([A, M, P]) + RA = cobra.Reaction("RA", lower_bound=0, upper_bound=1000) + RA.add_metabolites({A: -1, M: 1}) + RB = cobra.Reaction("RB", lower_bound=0, upper_bound=1000) + RB.add_metabolites({A: -1, M: 1}) + RP = cobra.Reaction("RP", lower_bound=0, upper_bound=1000) + RP.add_metabolites({M: -1, P: 1}) + ref.add_reactions([RA, RB, RP]) + task = Task(id="mkP", inputs=[("A[s]", 0.0, 1000.0)], outputs=[("P[s]", 1.0, 1000.0)]) + + gapped = ref.copy() + gapped.remove_reactions(["RA", "RB"], remove_orphans=False) # both routes to M removed + # exactly one equal-cost route is needed; canonical must pick the lower id (RA). + res = fill_tasks(gapped, ref, [task], canonical=True) + assert res.added_reactions == ["RA"] + assert not res.failed_tasks diff --git a/tests/test_io_yaml.py b/tests/test_io_yaml.py index eaaf450e..fe2231c6 100644 --- a/tests/test_io_yaml.py +++ b/tests/test_io_yaml.py @@ -251,7 +251,11 @@ def test_code_built_model_round_trips(tmp_path): assert rr.get_coefficient("a_c") == -1 assert rr.get_coefficient("b_c") == 2 assert rr.gene_reaction_rule == "G1" - assert rr.subsystem == "glycolysis" + # RAVEN/raven-toolbox's shared format always writes subsystem as a + # list (a reaction can carry more than one), even for the single- + # subsystem case set here as a plain string — so it round-trips as + # a one-item list, not the original bare string. + assert rr.subsystem == ["glycolysis"] ma = back.metabolites.get_by_id("a_c") assert ma.formula == "C6H12O6" assert ma.annotation["kegg.compound"] == ["C00031"] diff --git a/tests/test_io_yaml_parity.py b/tests/test_io_yaml_parity.py index 6fff479d..8c21c544 100644 --- a/tests/test_io_yaml_parity.py +++ b/tests/test_io_yaml_parity.py @@ -98,7 +98,11 @@ def test_round_trip_preserves_every_raven_field(src, tmp_path): r = reloaded.reactions.get_by_id("R1") assert r.bounds == (-1000.0, 1000.0) assert r.gene_reaction_rule == "G1" - assert r.subsystem == "glycolysis" + # RAVEN/raven-toolbox's shared format always writes subsystem as a + # list (a reaction can carry more than one), even for the single- + # subsystem case here — so it round-trips as a one-item list, not + # the source fixture's bare string. + assert r.subsystem == ["glycolysis"] # Metabolite RAVEN extras. a = reloaded.metabolites.get_by_id("s_0001") diff --git a/tests/test_io_yaml_raven_sections.py b/tests/test_io_yaml_raven_sections.py new file mode 100644 index 00000000..52df0683 --- /dev/null +++ b/tests/test_io_yaml_raven_sections.py @@ -0,0 +1,101 @@ +"""RAVEN MATLAB omits empty top-level sections; the reader must tolerate that. + +``writeYAMLmodel`` writes no ``genes:`` block at all for a model without genes +(RAVEN's own ``tutorial/small.yml`` is such a model). cobra's +``model_from_dict`` indexes ``obj["genes"]`` directly, so those files used to +fail to load with ``KeyError: 'genes'`` -- raven-toolbox could not read a valid +model written by the toolbox it is the counterpart of. +""" +from __future__ import annotations + +import textwrap + +import pytest + +from raven_toolbox.io import read_yaml_model, write_yaml_model + +# A minimal RAVEN-shaped model with no genes, and therefore no ``genes:`` +# section. Authored here rather than copied from RAVEN, which is GPL. +NO_GENES = textwrap.dedent( + """\ + --- + - metaData: !!omap + - id: tiny + - name: No-gene test model + - metabolites: + - !!omap + - id: a_c + - name: A + - compartment: c + - !!omap + - id: b_c + - name: B + - compartment: c + - reactions: + - !!omap + - id: R1 + - name: A to B + - metabolites: !!omap + - a_c: -1 + - b_c: 1 + - lower_bound: 0 + - upper_bound: 1000 + - compartments: !!omap + - c: cytoplasm + """ +) + + +@pytest.fixture +def no_gene_file(tmp_path): + path = tmp_path / "tiny.yml" + path.write_text(NO_GENES, encoding="utf-8") + return path + + +def test_reads_model_without_genes_section(no_gene_file): + model = read_yaml_model(no_gene_file) + + assert [r.id for r in model.reactions] == ["R1"] + assert {m.id for m in model.metabolites} == {"a_c", "b_c"} + assert len(model.genes) == 0 + + +def test_round_trips_model_without_genes_section(no_gene_file, tmp_path): + model = read_yaml_model(no_gene_file) + out = tmp_path / "rewritten.yml" + write_yaml_model(model, out) + + again = read_yaml_model(out) + assert [r.id for r in again.reactions] == [r.id for r in model.reactions] + assert [m.id for m in again.metabolites] == [m.id for m in model.metabolites] + assert again.reactions.R1.bounds == model.reactions.R1.bounds + + +def test_model_with_no_entity_sections_at_all(tmp_path): + """A skeleton model -- metadata and compartments only -- still loads. + + Only sections that are *empty* are omitted by RAVEN. A file with reactions + but no ``metabolites:`` is not a RAVEN file, it is a broken one, and is + expected to fail loudly rather than load a model whose reactions reference + metabolites that do not exist. + """ + path = tmp_path / "skeleton.yml" + path.write_text( + textwrap.dedent( + """\ + --- + - metaData: !!omap + - id: skeleton + - compartments: !!omap + - c: cytoplasm + """ + ), + encoding="utf-8", + ) + + model = read_yaml_model(path) + + assert len(model.reactions) == 0 + assert len(model.metabolites) == 0 + assert len(model.genes) == 0 diff --git a/tests/test_localization_certify.py b/tests/test_localization_certify.py index ec1105ae..b9047941 100644 --- a/tests/test_localization_certify.py +++ b/tests/test_localization_certify.py @@ -316,6 +316,49 @@ def test_gapfill_restores_function(): assert _grows(apply_assignment(m, res, universal=_universal())) +def test_gapfill_returns_a_growth_restoring_set(): + # Directly exercise the flux-based _gapfill: on a draft that cannot grow it returns the universal + # reaction that restores biomass, and nothing spurious. As a plain LP it is reliable where cobra's + # indicator gap-fill MILP is not (that MILP fails to find such a fill in most genome-scale cases even + # when it exists) -- and a returned set always actually restores growth. + from raven_toolbox.localization.certify import _gapfill + + applied = _gap_draft() # EX_A -> r1: A->B; bio: B + C -> (C has no producer) + added = _gapfill(applied, _universal(), "bio", min_growth=1.0) + assert added == ["rC"] # the exact missing producer of C, nothing extra + + ur = _universal().reactions.get_by_id("rC") + nr = cobra.Reaction(ur.id, lower_bound=ur.lower_bound, upper_bound=ur.upper_bound) + applied.add_reactions([nr]) + nr.add_metabolites({applied.metabolites.get_by_id(m.id): c for m, c in ur.metabolites.items()}) + assert _grows(applied) + + +def test_gapfill_offers_nothing_when_the_universal_cannot_restore_growth(): + # If even adding every candidate cannot reach the floor, _gapfill returns [] (no false fill). + from raven_toolbox.localization.certify import _gapfill + + applied = _gap_draft() + empty = cobra.Model("empty") # no candidate produces C + assert _gapfill(applied, empty, "bio", min_growth=1.0) == [] + + +def test_gapfill_warns_on_namespace_mismatch_instead_of_failing_silently(): + # A universal whose metabolite ids don't match the model can't connect; _gapfill returns [] but must + # WARN, so "found nothing" is distinguishable from "wrong namespace". + from raven_toolbox.localization.certify import _gapfill + + applied = _gap_draft() + foreign = cobra.Model("foreign") # same chemistry (A->C) under foreign ids + a, c = cobra.Metabolite("A_x", compartment="c"), cobra.Metabolite("C_x", compartment="c") + foreign.add_metabolites([a, c]) + rc = cobra.Reaction("rC", lower_bound=0, upper_bound=1000) + rc.add_metabolites({a: -1, c: 1}) + foreign.add_reactions([rc]) + with pytest.warns(UserWarning, match="namespace"): + assert _gapfill(applied, foreign, "bio", min_growth=1.0) == [] + + def test_no_gratuitous_gapfill(): # When the draft already grows, no universal candidate is pulled (it is only reached on a real # growth failure). diff --git a/tests/test_localization_curation.py b/tests/test_localization_curation.py index caa0f611..b9d5cb2e 100644 --- a/tests/test_localization_curation.py +++ b/tests/test_localization_curation.py @@ -369,3 +369,23 @@ def test_affected_for_transport_lists_dependent_reactions(): # the palmitoyl-CoA transport is coupled to the reactions that make/consume it (r0 in c, r1 in m) assert trow["n_affected"] >= 1 assert "r1" in trow["affected"] + + +def test_curation_priority_skips_curated_localization(): + """A reaction whose localization facet is mark_curated drops out of the review queue (closing the + score -> review -> curate loop); include_curated=True keeps it.""" + from raven_toolbox.confidence import mark_curated + + m = _linear() # r1: A_c -> B_c, gene g1 + scores = LocalizationScores(pd.DataFrame({"c": [0.4], "m": [0.9]}, index=["g1"])) + prop = _proposal({"r1": ["c"]}, unplaced=["r1"]) # no_evidence + override fire on r1 + + base = curation_priority(m, prop, scores, check_essential=False) + assert "r1" in set(base["target"]) # surfaces before curation + + mark_curated(m.reactions.get_by_id("r1"), facet="localization") + after = curation_priority(m, prop, scores, check_essential=False) + assert "r1" not in set(after["target"]) # dropped once curated + + kept = curation_priority(m, prop, scores, check_essential=False, include_curated=True) + assert "r1" in set(kept["target"]) # include_curated overrides diff --git a/tests/test_manipulation_compartments.py b/tests/test_manipulation_compartments.py index ba353c36..c01f8907 100644 --- a/tests/test_manipulation_compartments.py +++ b/tests/test_manipulation_compartments.py @@ -4,7 +4,19 @@ import cobra import pytest -from raven_toolbox.manipulation.compartments import copy_to_compartment, merge_compartments +from raven_toolbox import manipulation +from raven_toolbox.manipulation import copy_to_compartment, merge_compartments + + +def test_exported_from_the_package(): + """Both are public API, not just importable from the submodule. + + They were missing from ``manipulation.__all__`` for a long time precisely because these + tests reached into ``manipulation.compartments`` directly, so nothing exercised the path + a user actually takes. + """ + assert "merge_compartments" in manipulation.__all__ + assert "copy_to_compartment" in manipulation.__all__ def _two_compartment_model() -> cobra.Model: @@ -101,6 +113,40 @@ def test_merge_compartments_keeps_single_met_reactions_when_asked(): assert "sym" in {r.id for r in merged_keep.reactions} +def test_merge_compartments_keeps_pre_existing_exchange_reactions(): + """An exchange had one metabolite *before* merging, so it never became trivial and must + survive. Dropping these silently strips a model of its medium and its biomass reaction: + smallYeast grows at 0.1222 before flattening and at 0.0000 after, with no warning.""" + m = _two_compartment_model() + uptake = cobra.Reaction("EX_A", lower_bound=-1000, upper_bound=1000) + uptake.add_metabolites({m.metabolites.A_c: 1}) + m.add_reactions([uptake]) + + merged, deleted, _ = merge_compartments(m) + + assert "EX_A" not in deleted + assert "EX_A" in {r.id for r in merged.reactions} + assert len(merged.boundary) == 1 + # the genuinely-collapsed transport is still dropped + assert "tr_A" in deleted + + +def test_merge_compartments_carries_the_objective_over(): + """The merged model is rebuilt from scratch, so the objective must be copied across. + Without this the caller gets a model that silently optimises to 0.0.""" + m = _two_compartment_model() + m.objective = "r_c" + + merged, _, dupes = merge_compartments(m) + + # r_c and r_m merge to the same reaction; whichever survives carries the objective. + survivor = next(r for r in merged.reactions if r.id in {"r_c", "r_m"}) + assert str(merged.objective.expression) != "0" + if survivor.id == "r_c": + assert survivor.objective_coefficient == 1.0 + assert merged.objective_direction == m.objective_direction + + def test_merge_compartments_deduplicate_off_keeps_both(): m = _two_compartment_model() merged, _, dupes = merge_compartments(m, deduplicate_reactions=False) diff --git a/tests/test_manipulation_irreversible.py b/tests/test_manipulation_irreversible.py index ad4cd05d..80a6f7d4 100644 --- a/tests/test_manipulation_irreversible.py +++ b/tests/test_manipulation_irreversible.py @@ -142,3 +142,38 @@ def test_conversion_is_idempotent_after_first_pass(): second = convert_to_irreversible(model) assert second == [] assert "r1_REV_REV" not in {r.id for r in model.reactions} + + +def test_reverse_reaction_inherits_annotation_subsystem_and_notes(): + """MATLAB's convertToIrrev copies the per-reaction fields (eccodes, + rxnMiriams, subSystems, rxnNotes, ...) onto the reverse reaction. + Losing them here leaves e.g. GECKO's reverse reactions without an EC + code, and therefore without a kcat.""" + model = _build_model_with_bounds([ + ("r1", {"A": -1.0, "B": 1.0}, -500.0, 1000.0), + ]) + forward = model.reactions.get_by_id("r1") + forward.annotation["ec-code"] = "1.1.1.1" + forward.subsystem = "Glycolysis" + forward.notes["origin"] = "curated" + + convert_to_irreversible(model) + + rev = model.reactions.get_by_id("r1_REV") + assert rev.annotation["ec-code"] == "1.1.1.1" + assert rev.subsystem == "Glycolysis" + assert rev.notes["origin"] == "curated" + + +def test_reverse_reaction_annotation_is_a_copy_not_an_alias(): + """Editing the reverse reaction's annotations must not write through + to the forward reaction.""" + model = _build_model_with_bounds([ + ("r1", {"A": -1.0, "B": 1.0}, -500.0, 1000.0), + ]) + model.reactions.get_by_id("r1").annotation["ec-code"] = "1.1.1.1" + + convert_to_irreversible(model) + + model.reactions.get_by_id("r1_REV").annotation["ec-code"] = "2.2.2.2" + assert model.reactions.get_by_id("r1").annotation["ec-code"] == "1.1.1.1" diff --git a/tests/test_parameters.py b/tests/test_parameters.py index f681833a..0cea746f 100644 --- a/tests/test_parameters.py +++ b/tests/test_parameters.py @@ -1,8 +1,12 @@ -"""Tests for set_variance_bounds (the var mode of setParam).""" +"""Tests for set_variance_bounds (the var mode of setParam) and set_exchange_bounds.""" import cobra import pytest -from raven_toolbox.manipulation import add_reactions_from_equations, set_variance_bounds +from raven_toolbox.manipulation import ( + add_reactions_from_equations, + set_exchange_bounds, + set_variance_bounds, +) @pytest.fixture @@ -58,3 +62,92 @@ def test_length_mismatch_raises(model): def test_unknown_reaction_raises(model): with pytest.raises(ValueError, match="not found"): set_variance_bounds(model, "NOPE", 1, 5) + + +@pytest.fixture +def exchange_model(): + m = cobra.Model("t") + m.compartments = {"e": "extracellular", "c": "cytosol"} + a_e = cobra.Metabolite("a_e", compartment="e") + b_e = cobra.Metabolite("b_e", compartment="e") + c_c = cobra.Metabolite("c_c", compartment="c") + m.add_metabolites([a_e, b_e, c_c]) + # Consistent direction: the metabolite is a reactant (coeff -1) in every + # exchange, so positive flux always means export, negative always import. + ex_a = cobra.Reaction("EX_a", lower_bound=0, upper_bound=1000) + ex_a.add_metabolites({a_e: -1}) + ex_b = cobra.Reaction("EX_b", lower_bound=0, upper_bound=1000) + ex_b.add_metabolites({b_e: -1}) + sk_c = cobra.Reaction("SK_c", lower_bound=0, upper_bound=1000) + sk_c.add_metabolites({c_c: -1}) + m.add_reactions([ex_a, ex_b, sk_c]) + return m + + +def test_exchange_bounds_basic_and_close_others(exchange_model): + unused = set_exchange_bounds(exchange_model, "a_e", -10, 5) + assert unused == [] + assert exchange_model.reactions.EX_a.bounds == (-10, 5) + # direction is "backward" (import = negative flux); close_others closes + # only import, i.e. sets lb=0, leaving export (ub) untouched. + assert exchange_model.reactions.EX_b.bounds == (0, 1000) + assert exchange_model.reactions.SK_c.bounds == (0, 1000) + + +def test_exchange_bounds_media_only(exchange_model): + set_exchange_bounds(exchange_model, None, -5, 5, media_only=True) + assert exchange_model.reactions.EX_a.bounds == (-5, 5) + assert exchange_model.reactions.EX_b.bounds == (-5, 5) + # SK_c's metabolite (c_c) is cytosolic, not extracellular: untouched. + assert exchange_model.reactions.SK_c.bounds == (0, 1000) + + +def test_exchange_bounds_media_only_requires_extracellular_compartment(): + m = cobra.Model("t") + m.compartments = {"c": "cytosol"} + met = cobra.Metabolite("a_c", compartment="c") + m.add_metabolites([met]) + rxn = cobra.Reaction("EX_a", lower_bound=0, upper_bound=1000) + rxn.add_metabolites({met: -1}) + m.add_reactions([rxn]) + with pytest.raises(ValueError, match="extracellular"): + set_exchange_bounds(m, None, -5, 5, media_only=True) + + +def test_exchange_bounds_mixed_direction_disables_close_others(exchange_model): + # Flip EX_b to the opposite sign convention (metabolite as product). + exchange_model.reactions.EX_b.add_metabolites({exchange_model.metabolites.b_e: 2}, combine=True) + with pytest.warns(UserWarning, match="differ in direction"): + set_exchange_bounds(exchange_model, "a_e", -10, 5, close_others=True) + assert exchange_model.reactions.EX_a.bounds == (-10, 5) + # close_others was forced off: EX_b and SK_c keep their original bounds. + assert exchange_model.reactions.EX_b.bounds == (0, 1000) + assert exchange_model.reactions.SK_c.bounds == (0, 1000) + + +def test_exchange_bounds_unused_mets(exchange_model): + unused = set_exchange_bounds(exchange_model, ["a_e", "nonexistent"], [-10, -5], [0, 0]) + assert unused == ["nonexistent"] + assert exchange_model.reactions.EX_a.bounds == (-10, 0) + + +def test_exchange_bounds_no_mets_requires_scalar(exchange_model): + with pytest.raises(ValueError, match="Only one upper and one lower bound"): + set_exchange_bounds(exchange_model, None, [-10, -5], [0, 0]) + + +def test_exchange_bounds_no_mets_applies_to_all(exchange_model): + set_exchange_bounds(exchange_model, None, -5, 5) + for rid in ("EX_a", "EX_b", "SK_c"): + assert exchange_model.reactions.get_by_id(rid).bounds == (-5, 5) + + +def test_exchange_bounds_warns_on_metabolite_in_multiple_exchanges(exchange_model): + extra = cobra.Reaction("EX_a2", lower_bound=0, upper_bound=1000) + extra.add_metabolites({exchange_model.metabolites.a_e: -1}) + exchange_model.add_reactions([extra]) + with pytest.warns(UserWarning, match="more than one exchange reaction"): + set_exchange_bounds(exchange_model, "a_e", -10, 5) + # both reactions exchanging a_e get the requested bounds. + assert exchange_model.reactions.EX_a.bounds == (-10, 5) + assert exchange_model.reactions.EX_a2.bounds == (-10, 5) diff --git a/tests/test_reconstruction_homology_candidates.py b/tests/test_reconstruction_homology_candidates.py new file mode 100644 index 00000000..4b8348c3 --- /dev/null +++ b/tests/test_reconstruction_homology_candidates.py @@ -0,0 +1,138 @@ +"""``review_identity`` reports the near misses without letting them in. + +The filters are set strict on purpose: a wrongly transferred reaction is hard to +find later and harder to remove, while a missing one can be gap-filled. That +argument justifies rejecting borderline hits -- it does not justify throwing away +the evidence for them, which is what these tests pin down. +""" +from __future__ import annotations + +import cobra +import pandas as pd +import pytest + +from raven_toolbox.reconstruction.homology import get_model_from_homology +from raven_toolbox.reconstruction.homology.hits import HIT_COLUMNS + + +def _template() -> cobra.Model: + model = cobra.Model("tmpl") + mets = {m: cobra.Metabolite(m, compartment="c") for m in ("a", "b", "c")} + model.add_metabolites(list(mets.values())) + for rid, stoich, gene in ( + ("STRONG", {"a": -1, "b": 1}, "gStrong"), + ("WEAK", {"b": -1, "c": 1}, "gWeak"), + ): + rxn = cobra.Reaction(rid, lower_bound=0.0, upper_bound=1000.0) + rxn.add_metabolites({mets[m]: v for m, v in stoich.items()}) + model.add_reactions([rxn]) + rxn.gene_reaction_rule = gene + return model + + +def _hits(weak_identity: float) -> pd.DataFrame: + """One comfortable ortholog and one whose identity is the variable.""" + rows = [] + for template_gene, target_gene, identity in ( + ("gStrong", "tStrong", 90.0), + ("gWeak", "tWeak", weak_identity), + ): + rows.append(("tgt", "tmpl", target_gene, template_gene, 0.0, identity, 500, 400.0, 100.0)) + rows.append(("tmpl", "tgt", template_gene, target_gene, 0.0, identity, 500, 400.0, 100.0)) + return pd.DataFrame(rows, columns=HIT_COLUMNS) + + +def test_no_candidates_reported_unless_asked(): + result = get_model_from_homology([_template()], _hits(30.0), "tgt") + + assert result.candidates is None + assert {r.id for r in result.model.reactions} == {"STRONG"} + + +def test_borderline_reaction_is_reported_but_not_transferred(): + result = get_model_from_homology( + [_template()], _hits(30.0), "tgt", review_identity=25 + ) + + assert {r.id for r in result.model.reactions} == {"STRONG"}, ( + "a candidate must never reach the model -- that is the whole point" + ) + assert list(result.candidates.reaction) == ["WEAK"] + + row = result.candidates.iloc[0] + assert row.template_gene == "gWeak" + assert row.target_gene == "tWeak" + assert row.identity == pytest.approx(30.0) + assert row.n_support == 1 + + +def test_reactions_that_pass_are_not_also_listed_as_candidates(): + result = get_model_from_homology( + [_template()], _hits(80.0), "tgt", review_identity=25 + ) + + assert {r.id for r in result.model.reactions} == {"STRONG", "WEAK"} + assert result.candidates.empty, "nothing was rejected, so nothing is a near miss" + + +def test_candidates_are_ordered_strongest_first(): + """A curator reads from the top, so the best-supported must be there.""" + template = _template() + extra = cobra.Reaction("WEAKER", lower_bound=0.0, upper_bound=1000.0) + extra.add_metabolites({template.metabolites.get_by_id("c"): -1}) + template.add_reactions([extra]) + extra.gene_reaction_rule = "gWeaker" + + hits = pd.concat([ + _hits(30.0), + pd.DataFrame( + [ + ("tgt", "tmpl", "tWeaker", "gWeaker", 0.0, 27.0, 500, 300.0, 100.0), + ("tmpl", "tgt", "gWeaker", "tWeaker", 0.0, 27.0, 500, 300.0, 100.0), + ], + columns=HIT_COLUMNS, + ), + ]) + + result = get_model_from_homology([template], hits, "tgt", review_identity=25) + + assert list(result.candidates.reaction) == ["WEAK", "WEAKER"] + assert list(result.candidates.identity) == pytest.approx([30.0, 27.0]) + + +def test_review_identity_above_min_identity_is_rejected(): + """It exists to catch what min_identity turns away; above it, it catches nothing.""" + with pytest.raises(ValueError, match="must be below min_identity"): + get_model_from_homology( + [_template()], _hits(30.0), "tgt", min_identity=40, review_identity=45 + ) + + +def test_reported_identity_is_the_direction_that_did_the_rejecting(): + """A one-sided strong hit must not be reported as though it were the reason. + + Matching is bidirectional: a pair has to clear the threshold both ways. If + the forward hit is 44 % and the reverse 30 %, the pair is rejected on the 30. + Reporting the 44 would show a curator a comfortable match with no explanation + of why it was turned away. + """ + template = _template() + hits = pd.DataFrame( + [ + # STRONG transfers normally. + ("tgt", "tmpl", "tStrong", "gStrong", 0.0, 90.0, 500, 400.0, 100.0), + ("tmpl", "tgt", "gStrong", "tStrong", 0.0, 90.0, 500, 400.0, 100.0), + # WEAK looks comfortable one way and marginal the other. + ("tgt", "tmpl", "tWeak", "gWeak", 0.0, 44.0, 500, 400.0, 100.0), + ("tmpl", "tgt", "gWeak", "tWeak", 0.0, 30.0, 500, 300.0, 100.0), + ], + columns=HIT_COLUMNS, + ) + + result = get_model_from_homology([template], hits, "tgt", review_identity=25) + + assert {r.id for r in result.model.reactions} == {"STRONG"} + assert list(result.candidates.reaction) == ["WEAK"] + assert result.candidates.iloc[0].identity == pytest.approx(30.0), ( + "the limiting direction, not the flattering one" + ) diff --git a/tests/test_reconstruction_kegg_query.py b/tests/test_reconstruction_kegg_query.py index 0dd23f09..65c981f3 100644 --- a/tests/test_reconstruction_kegg_query.py +++ b/tests/test_reconstruction_kegg_query.py @@ -127,6 +127,50 @@ def test_get_model_from_sequences(reference_and_tables, monkeypatch): assert model.id == "myorg" r = model.reactions.get_by_id("R90010") assert set(r.gene_reaction_rule.split(" or ")) == {"myGeneA", "myGeneB"} - assert r.notes["note"].endswith("(using HMMs)") + assert r.notes["note"] == "Included by KEGG HMM reconstruction" + # Its single KO matched, so the kegg.orthology annotation is unchanged. + assert r.annotation["kegg.orthology"] == ["K90001"] # R90200/R90300 had no matched KOs and are not spontaneous -> absent. assert "R90200" not in model.reactions + + +def test_model_id_defaults_to_fasta_stem(reference_and_tables, monkeypatch): + """RAVEN always sets model.id; with no model_id we default it to the FASTA stem + rather than inheriting the reference model's id.""" + model_ref, tables = reference_and_tables + monkeypatch.setattr( + "raven_toolbox.reconstruction.kegg.query.run_hmmsearch", + lambda *a, **k: "myGeneA - K90001 - 1e-120 400 0\n", + ) + model = get_kegg_model_from_sequences( + "/some/path/eco.faa", model_ref, tables["ko_reaction"], "ignored.hmm", + rxn_flags=tables["rxn_flags"], + ) + assert model.id == "eco" + + +def test_prune_orthology_keeps_only_matched_kos(): + """The FASTA path prunes a kept reaction's kegg.orthology to the KOs that + matched a gene (RAVEN getKEGGModelForOrganism HMM branch), preserving order.""" + import cobra + + from raven_toolbox.reconstruction.kegg.assemble import assemble_model_from_ko_genes + + ref = cobra.Model("ref") + met = cobra.Metabolite("C1", compartment="s") + rxn = cobra.Reaction("R1") + rxn.add_metabolites({met: -1.0}) + rxn.annotation["kegg.orthology"] = ["K1", "K2", "K3"] + ref.add_reactions([rxn]) + ko_reaction = pd.DataFrame( + [("K1", "R1"), ("K2", "R1"), ("K3", "R1")], columns=["ko", "reaction"] + ) + ko_to_genes = {"K2": ["g"]} # only K2 matched a gene + + pruned, _ = assemble_model_from_ko_genes( + ref, ko_reaction, ko_to_genes, prune_orthology=True + ) + assert pruned.reactions.get_by_id("R1").annotation["kegg.orthology"] == ["K2"] + # Default (organism path) keeps the full reference KO list. + full, _ = assemble_model_from_ko_genes(ref, ko_reaction, ko_to_genes) + assert full.reactions.get_by_id("R1").annotation["kegg.orthology"] == ["K1", "K2", "K3"] diff --git a/tests/test_workflows.py b/tests/test_workflows.py new file mode 100644 index 00000000..8bf48351 --- /dev/null +++ b/tests/test_workflows.py @@ -0,0 +1,102 @@ +"""Every ``run:`` block in a workflow has to be valid shell. + +Nothing checks a workflow's shell until the job runs. For the workflows that run +on a schedule, that means a quoting mistake sits there silently and costs a whole +night -- and PR CI never exercises them, so review will not catch it either. + +Not hypothetical: a stray backslash-quote at the end of one line in +``parity-nightly.yml`` killed the job at its licence step. +""" + +from __future__ import annotations + +import shutil +import subprocess +from pathlib import Path + +import pytest +from ruamel.yaml import YAML + + +def usable_bash() -> str | None: + """Return a bash that can actually parse a script, or None. + + On Windows `bash` often resolves to the WSL stub, which exits non-zero with + "Windows Subsystem for Linux has no installed distributions" no matter what + it is handed. Checking only that the name resolves turns that into three + confusing failures about the workflows, which are fine. + """ + candidate = shutil.which("bash") + if not candidate: + return None + probe = subprocess.run([candidate, "-n"], input=b":\n", capture_output=True) + return candidate if probe.returncode == 0 else None + + +BASH = usable_bash() + + +WORKFLOWS = sorted((Path(__file__).resolve().parents[1] / ".github" / "workflows").glob("*.yml")) + + +def read(path: Path) -> dict: + """Parse a workflow, with CRLF normalised away. + + Git stores these files with LF, which is what a runner gets. A Windows + working copy has CRLF, and bash reports a stray carriage return as a syntax + error -- so without this the check would fail on every Windows machine and + pass in CI, which is the wrong way round. + """ + return YAML(typ="safe").load(path.read_text(encoding="utf-8").replace("\r\n", "\n")) + + +def run_blocks(path: Path): + """Yield (step label, script) for every ``run:`` in the workflow.""" + document = read(path) + for job_name, job in (document.get("jobs") or {}).items(): + for index, step in enumerate(job.get("steps") or []): + script = step.get("run") + if script: + yield f"{path.name}:{job_name}:{step.get('name') or index}", script + + +def strip_expressions(script: str) -> str: + """Replace ``${{ ... }}`` with a literal, so this checks shell not templating. + + Actions substitutes these before bash sees them. Leaving them in would make + bash parse ``{{`` as shell; dropping them entirely would let an unquoted + empty expansion look like valid syntax when it is not. + """ + lines = [] + for line in script.split("\n"): + while "${{" in line and "}}" in line: + head, rest = line.split("${{", 1) + line = head + "PLACEHOLDER" + rest.split("}}", 1)[1] + lines.append(line) + return "\n".join(lines) + + +def test_the_workflow_directory_was_actually_found(): + """Otherwise every test below passes by having nothing to check.""" + assert WORKFLOWS, "no workflow files found" + + +@pytest.mark.parametrize("path", WORKFLOWS, ids=lambda p: p.name) +def test_workflow_is_valid_yaml(path: Path): + assert read(path) + + +@pytest.mark.skipif(BASH is None, reason="no usable bash") +@pytest.mark.parametrize("path", WORKFLOWS, ids=lambda p: p.name) +def test_every_run_block_parses_as_shell(path: Path): + assert BASH is not None # skipif guarantees this; mypy does not know that + for label, script in run_blocks(path): + # Bytes, not text: in text mode Python translates "\n" to "\r\n" on the + # way into stdin under Windows, and bash then rejects the carriage + # returns it just received. + result = subprocess.run( + [BASH, "-n"], + input=strip_expressions(script).encode("utf-8"), + capture_output=True, + ) + assert result.returncode == 0, f"{label}\n{result.stderr.decode('utf-8', 'replace')}"