Repository navigation
Support non-perturbed scRNA-seq and mouse data - #23
Merged
Merged
Conversation
Non-perturbed data (no guide information): - K-selection: new --no_trait / --no_perturbation flags skip the trait and perturbation inputs and plots; the shared regulator row is factored into _plot_regulator_row and hidden when there is no perturbation data. - Program analysis: tolerate a missing uns['guide_targets']. - Excel summary: skip the Targets Summary sheet (and the specificity scores it writes) when there are no guides; fall back to var_names when the gene symbol column is absent. - Evaluation: fail early with a clear message when --Perform_perturbation is requested without guide information. Mouse data: - Mouse genes are matched to human gene-set libraries and gene summaries through curated orthologs (union of MGI and Ensembl, new orthologs.py) instead of upper-casing symbols. The table is downloaded once and cached in src/Stage2_Evaluation/Resources/; --ortholog_table pins a fixed copy. Genes without a human ortholog are excluded from enrichment; one-to-many orthologs keep all human genes. MSigDB mouse gene sets keep mouse symbols. - Excel summary: --organism mouse annotates genes with their human orthologs' MyGene summaries and a 'Human ortholog' column. - Evaluation: trait (GWAS) enrichment is human-only and is skipped with a message for other organisms. - Plotting utilities: honor the species argument of the MyGene helpers. Human runs are unchanged: existing Stage 2/3 tests give identical per-test outcomes before and after, and human evaluation outputs match apart from gene order within the Genes column and last-digit rounding. Tests: tests/Script/test_non_perturbed.py and tests/Script/Stage2_Evaluation/test_orthologs.py (synthetic data, offline). Docs: README section on non-perturbed and mouse data, parameter tables. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Data converted from Seurat stores its embeddings as obsm['UMAP'] / obsm['PCA'] (often as DataFrames), which scanpy does not see, so program and perturbed-gene plots recomputed a different UMAP. ensure_umap now copies such embeddings to 'X_umap' (first two columns) / 'X_pca' before deciding whether to compute one. Existing X_umap/X_pca entries and the original keys are left untouched; files without these keys behave as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- orthologs.py: require BioMart's completion stamp so a truncated Ensembl response is rejected instead of cached; write the cache atomically (temp file + os.replace) so parallel jobs never read a partial table; on download failure, point to --ortholog_table. - Excel summary: map all top genes to orthologs once instead of once per program. - --organism accepts only human/mouse (case-insensitive) in evaluation and the Excel summary. - Evaluation checks for guide information in every h5mu before computing any metric when --Perform_perturbation is set. - run_add_umap.sh: correct the header (no SBATCH_MAIL_USER; create the log directory on the manual path); search_nature.py: generic data-source text. - Docs: CHANGELOG entries; README notes that program plots without perturbation results need PDF/SVG output and that custom gene sets for mouse data must use human symbols; Human ortholog column documented. - Tests for the download handling, organism validation, and the early guide check. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Evaluation guide pre-check: if the --prog_key modality is not in the h5mu, say so instead of reporting missing guide information. - orthologs.py: remove the temporary file if writing the cache fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
load_ortholog_table aborted the run if writing the cache failed (e.g. a read-only shared Resources/ directory), even though the download worked, and every later run downloaded again and failed the same way. Now the write error is logged as a warning and the in-memory table is used, as get_geneset already does for gene-set caches. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stage 1 inference takes --species (human | mouse); evaluation, U-test calibration, the Excel summary and annotation prep called the same setting --organism. All CLIs now use --species, along with their SLURM runners, tests and docs. Breaking: --organism is no longer accepted. Python function keywords (organism= in the enrichment and ortholog code) and the "organism" key in the annotation program JSON are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Input validation belongs in the perturbNMF-runner skill's validation step, not in the pipeline. Drops _h5mu_has_uns_key, the loop that checked every h5mu for uns[guide_targets] (and the --prog_key modality) before any metric ran, and its two tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
validate_data.py only checked the inference input. --stage evaluation checks the cNMF h5mu files an evaluation run will read (built from --out_dir/--run_name/--K/--sel_threshs, as the pipeline does), by key existence with h5py so it takes seconds on large files: the data and program modalities, varm['loadings'], obs[categorical_key] when --Perform_categorical/--Perform_perturbation is used, an optional gene-name column, and with --Perform_perturbation the guide names, targets and assignment. Without it, it reports whether the data has guides. --stage inference (default) is unchanged. 02-evaluation.md runs it before the script is generated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets the pipeline run on scRNA-seq data without perturbations (no guide information) and on mouse data. Human perturbation runs are unchanged.
This is #20 by @OlgaPushkarev, rebuilt on current code. Her 4 commits are kept as-is (authorship preserved), with conflicts resolved. Four commits follow on top of them. Supersedes #20.
Stacked: based on #22 (
ruff-format-v2), which is based onremove-lab-content-agent-skill. GitHub retargets this PR tomainas those merge.From #20 (see #20 for the full description)
--no_perturbation/--no_trait; program analysis and the Excel summary run withoutuns['guide_targets'](the Targets Summary sheet is skipped).src/Stage2_Evaluation/A_Metrics/src/orthologs.py) instead of upper-casing symbols; aHuman orthologcolumn in the Excel summary; trait (GWAS) enrichment skipped for mouse.ensure_umapreusesobsm['UMAP']/obsm['PCA'].Conflict resolution
cNMF_excel_summary.py: kept Support non-perturbed scRNA-seq and mouse data #20'shas_guideslogic, but calledCompile_Target_Summary_sheet/Compile_Summary_sheetwithconditions=(mainrenamed it fromSample=, which would otherwise raise a TypeError).02-evaluation.md: keptmain's--balancedrows and Support non-perturbed scRNA-seq and mouse data #20's--organism/--ortholog_tablerows.CHANGELOG.md: dropped the stale Remove remaining lab- and study-specific content #18 entry about annotation runners that no longer exist.New commits
load_ortholog_tableno longer aborts when the cache can't be written (e.g. a read-only sharedResources/). It logs a warning and uses the downloaded table, likeget_genesetdoes. New test included; it fails on the old code.--organismrenamed--species, matching Stage 1 inference, in evaluation, U-test calibration, the Excel summary andprepare_programs.py, plus their SLURM runners, tests and docs. Python function keywords (organism=) and the"organism"key in the annotation JSON are unchanged._h5mu_has_uns_key) and its two tests. Input validation moves to the skill.validate_data.py --stage evaluationchecks the h5mu files an evaluation run will read (modalities,varm['loadings'], the condition column, an optional gene-name column, and with--Perform_perturbationthe guide names/targets/assignment), by key existence only.02-evaluation.mdruns it before generating the script.--stage inference(default) output is unchanged.Testing
tests/Script/test_non_perturbed.py,tests/Script/Stage2_Evaluation/test_orthologs.py,tests/Script/Stage3_Interpretation/A_Plotting/Program/test_ensure_umap.py: 24 passed. That's Support non-perturbed scRNA-seq and mouse data #20's 25, minus the 2 removed pre-check tests, plus the new cache test.--prog_key, missing K, missing condition column): expected exit codes and messages. On a real 60-program run it takes under a second.--helpof the four renamed CLIs shows--species.ruff format --check .andtools/check_no_lab_specific_content.pypass.🤖 Generated with Claude Code