Seed novel's internal validation split - #46
Merged
Conversation
The competition batches s1d2/s3d7 aren't in bmmc_multiome, so both multiome datasets fall through to an unseeded np.random.choice and pick different validation batches on every run. Add --seed and log the pick.
This comment was marked as outdated.
This comment was marked as outdated.
8 tasks
…tch-fallback # Conflicts: # CHANGELOG.md # src/methods/novel/novel_train/config.vsh.yaml # src/methods/novel/novel_train/script.py
…k' into fix/novel-unseeded-batch-fallback # Conflicts: # CHANGELOG.md
…tch-fallback # Conflicts: # CHANGELOG.md
This comment was marked as outdated.
This comment was marked as outdated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
novel_trainreproduces the competition's phase-1 split by name, with a fallback:I'd assumed the fallback was dormant. It isn't --
bmmc_multiomedoesn't contain either batch:So on GEX2ATAC and ATAC2GEX -- half our datasets -- the validation batch is drawn unseeded:
That's the split
train_and_valid()uses to decide which checkpoint to keep, so which modelnovelships changes run to run, and there's no way to pin it. Added a--seedargument (default 1) and a line logging the chosen batches, since "which batches were held out" is otherwise invisible in the logs.Deliberately not touching the hardcoded
{'s1d2', 's3d7'}-- that's faithful to the original submission and does the right thing on CITE.Note this changes which batch is held out on the multiome datasets (whatever the RNG happened to give, to
s2d1), sonovel's multiome scores will shift slightly. That seemed better than leaving them unpinnable.Part of a series of PRs coming out of a pre-run review of the benchmark.
Checklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!