Full eeg leaderboards - #30
Merged
Merged
Conversation
Full archive. docs/leaderboard_full.md ranks every estimator with a resample-0 result on all 100 datasets of the Multiverse archive paper's full-archive comparison: the datasets on which all 17 of its classifiers completed, now fixed in results/multiverse/paper_datasets.txt. The 17 ranked are exactly the paper's, with HC2 first at 6.31 and MRHydra second at 6.86 as in the paper. Below the table, the twelve core-table estimators that have not finished those 100 are listed with what each is missing, and pending_full.csv holds the same 508 gaps as estimator,dataset rows to queue runs from. EEG. docs/leaderboard_eeg.md does the same over the 26 multivariate datasets of the EEG archive (eeg2026 less the univariate EpilepticSeizures and Sleep): 14 estimators ranked, 13 pending, 188 gaps in pending_eeg.csv. The averaged results already in results/eeg, which include the EEG-specific CSP-SVM, R-KNN and R-MDM, follow as a separate table, marked as not directly comparable. Both are linked from the README and built by python -m multiverse.experiments.tables, with sortable HTML versions beside the core one. To support them: - Result files now hold every Multiverse dataset an estimator has a resample-0 result for, not only Multiverse-core: 6041 rows added, none changed. The core and UEA tables are unchanged, since they select their datasets. - CIF-500 and DrCIF-500, the configurations the paper reports as CIF and DrCIF, are added. They are withheld from the core table, with the reason stated there, so it still reports the default CIF and DrCIF. - ingest scores a test split that is missing a class. tsml-eval raises on log loss and AUROC for those, so the two are computed with the full label set; the KERAAL multiclass problems in the 100 need it. - leaderboard() gains core_sections, to leave the core table's held-out lists off other collections' pages, and extra_html. - A gap outside Multiverse-core with no recorded reason is now reported as not run outside Multiverse-core, which is what it is, rather than as reason not recorded. This changes the UEA page's missing list, not its table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A one-line menu under the opening description links the Multiverse-core, full-archive and EEG leaderboards, replacing the list that sat below the core table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Checklist
results/schema.md.