diff --git a/README.md b/README.md index ae01abd..65565ce 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ The **Multiverse** is an expanded archive for multivariate time series classific datasets from the original UEA MTSC archive, newer MTSC collections, donated standalone datasets, and associated benchmark results into a single open repository. -**Leaderboards:** [Multiverse-core](#multiverse-core-leaderboard) (66 datasets) · +**Leaderboards:** [Multiverse-core](#multiverse-core-leaderboard) (65 datasets) · [Full archive](docs/leaderboard_full.md) (the paper's 100 datasets) · [EEG](docs/leaderboard_eeg.md) (26 datasets) @@ -37,40 +37,40 @@ The current paper version describes: - 133 unique MTSC problems - 147 released datasets when preprocessing variants are included -- a curated 66 dataset subset, **Multiverse-core (MV-core)**, for algorithm benchmarking +- a curated 65 dataset subset, **Multiverse-core (MV-core)**, for algorithm benchmarking ### Multiverse-core leaderboard | # | Estimator | Accuracy rank | Accuracy | Balanced accuracy | AUROC | F1 | Log loss ↓ | Sensitivity | Specificity | |---|---|---|---|---|---|---|---|---|---| -| 1 | HC2 | **7.52** | **0.7936** | **0.7528** | **0.8949** | **0.7337** | **0.5307** | 0.7499 | **0.8018** | -| 2 | MRHydra | 9.02 | 0.7806 | 0.7503 | 0.8039 | 0.7293 | 7.9077 | **0.7584** | 0.7794 | -| 3 | RDST | 9.34 | 0.7746 | 0.7371 | 0.7933 | 0.7113 | 8.1235 | 0.7199 | 0.7924 | -| 4 | RIST | 9.84 | 0.7750 | 0.7428 | 0.8685 | 0.7196 | 0.6087 | 0.7424 | 0.7758 | -| 5 | CIF | 10.19 | 0.7782 | 0.7459 | 0.8847 | 0.7273 | 0.6376 | 0.7473 | 0.7799 | -| 6 | DrCIF | 10.36 | 0.7731 | 0.7401 | 0.8743 | 0.7202 | 0.6370 | 0.7396 | 0.7752 | -| 7 | QUANT | 10.87 | 0.7667 | 0.7370 | 0.8693 | 0.7177 | 0.7198 | 0.7476 | 0.7594 | -| 8 | Arsenal | 10.99 | 0.7683 | 0.7309 | 0.8425 | 0.7096 | 3.5868 | 0.7293 | 0.7777 | -| 9 | ROCKET | 11.14 | 0.7698 | 0.7328 | 0.7899 | 0.7088 | 8.2968 | 0.7245 | 0.7811 | -| 10 | LITETime-MV | 11.16 | 0.7505 | 0.7274 | 0.8511 | 0.6864 | 1.3223 | 0.7113 | 0.7707 | -| 11 | STSF | 11.23 | 0.7735 | 0.7486 | 0.8722 | 0.7188 | 0.6639 | 0.7417 | 0.7862 | -| 12 | H-InceptionTime | 11.76 | 0.7418 | 0.7179 | 0.8460 | 0.6881 | 1.3451 | 0.7233 | 0.7428 | -| 13 | ConvTran | 12.95 | 0.7485 | 0.7152 | 0.8545 | 0.6892 | 0.8930 | 0.7251 | 0.7401 | -| 14 | Catch22 | 13.09 | 0.7495 | 0.7189 | 0.8622 | 0.6977 | 0.6971 | 0.7263 | 0.7459 | -| 15 | PatchMTSC | 13.16 | 0.7478 | 0.6996 | 0.8276 | 0.6723 | 0.7796 | 0.7018 | 0.7425 | -| 16 | DisjointCNN | 13.44 | 0.7267 | 0.7021 | 0.8267 | 0.6662 | 2.1025 | 0.6857 | 0.7344 | -| 17 | STC | 13.84 | 0.7527 | 0.7109 | 0.8633 | 0.6843 | 0.6397 | 0.7072 | 0.7629 | -| 18 | TSF | 14.06 | 0.7434 | 0.7142 | 0.8570 | 0.6918 | 0.9812 | 0.7105 | 0.7536 | -| 19 | TDE | 15.01 | 0.7266 | 0.6811 | 0.8344 | 0.6488 | 0.8411 | 0.6800 | 0.7351 | -| 20 | TS2Vec | 15.41 | 0.7203 | 0.6798 | 0.8005 | 0.6556 | 0.7806 | 0.6886 | 0.7154 | -| 21 | Summary | 16.66 | 0.6886 | 0.6585 | 0.8135 | 0.6357 | 0.9475 | 0.6661 | 0.6901 | -| 22 | XCM | 16.78 | 0.6707 | 0.6357 | 0.7914 | 0.5827 | 2.1902 | 0.6233 | 0.6824 | -| 23 | TimesNet | 17.22 | 0.7039 | 0.6685 | 0.8236 | 0.6437 | 1.2796 | 0.6824 | 0.6968 | -| 24 | TimesURL | 17.28 | 0.6973 | 0.6538 | 0.7828 | 0.6099 | 0.9919 | 0.6345 | 0.7050 | -| 25 | Dummy | 22.66 | 0.3802 | 0.3105 | 0.5000 | 0.1804 | 1.3681 | 0.3192 | 0.3882 | - -Average over the 58 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. +| 1 | HC2 | **7.40** | **0.7938** | **0.7543** | **0.8951** | **0.7378** | **0.5316** | 0.7570 | **0.7987** | +| 2 | MRHydra | 8.77 | 0.7838 | 0.7529 | 0.8074 | 0.7336 | 7.7909 | **0.7609** | 0.7828 | +| 3 | RDST | 9.18 | 0.7752 | 0.7396 | 0.7968 | 0.7179 | 8.1012 | 0.7289 | 0.7891 | +| 4 | RIST | 9.82 | 0.7742 | 0.7430 | 0.8677 | 0.7213 | 0.6123 | 0.7469 | 0.7724 | +| 5 | CIF | 10.16 | 0.7775 | 0.7463 | 0.8843 | 0.7293 | 0.6414 | 0.7521 | 0.7767 | +| 6 | DrCIF | 10.37 | 0.7722 | 0.7402 | 0.8734 | 0.7216 | 0.6413 | 0.7437 | 0.7718 | +| 7 | Arsenal | 10.80 | 0.7709 | 0.7336 | 0.8469 | 0.7145 | 3.6382 | 0.7340 | 0.7792 | +| 8 | QUANT | 10.92 | 0.7651 | 0.7362 | 0.8678 | 0.7177 | 0.7263 | 0.7506 | 0.7554 | +| 9 | ROCKET | 11.04 | 0.7701 | 0.7345 | 0.7927 | 0.7134 | 8.2862 | 0.7315 | 0.7781 | +| 10 | LITETime-MV | 11.24 | 0.7483 | 0.7260 | 0.8493 | 0.6851 | 1.3395 | 0.7129 | 0.7668 | +| 11 | STSF | 11.41 | 0.7699 | 0.7447 | 0.8700 | 0.7145 | 0.6742 | 0.7379 | 0.7827 | +| 12 | H-InceptionTime | 11.89 | 0.7379 | 0.7137 | 0.8434 | 0.6837 | 1.3662 | 0.7192 | 0.7389 | +| 13 | Catch22 | 12.90 | 0.7539 | 0.7224 | 0.8680 | 0.7027 | 0.6972 | 0.7290 | 0.7507 | +| 14 | ConvTran | 13.12 | 0.7446 | 0.7110 | 0.8520 | 0.6846 | 0.9070 | 0.7219 | 0.7356 | +| 15 | PatchMTSC | 13.31 | 0.7443 | 0.6959 | 0.8247 | 0.6682 | 0.7905 | 0.6996 | 0.7381 | +| 16 | DisjointCNN | 13.64 | 0.7224 | 0.6975 | 0.8239 | 0.6611 | 2.1344 | 0.6814 | 0.7299 | +| 17 | STC | 13.86 | 0.7518 | 0.7106 | 0.8637 | 0.6853 | 0.6428 | 0.7104 | 0.7599 | +| 18 | TSF | 14.15 | 0.7419 | 0.7138 | 0.8558 | 0.6926 | 0.9914 | 0.7141 | 0.7498 | +| 19 | TDE | 14.94 | 0.7272 | 0.6843 | 0.8379 | 0.6601 | 0.8450 | 0.6919 | 0.7304 | +| 20 | TS2Vec | 15.41 | 0.7192 | 0.6790 | 0.7995 | 0.6562 | 0.7869 | 0.6907 | 0.7125 | +| 21 | Summary | 16.52 | 0.6936 | 0.6614 | 0.8194 | 0.6393 | 0.9505 | 0.6648 | 0.6979 | +| 22 | XCM | 16.80 | 0.6691 | 0.6359 | 0.7915 | 0.5859 | 2.1699 | 0.6299 | 0.6769 | +| 23 | TimesURL | 17.23 | 0.6975 | 0.6565 | 0.7833 | 0.6206 | 0.9962 | 0.6456 | 0.6998 | +| 24 | TimesNet | 17.41 | 0.7001 | 0.6647 | 0.8209 | 0.6399 | 1.2957 | 0.6804 | 0.6919 | +| 25 | Dummy | 22.70 | 0.3748 | 0.3072 | 0.5000 | 0.1836 | 1.3810 | 0.3248 | 0.3774 | + +Average over the 57 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. Rebuilt with `python -m multiverse.experiments.tables`, which also writes a sortable diff --git a/docs/classifiers.md b/docs/classifiers.md index 470ec09..bd9d56a 100644 --- a/docs/classifiers.md +++ b/docs/classifiers.md @@ -260,14 +260,17 @@ It also seeds from ``random_state`` directly rather than through ``check_random_state``; that one is not a fidelity constraint, since TSLib simply sets a global seed in ``run.py``, and it could be unified with the other two ports. -TimesNet reproduces TSLib's learning rate schedule, which the classification loop -applies every five epochs. With the default ``lr_adjust="type1"`` the rate becomes -``learning_rate * 0.5 ** (epoch - 1)`` at epochs 5, 10, 15 and so on, so from the -published ``learning_rate=0.001`` it falls to 6.3e-5 by epoch 5 and 6.1e-8 by epoch 15: -over a default 30 epoch run the model is effectively frozen well before the end. Pass -``lr_adjust=None`` to train at a constant rate instead. Omitting this schedule, as -earlier versions of this port did, is a materially different optimisation and makes -results incomparable with the published ones. +TimesNet trains at a constant learning rate by default, ``lr_adjust=None``, which +departs from TSLib. TSLib defaults to ``lradj="type1"``, applied every five epochs: the +rate becomes ``learning_rate * 0.5 ** (epoch - 1)`` at epochs 5, 10, 15 and so on, so +from the published ``learning_rate=0.001`` it falls to 6.3e-5 by epoch 5 and 6.1e-8 by +epoch 15, and over a 30 epoch run the model is effectively frozen well before the end. +That is harmless in TSLib only because it selects the retained epoch on the test set, +keeping an early epoch from before the collapse. This port selects on a held-out split +of the training data, so with the schedule on it keeps a model that has stopped +learning: on ERing, 0.933 without the schedule against 0.578 with it, at otherwise +identical settings. Pass ``lr_adjust="type1"`` to reproduce TSLib. The published +TimesNet results here are from the default, constant rate. ### Equivalence testing diff --git a/docs/datasets.md b/docs/datasets.md index 3fe6424..165a063 100644 --- a/docs/datasets.md +++ b/docs/datasets.md @@ -83,7 +83,7 @@ The project uses the archive collections exposed by aeon: ```python from aeon.datasets.tsc_datasets import multiverse_core, multiverse2026, eeg2026 -print(len(multiverse_core)) # 66 +print(len(multiverse_core)) # 65 print(len(multiverse2026)) # 133 print(len(eeg2026)) # 28 ``` @@ -94,6 +94,11 @@ very simple and zero-information datasets, and has a useful spread of dataset si series lengths. The current benchmark results use this core list unless stated otherwise. +BenzeneConcentration_disc has been removed from Multiverse-core, since the version the +results were run on has a channel that is a deterministic function of the target +([version 2](https://zenodo.org/records/21871727) drops it). aeon versions that +predate the change still list it, and 66 datasets. It stays in `multiverse2026`. + The EEG collection is a separate classification archive used for EEG-specific experiments. It is based on [aeon-neuro](https://github.com/aeon-toolkit/aeon-neuro). diff --git a/docs/leaderboard_eeg.md b/docs/leaderboard_eeg.md index e356583..0410bb2 100644 --- a/docs/leaderboard_eeg.md +++ b/docs/leaderboard_eeg.md @@ -58,8 +58,9 @@ datasets, so are not ranked above. The same gaps are in ## EEG archive study results These are the averaged results held in [`results/eeg`](../results/eeg), for twelve -estimators including the EEG-specific CSP-SVM, R-KNN and R-MDM. They come from a separate -study, not the resample-0 runs above, so the two tables are not directly comparable. +estimators including the EEG-specific CSP-SVM, R-KNN and R-MDM. They come from the +[EEG archive paper](https://openreview.net/pdf?id=oPQpQlsfD0), not the resample-0 runs +above, so the two tables are not directly comparable. Their 26 datasets also differ: they include the univariate Sleep and leave out FeedbackButton. diff --git a/multiverse/experiments/tables.py b/multiverse/experiments/tables.py index 5256552..6376ef0 100644 --- a/multiverse/experiments/tables.py +++ b/multiverse/experiments/tables.py @@ -41,6 +41,7 @@ import base64 import io +import re from datetime import date from html import escape from pathlib import Path @@ -369,6 +370,13 @@ def _missing_by_estimator(frames, estimators, common, datasets): "cannot complete the archive within the walltime available: exceeded the " "limit on BIDMC32HR_disc, with no result recorded for BIDMC32SpO2_disc" ), + "MUSE": ( + "cannot complete the archive as implemented: on STEW and Skoda its word " + "bag outgrows the 32-bit sparse indices scikit-learn's classifier " + "accepts, which no memory allocation fixes, and on USCActivity its chi2 " + "selection needs 287 GiB. MotorImagery exhausted 16 GB and awaits a " + "rerun at the configuration of its other results" + ), "CIF-500": ( "the 500-tree configuration run for the Multiverse archive paper's " "full-archive benchmark, where it is reported as CIF. This table reports " @@ -392,7 +400,7 @@ def _missing_by_estimator(frames, estimators, common, datasets): # Datasets held out of the collection, and why. Distinct from the datasets an # individual estimator is missing: these are ones no amount of scheduling will # close, so leaving them in the denominator only makes the scored fraction look -# like a queue that is still draining. +# like a queue that is still draining. A URL in a reason is rendered as a link. DEFERRED_DATASETS = { "AustraliaRainfall_disc": ( "112186 cases, and three estimators fail on it for reasons compute cannot " @@ -404,15 +412,26 @@ def _missing_by_estimator(frames, estimators, common, datasets): "the series are length 8 and MRHydra requires at least 9, so the dataset " "cannot complete while MRHydra is a column" ), + "BenzeneConcentration_disc": ( + "removed from Multiverse-core. The results here are on version 1, whose " + "PT08.S2 channel is a deterministic function of the target; version 2 " + "drops it, on the advice of the original UCI Air Quality donors: " + "https://zenodo.org/records/21871727" + ), } +def _link_urls(text: str) -> str: + """Turn each https URL in already escaped text into a link.""" + return re.sub(r"(https://[^\s<]+)", r'\1', text) + + def _deferred_html() -> str: """Name the datasets held out of the collection, and why.""" if not DEFERRED_DATASETS: return "" items = "".join( - f"
  • {escape(name)} — {escape(reason)}.
    • " + f"
  • {escape(name)} — {_link_urls(escape(reason))}.
    • " for name, reason in DEFERRED_DATASETS.items() ) return ( diff --git a/results/multiverse/LiteTIME/LiteTIME_accuracy.csv b/results/multiverse/LiteTIME/LiteTIME_accuracy.csv index 46a0c64..1ecce15 100644 --- a/results/multiverse/LiteTIME/LiteTIME_accuracy.csv +++ b/results/multiverse/LiteTIME/LiteTIME_accuracy.csv @@ -7,6 +7,7 @@ AsphaltRegularityCoordinates,0.9893475366178429 AtrialFibrillation,0.13333333333333333 AustraliaRainfall_disc,0.7788107568478193 AutomotiveRoadTrials,0.7922077922077922 +BasicMotions,1.0 BeijingPM10Quality_disc,0.8195324881141046 BeijingPM25Quality_disc,0.8785657686212361 BenzeneConcentration_disc,0.9101297695138485 @@ -25,6 +26,7 @@ Epilepsy,0.9927536231884058 EthanolConcentration,0.21673003802281368 EyesOpenShut,0.42857142857142855 FaceDetection,0.6677071509648127 +FingerMovements,0.57 FordChallenge,0.7440033085194375 HandMovementDirection,0.35135135135135137 Handwriting,0.5988235294117648 @@ -52,6 +54,7 @@ PhotoStimulation,0.2222222222222222 RacketSports,0.875 STEW,0.6285072951739619 SelfRegulationSCP1,0.7952218430034129 +SelfRegulationSCP2,0.5166666666666667 Skoda,0.9639193491333569 SpokenArabicDigits,0.9931787175989086 StandWalkJump,0.4666666666666667 diff --git a/results/multiverse/LiteTIME/LiteTIME_auroc.csv b/results/multiverse/LiteTIME/LiteTIME_auroc.csv index 22ecbe1..85589dd 100644 --- a/results/multiverse/LiteTIME/LiteTIME_auroc.csv +++ b/results/multiverse/LiteTIME/LiteTIME_auroc.csv @@ -7,6 +7,7 @@ AsphaltRegularityCoordinates,0.9969922678584096 AtrialFibrillation,0.3733333333333333 AustraliaRainfall_disc,0.8600906211975378 AutomotiveRoadTrials,0.8566243194192378 +BasicMotions,1.0 BeijingPM10Quality_disc,0.8716951140762138 BeijingPM25Quality_disc,0.9306582924184476 BenzeneConcentration_disc,0.9668722682717931 @@ -25,6 +26,7 @@ Epilepsy,0.9997889002947116 EthanolConcentration,0.5087754852573134 EyesOpenShut,0.3900226757369614 FaceDetection,0.7264014811360014 +FingerMovements,0.6182472989195679 FordChallenge,0.853777545661815 HandMovementDirection,0.6642613723122198 Handwriting,0.9410721040377091 @@ -52,6 +54,7 @@ PhotoStimulation,0.3781196581196581 RacketSports,0.9654502871283713 STEW,0.8105854734525703 SelfRegulationSCP1,0.9244012673562576 +SelfRegulationSCP2,0.5190123456790123 Skoda,0.9978507333547619 SpokenArabicDigits,0.9999843743573092 StandWalkJump,0.5666666666666667 diff --git a/results/multiverse/LiteTIME/LiteTIME_balacc.csv b/results/multiverse/LiteTIME/LiteTIME_balacc.csv index dfabe02..e4abfdd 100644 --- a/results/multiverse/LiteTIME/LiteTIME_balacc.csv +++ b/results/multiverse/LiteTIME/LiteTIME_balacc.csv @@ -7,6 +7,7 @@ AsphaltRegularityCoordinates,0.989501312335958 AtrialFibrillation,0.13333333333333333 AustraliaRainfall_disc,0.423103469854606 AutomotiveRoadTrials,0.7912885662431942 +BasicMotions,1.0 BeijingPM10Quality_disc,0.7767887096835823 BeijingPM25Quality_disc,0.8600541587311511 BenzeneConcentration_disc,0.8996512888796334 @@ -25,6 +26,7 @@ Epilepsy,0.9926470588235294 EthanolConcentration,0.21643356643356643 EyesOpenShut,0.42857142857142855 FaceDetection,0.6677071509648127 +FingerMovements,0.5704281712685074 FordChallenge,0.7706758629262975 HandMovementDirection,0.3726190476190476 Handwriting,0.5970573977871463 @@ -52,6 +54,7 @@ PhotoStimulation,0.21919191919191916 RacketSports,0.8851744186046512 STEW,0.6285072951739619 SelfRegulationSCP1,0.795871773366881 +SelfRegulationSCP2,0.5166666666666666 Skoda,0.9628853708859495 SpokenArabicDigits,0.9931818181818182 StandWalkJump,0.4666666666666666 diff --git a/results/multiverse/LiteTIME/LiteTIME_f1.csv b/results/multiverse/LiteTIME/LiteTIME_f1.csv index 0c28c91..b1ccc80 100644 --- a/results/multiverse/LiteTIME/LiteTIME_f1.csv +++ b/results/multiverse/LiteTIME/LiteTIME_f1.csv @@ -7,6 +7,7 @@ AsphaltRegularityCoordinates,0.9893048128342246 AtrialFibrillation,0.10822510822510822 AustraliaRainfall_disc,0.7687181575723717 AutomotiveRoadTrials,0.6521739130434783 +BasicMotions,1.0 BeijingPM10Quality_disc,0.6837903505727178 BeijingPM25Quality_disc,0.8019386106623586 BenzeneConcentration_disc,0.8575813382443217 @@ -25,6 +26,7 @@ Epilepsy,0.9927478549282573 EthanolConcentration,0.20782350943667244 EyesOpenShut,0.42857142857142855 FaceDetection,0.6762510367708046 +FingerMovements,0.5742574257425742 FordChallenge,0.721213031076415 HandMovementDirection,0.3511120103671958 Handwriting,0.5756336373055114 @@ -52,6 +54,7 @@ PhotoStimulation,0.2236983360171766 RacketSports,0.8749272515640915 STEW,0.43442973088423753 SelfRegulationSCP1,0.8275862068965517 +SelfRegulationSCP2,0.47904191616766467 Skoda,0.9637774526594892 SpokenArabicDigits,0.9931705004917735 StandWalkJump,0.3904761904761905 diff --git a/results/multiverse/LiteTIME/LiteTIME_logloss.csv b/results/multiverse/LiteTIME/LiteTIME_logloss.csv index 2f29f2a..4659003 100644 --- a/results/multiverse/LiteTIME/LiteTIME_logloss.csv +++ b/results/multiverse/LiteTIME/LiteTIME_logloss.csv @@ -7,6 +7,7 @@ AsphaltRegularityCoordinates,0.0633709328876837 AtrialFibrillation,3.830235855233876 AustraliaRainfall_disc,0.5082520114132473 AutomotiveRoadTrials,0.7647017220924607 +BasicMotions,0.0031483720772545803 BeijingPM10Quality_disc,1.2811467872585638 BeijingPM25Quality_disc,1.0247847706659101 BenzeneConcentration_disc,0.24977842997786107 @@ -25,6 +26,7 @@ Epilepsy,0.026125259800652865 EthanolConcentration,2.8835702178775393 EyesOpenShut,1.7713254378257652 FaceDetection,1.4778285212115687 +FingerMovements,1.4255986332995871 FordChallenge,0.6414512694285133 HandMovementDirection,1.605375679310031 Handwriting,1.5516101478469824 @@ -52,6 +54,7 @@ PhotoStimulation,3.861648651725385 RacketSports,0.4355536405511637 STEW,2.914516308229351 SelfRegulationSCP1,0.8235472491231484 +SelfRegulationSCP2,1.01696868038982 Skoda,0.2011567722867202 SpokenArabicDigits,0.01492038685305996 StandWalkJump,2.9512630582435757 diff --git a/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv b/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv index 3e52cd3..11cc91d 100644 --- a/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv +++ b/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv @@ -7,6 +7,7 @@ AsphaltRegularityCoordinates,1.0 AtrialFibrillation,0.13333333333333333 AustraliaRainfall_disc,0.7788107568478193 AutomotiveRoadTrials,0.7894736842105263 +BasicMotions,1.0 BeijingPM10Quality_disc,0.6755829903978052 BeijingPM25Quality_disc,0.8132372214941023 BenzeneConcentration_disc,0.8720349563046192 @@ -25,6 +26,7 @@ Epilepsy,0.9927536231884058 EthanolConcentration,0.21673003802281368 EyesOpenShut,0.42857142857142855 FaceDetection,0.6940976163450624 +FingerMovements,0.5918367346938775 FordChallenge,0.8788876692279546 HandMovementDirection,0.35135135135135137 Handwriting,0.5988235294117648 @@ -52,6 +54,7 @@ PhotoStimulation,0.2222222222222222 RacketSports,0.875 STEW,0.28535353535353536 SelfRegulationSCP1,0.9863013698630136 +SelfRegulationSCP2,0.4444444444444444 Skoda,0.9639193491333569 SpokenArabicDigits,0.9931787175989086 StandWalkJump,0.4666666666666667 diff --git a/results/multiverse/LiteTIME/LiteTIME_specificity.csv b/results/multiverse/LiteTIME/LiteTIME_specificity.csv index 1c0cb2e..962a8f7 100644 --- a/results/multiverse/LiteTIME/LiteTIME_specificity.csv +++ b/results/multiverse/LiteTIME/LiteTIME_specificity.csv @@ -7,6 +7,7 @@ AsphaltRegularityCoordinates,0.979002624671916 AtrialFibrillation,0.13333333333333333 AustraliaRainfall_disc,0.7788107568478193 AutomotiveRoadTrials,0.7931034482758621 +BasicMotions,1.0 BeijingPM10Quality_disc,0.8779944289693593 BeijingPM25Quality_disc,0.9068710959681999 BenzeneConcentration_disc,0.9272676214546476 @@ -25,6 +26,7 @@ Epilepsy,0.9927536231884058 EthanolConcentration,0.21673003802281368 EyesOpenShut,0.42857142857142855 FaceDetection,0.641316685584563 +FingerMovements,0.5490196078431373 FordChallenge,0.6624640566246406 HandMovementDirection,0.35135135135135137 Handwriting,0.5988235294117648 @@ -52,6 +54,7 @@ PhotoStimulation,0.2222222222222222 RacketSports,0.875 STEW,0.9716610549943884 SelfRegulationSCP1,0.6054421768707483 +SelfRegulationSCP2,0.5888888888888889 Skoda,0.9639193491333569 SpokenArabicDigits,0.9931787175989086 StandWalkJump,0.4666666666666667 diff --git a/results/multiverse/MUSE/MUSE_accuracy.csv b/results/multiverse/MUSE/MUSE_accuracy.csv new file mode 100644 index 0000000..3b884be --- /dev/null +++ b/results/multiverse/MUSE/MUSE_accuracy.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4883720930232558 +AppliancesEnergy_disc,0.7857142857142857 +ArticularyWordRecognition,0.9933333333333333 +AsphaltObstaclesCoordinates,0.8081841432225064 +AsphaltRegularityCoordinates,0.9653794940079894 +AtrialFibrillation,0.13333333333333333 +AutomotiveRoadTrials,0.8051948051948052 +BIDMC32HR_disc,0.8257607336390163 +BIDMC32SpO2_disc,0.6277615673197166 +BeijingPM10Quality_disc,0.7759508716323297 +BeijingPM25Quality_disc,0.8181458003169572 +BenzeneConcentration_disc,0.7023048615146232 +Blink,0.8955555555555555 +BoneIntensitiesAgeGroup,0.8337078651685393 +BoneProbAgeGroup,0.6629213483146067 +CharacterTrajectories,0.9902506963788301 +CounterMovementJump,0.8603351955307262 +Cricket,1.0 +CrowdSourced,0.7285563007412637 +DuckDuckGeese,0.48 +ERing,0.9592592592592593 +EigenWorms,0.9465648854961832 +EmoPain,0.8732394366197183 +Epilepsy,1.0 +EthanolConcentration,0.5247148288973384 +EyesOpenShut,0.4523809523809524 +FaceDetection,0.6637343927355278 +FordChallenge,0.9029500964984836 +HandMovementDirection,0.3108108108108108 +Handwriting,0.32 +Heartbeat,0.7414634146341463 +HouseholdPowerConsumption1_disc,0.8454810495626822 +HouseholdPowerConsumption2_disc,0.7346938775510204 +IEEEPPG_disc,0.5768072289156626 +IRDS-SFL,0.8793103448275862 +JapaneseVowels,0.9351351351351351 +KERAAL-RTK,0.7857142857142857 +KIMORE-PR-C,0.7142857142857143 +KINECAL-QSEO,0.9411764705882353 +LSST,0.610705596107056 +Libras,0.9111111111111111 +Locust2022,0.92209760533495 +LowCost,0.575 +MindReading,0.5834609494640123 +MotionSenseHAR,1.0 +NATOPS,0.9111111111111111 +PEMS-SF,0.9826589595375722 +PenDigits,0.9519725557461407 +PhonemeSpectra,0.330450342976439 +PhotoStimulation,0.3888888888888889 +RacketSports,0.8421052631578947 +SelfRegulationSCP1,0.78839590443686 +SpokenArabicDigits,0.9872669395179627 +StandWalkJump,0.4 +TactileTextureRecognition,1.0 +Tiselac,0.7450455005055612 +UCDHE-Rowing-MC,0.7681818181818182 +UCIActivity,0.978758850478967 +UIPRMD-DS-C,0.9444444444444444 +UWaveGestureLibrary,0.903125 +WISDM,0.8547421286459339 diff --git a/results/multiverse/MUSE/MUSE_auroc.csv b/results/multiverse/MUSE/MUSE_auroc.csv new file mode 100644 index 0000000..5a5ab8b --- /dev/null +++ b/results/multiverse/MUSE/MUSE_auroc.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.6076523656776263 +AppliancesEnergy_disc,0.7830882352941176 +ArticularyWordRecognition,0.9999421296296297 +AsphaltObstaclesCoordinates,0.9515387172272893 +AsphaltRegularityCoordinates,0.9919486415549408 +AtrialFibrillation,0.34666666666666673 +AutomotiveRoadTrials,0.8865698729582577 +BIDMC32HR_disc,0.7800773501997169 +BIDMC32SpO2_disc,0.5776039480285454 +BeijingPM10Quality_disc,0.8203174494002927 +BeijingPM25Quality_disc,0.8790941864766162 +BenzeneConcentration_disc,0.6608079236814695 +Blink,0.94184 +BoneIntensitiesAgeGroup,0.940059101141633 +BoneProbAgeGroup,0.7976467667147088 +CharacterTrajectories,0.9997913004585031 +CounterMovementJump,0.994418493654445 +Cricket,1.0 +CrowdSourced,0.7390344844084542 +DuckDuckGeese,0.81 +ERing,0.9983045267489712 +EigenWorms,0.9924324282870718 +EmoPain,0.952765214234431 +Epilepsy,1.0 +EthanolConcentration,0.760443601442617 +EyesOpenShut,0.5011337868480725 +FaceDetection,0.7181622111907193 +FordChallenge,0.9454127678185622 +HandMovementDirection,0.520910201418676 +Handwriting,0.7839683142398908 +Heartbeat,0.761379800853485 +HouseholdPowerConsumption1_disc,0.9224167831583286 +HouseholdPowerConsumption2_disc,0.7910207485218832 +IEEEPPG_disc,0.760747418579442 +IRDS-SFL,0.9660326086956521 +JapaneseVowels,0.9956514105717534 +KERAAL-RTK,0.875 +KIMORE-PR-C,0.6666666666666667 +KINECAL-QSEO,0.75 +LSST,0.8737265464075589 +Libras,0.9937830687830687 +Locust2022,0.8566740239657205 +LowCost,0.5825666666666667 +MindReading,0.8571976098519972 +MotionSenseHAR,1.0 +NATOPS,0.9863703703703705 +PEMS-SF,1.0 +PenDigits,0.9977061829257353 +PhonemeSpectra,0.8850873374441747 +PhotoStimulation,0.5125844525844526 +RacketSports,0.9670583784576119 +SelfRegulationSCP1,0.9234460907650732 +SpokenArabicDigits,0.9996157938332835 +StandWalkJump,0.6133333333333333 +TactileTextureRecognition,1.0 +Tiselac,0.937404980888222 +UCDHE-Rowing-MC,0.9284418942433648 +UCIActivity,0.9987077359827623 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.9918638392857144 +WISDM,0.9677509985252801 diff --git a/results/multiverse/MUSE/MUSE_balacc.csv b/results/multiverse/MUSE/MUSE_balacc.csv new file mode 100644 index 0000000..fcd400f --- /dev/null +++ b/results/multiverse/MUSE/MUSE_balacc.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4783549783549783 +AppliancesEnergy_disc,0.6286764705882353 +ArticularyWordRecognition,0.9933333333333333 +AsphaltObstaclesCoordinates,0.8070173118722899 +AsphaltRegularityCoordinates,0.9654891111584025 +AtrialFibrillation,0.13333333333333333 +AutomotiveRoadTrials,0.8352994555353902 +BIDMC32HR_disc,0.7712767316516462 +BIDMC32SpO2_disc,0.5031518018340859 +BeijingPM10Quality_disc,0.6652951156046173 +BeijingPM25Quality_disc,0.748049891228548 +BenzeneConcentration_disc,0.5812405933190083 +Blink,0.9015 +BoneIntensitiesAgeGroup,0.8441589912207572 +BoneProbAgeGroup,0.6819892132473416 +CharacterTrajectories,0.9897545996789949 +CounterMovementJump,0.8599811676082862 +Cricket,1.0 +CrowdSourced,0.7286408681506644 +DuckDuckGeese,0.4800000000000001 +ERing,0.9592592592592593 +EigenWorms,0.9401445897098071 +EmoPain,0.5768004950878007 +Epilepsy,1.0 +EthanolConcentration,0.5254662004662005 +EyesOpenShut,0.4523809523809524 +FaceDetection,0.6637343927355278 +FordChallenge,0.8902329439078179 +HandMovementDirection,0.3130952380952381 +Handwriting,0.3096332707269738 +Heartbeat,0.5782361308677098 +HouseholdPowerConsumption1_disc,0.5957076964428305 +HouseholdPowerConsumption2_disc,0.6375397813753978 +IEEEPPG_disc,0.6076248194208804 +IRDS-SFL,0.9085144927536233 +JapaneseVowels,0.9360693379047105 +KERAAL-RTK,0.75 +KIMORE-PR-C,0.8333333333333333 +KINECAL-QSEO,0.5 +LSST,0.3231269878580135 +Libras,0.9111111111111111 +Locust2022,0.6661636908208611 +LowCost,0.575 +MindReading,0.5709854824048824 +MotionSenseHAR,1.0 +NATOPS,0.9111111111111111 +PEMS-SF,0.9805194805194805 +PenDigits,0.9520911161430158 +PhonemeSpectra,0.3304886176295206 +PhotoStimulation,0.3767676767676768 +RacketSports,0.854796511627907 +SelfRegulationSCP1,0.7889991613083589 +SpokenArabicDigits,0.9872727272727273 +StandWalkJump,0.4000000000000001 +TactileTextureRecognition,1.0 +Tiselac,0.5558603282441104 +UCDHE-Rowing-MC,0.774 +UCIActivity,0.9795129896152915 +UIPRMD-DS-C,0.9444444444444444 +UWaveGestureLibrary,0.903125 +WISDM,0.5469251819678482 diff --git a/results/multiverse/MUSE/MUSE_f1.csv b/results/multiverse/MUSE/MUSE_f1.csv new file mode 100644 index 0000000..8bbfd6c --- /dev/null +++ b/results/multiverse/MUSE/MUSE_f1.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4868351286955938 +AppliancesEnergy_disc,0.4 +ArticularyWordRecognition,0.9933217391304349 +AsphaltObstaclesCoordinates,0.8052463497135259 +AsphaltRegularityCoordinates,0.9651474530831099 +AtrialFibrillation,0.14285714285714282 +AutomotiveRoadTrials,0.6938775510204082 +BIDMC32HR_disc,0.8069954149900985 +BIDMC32SpO2_disc,0.24641350210970464 +BeijingPM10Quality_disc,0.5097529258777633 +BeijingPM25Quality_disc,0.6548872180451127 +BenzeneConcentration_disc,0.353386621792175 +Blink,0.8904428904428905 +BoneIntensitiesAgeGroup,0.8328004445105576 +BoneProbAgeGroup,0.6655080435931681 +CharacterTrajectories,0.9902489947710118 +CounterMovementJump,0.864107714663231 +Cricket,1.0 +CrowdSourced,0.780974081458274 +DuckDuckGeese,0.4672681704260651 +ERing,0.9594974243340849 +EigenWorms,0.9471020268415042 +EmoPain,0.8595926563396443 +Epilepsy,1.0 +EthanolConcentration,0.5243639903706636 +EyesOpenShut,0.5660377358490566 +FaceDetection,0.6401457637412694 +FordChallenge,0.8668683812405447 +HandMovementDirection,0.3135662648847233 +Handwriting,0.2919022519335654 +Heartbeat,0.3116883116883117 +HouseholdPowerConsumption1_disc,0.8209565565123326 +HouseholdPowerConsumption2_disc,0.7392787934792032 +IEEEPPG_disc,0.5630180860937252 +IRDS-SFL,0.7666666666666667 +JapaneseVowels,0.9354130977593966 +KERAAL-RTK,0.6666666666666666 +KIMORE-PR-C,0.5 +KINECAL-QSEO,0.0 +LSST,0.5498219290562014 +Libras,0.9075549823785707 +Locust2022,0.44731182795698926 +LowCost,0.5611015490533563 +MindReading,0.5797870838833633 +MotionSenseHAR,1.0 +NATOPS,0.9103071415112115 +PEMS-SF,0.9824808743980351 +PenDigits,0.9520024054867928 +PhonemeSpectra,0.3278863589178971 +PhotoStimulation,0.39258458952856234 +RacketSports,0.8402637201979308 +SelfRegulationSCP1,0.8197674418604651 +SpokenArabicDigits,0.9872510097508786 +StandWalkJump,0.35978835978835977 +TactileTextureRecognition,1.0 +Tiselac,0.7323370340555578 +UCDHE-Rowing-MC,0.7617676725536525 +UCIActivity,0.9784449531297645 +UIPRMD-DS-C,0.9411764705882353 +UWaveGestureLibrary,0.9026008006493242 +WISDM,0.8375996373567506 diff --git a/results/multiverse/MUSE/MUSE_logloss.csv b/results/multiverse/MUSE/MUSE_logloss.csv new file mode 100644 index 0000000..cd3526b --- /dev/null +++ b/results/multiverse/MUSE/MUSE_logloss.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,7.069116989628819 +AppliancesEnergy_disc,1.053546581202111 +ArticularyWordRecognition,0.04082016142241 +AsphaltObstaclesCoordinates,0.8810328961203839 +AsphaltRegularityCoordinates,0.17353552083799653 +AtrialFibrillation,4.715067615866889 +AutomotiveRoadTrials,0.813329748528514 +BIDMC32HR_disc,3.3437674554807133 +BIDMC32SpO2_disc,4.924230850228949 +BeijingPM10Quality_disc,0.8761936463448279 +BeijingPM25Quality_disc,0.670608492852281 +BenzeneConcentration_disc,4.1916377436109435 +Blink,0.7806698865673884 +BoneIntensitiesAgeGroup,0.7048990230005399 +BoneProbAgeGroup,1.4328016454607087 +CharacterTrajectories,0.052086675091941294 +CounterMovementJump,0.40056992504298183 +Cricket,0.004862218089833101 +CrowdSourced,3.464982109597157 +DuckDuckGeese,2.230056666757709 +ERing,0.12199653380002612 +EigenWorms,0.4529916286549567 +EmoPain,0.8685186114796746 +Epilepsy,0.0008558507070488771 +EthanolConcentration,2.6506888301223204 +EyesOpenShut,1.4662233691441156 +FaceDetection,0.9189024487614483 +FordChallenge,0.5423416409078019 +HandMovementDirection,2.6055820224325634 +Handwriting,3.885076945956805 +Heartbeat,1.0939775382653658 +HouseholdPowerConsumption1_disc,0.8020115561394754 +HouseholdPowerConsumption2_disc,1.772488898454323 +IEEEPPG_disc,3.4952157898670486 +IRDS-SFL,0.37713032309585576 +JapaneseVowels,0.21156689680211369 +KERAAL-RTK,1.2524289279724168 +KIMORE-PR-C,2.0318643171264563 +KINECAL-QSEO,0.7109004298330451 +LSST,1.912864122768892 +Libras,0.30119244326696337 +Locust2022,0.4956204710023518 +LowCost,1.6505618940555107 +MindReading,2.382386034042259 +MotionSenseHAR,0.0018436580967774658 +NATOPS,0.22110951726193614 +PEMS-SF,0.053997171960616576 +PenDigits,0.1768523746370157 +PhonemeSpectra,3.9366613612198784 +PhotoStimulation,9.588279298770049 +RacketSports,0.31579178003604425 +SelfRegulationSCP1,1.588516263336786 +SpokenArabicDigits,0.0643341452227117 +StandWalkJump,4.736208769608975 +TactileTextureRecognition,0.0003394988094306201 +Tiselac,1.3711494119839012 +UCDHE-Rowing-MC,1.3749177190618096 +UCIActivity,0.10613006200750534 +UIPRMD-DS-C,0.1519315212204995 +UWaveGestureLibrary,0.4147574020034093 +WISDM,1.1068287241109513 diff --git a/results/multiverse/MUSE/MUSE_sensitivity.csv b/results/multiverse/MUSE/MUSE_sensitivity.csv new file mode 100644 index 0000000..7db7ed4 --- /dev/null +++ b/results/multiverse/MUSE/MUSE_sensitivity.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4883720930232558 +AppliancesEnergy_disc,0.375 +ArticularyWordRecognition,0.9933333333333333 +AsphaltObstaclesCoordinates,0.8081841432225064 +AsphaltRegularityCoordinates,0.972972972972973 +AtrialFibrillation,0.13333333333333333 +AutomotiveRoadTrials,0.8947368421052632 +BIDMC32HR_disc,0.8257607336390163 +BIDMC32SpO2_disc,0.21376281112737922 +BeijingPM10Quality_disc,0.40329218106995884 +BeijingPM25Quality_disc,0.5707732634338138 +BenzeneConcentration_disc,0.26217228464419473 +Blink,0.955 +BoneIntensitiesAgeGroup,0.8337078651685393 +BoneProbAgeGroup,0.6629213483146067 +CharacterTrajectories,0.9902506963788301 +CounterMovementJump,0.8603351955307262 +Cricket,1.0 +CrowdSourced,0.9682203389830508 +DuckDuckGeese,0.48 +ERing,0.9592592592592593 +EigenWorms,0.9465648854961832 +EmoPain,0.8732394366197183 +Epilepsy,1.0 +EthanolConcentration,0.5247148288973384 +EyesOpenShut,0.7142857142857143 +FaceDetection,0.5981838819523269 +FordChallenge,0.8386388583973655 +HandMovementDirection,0.3108108108108108 +Handwriting,0.32 +Heartbeat,0.21052631578947367 +HouseholdPowerConsumption1_disc,0.8454810495626822 +HouseholdPowerConsumption2_disc,0.7346938775510204 +IEEEPPG_disc,0.5768072289156626 +IRDS-SFL,0.9583333333333334 +JapaneseVowels,0.9351351351351351 +KERAAL-RTK,0.5 +KIMORE-PR-C,1.0 +KINECAL-QSEO,0.0 +LSST,0.610705596107056 +Libras,0.9111111111111111 +Locust2022,0.35494880546075086 +LowCost,0.5433333333333333 +MindReading,0.5834609494640123 +MotionSenseHAR,1.0 +NATOPS,0.9111111111111111 +PEMS-SF,0.9826589595375722 +PenDigits,0.9519725557461407 +PhonemeSpectra,0.330450342976439 +PhotoStimulation,0.3888888888888889 +RacketSports,0.8421052631578947 +SelfRegulationSCP1,0.9657534246575342 +SpokenArabicDigits,0.9872669395179627 +StandWalkJump,0.4 +TactileTextureRecognition,1.0 +Tiselac,0.7450455005055612 +UCDHE-Rowing-MC,0.7681818181818182 +UCIActivity,0.978758850478967 +UIPRMD-DS-C,0.8888888888888888 +UWaveGestureLibrary,0.903125 +WISDM,0.8547421286459339 diff --git a/results/multiverse/MUSE/MUSE_specificity.csv b/results/multiverse/MUSE/MUSE_specificity.csv new file mode 100644 index 0000000..aef51dd --- /dev/null +++ b/results/multiverse/MUSE/MUSE_specificity.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4883720930232558 +AppliancesEnergy_disc,0.8823529411764706 +ArticularyWordRecognition,0.9933333333333333 +AsphaltObstaclesCoordinates,0.8081841432225064 +AsphaltRegularityCoordinates,0.958005249343832 +AtrialFibrillation,0.13333333333333333 +AutomotiveRoadTrials,0.7758620689655172 +BIDMC32HR_disc,0.8257607336390163 +BIDMC32SpO2_disc,0.7925407925407926 +BeijingPM10Quality_disc,0.9272980501392758 +BeijingPM25Quality_disc,0.9253265190232822 +BenzeneConcentration_disc,0.900308901993822 +Blink,0.848 +BoneIntensitiesAgeGroup,0.8337078651685393 +BoneProbAgeGroup,0.6629213483146067 +CharacterTrajectories,0.9902506963788301 +CounterMovementJump,0.8603351955307262 +Cricket,1.0 +CrowdSourced,0.4890613973182781 +DuckDuckGeese,0.48 +ERing,0.9592592592592593 +EigenWorms,0.9465648854961832 +EmoPain,0.8732394366197183 +Epilepsy,1.0 +EthanolConcentration,0.5247148288973384 +EyesOpenShut,0.19047619047619047 +FaceDetection,0.7292849035187288 +FordChallenge,0.9418270294182703 +HandMovementDirection,0.3108108108108108 +Handwriting,0.32 +Heartbeat,0.9459459459459459 +HouseholdPowerConsumption1_disc,0.8454810495626822 +HouseholdPowerConsumption2_disc,0.7346938775510204 +IEEEPPG_disc,0.5768072289156626 +IRDS-SFL,0.8586956521739131 +JapaneseVowels,0.9351351351351351 +KERAAL-RTK,1.0 +KIMORE-PR-C,0.6666666666666666 +KINECAL-QSEO,1.0 +LSST,0.610705596107056 +Libras,0.9111111111111111 +Locust2022,0.9773785761809713 +LowCost,0.6066666666666667 +MindReading,0.5834609494640123 +MotionSenseHAR,1.0 +NATOPS,0.9111111111111111 +PEMS-SF,0.9826589595375722 +PenDigits,0.9519725557461407 +PhonemeSpectra,0.330450342976439 +PhotoStimulation,0.3888888888888889 +RacketSports,0.8421052631578947 +SelfRegulationSCP1,0.6122448979591837 +SpokenArabicDigits,0.9872669395179627 +StandWalkJump,0.4 +TactileTextureRecognition,1.0 +Tiselac,0.7450455005055612 +UCDHE-Rowing-MC,0.7681818181818182 +UCIActivity,0.978758850478967 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.903125 +WISDM,0.8547421286459339 diff --git a/results/multiverse/datasets.html b/results/multiverse/datasets.html index 35734be..638fd6b 100644 --- a/results/multiverse/datasets.html +++ b/results/multiverse/datasets.html @@ -60,7 +60,7 @@ details { margin-top: .6rem; } summary { cursor: pointer; color: var(--accent); } code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: .9em; } -tr.nosignal td { background: rgba(214, 158, 46, .16); }tr.saturated td { background: rgba(56, 161, 105, .14); }

    Multiverse-core datasets: accuracy

    64 datasets · accuracy · best of up to 24 estimators against the Dummy baseline · built 2026-09-19

    DatasetDummyMedianBestBest estimatorGain over dummySpreadEstimators
    KINECAL-QSEO0.94120.94120.9412Arsenal0.00000.647124
    BIDMC32SpO2_disc0.71530.65740.7203ROCKET0.00500.190124
    Locust20220.91120.90820.9206MRHydra0.00940.052423
    Heartbeat0.72200.74880.7854CIF0.06340.126824
    HouseholdPowerConsumption2_disc0.72160.76530.7872DisjointCNN0.06560.097724
    MotorImagery0.50000.51000.5900HC20.09000.130024
    AutomotiveRoadTrials0.75320.78570.8442CIF0.09090.233824
    EyesOpenShut0.50000.50000.5952STSF0.09520.190524
    EmoPain0.78310.83380.8845Catch220.10140.256324
    AppliancesEnergy_disc0.80950.82140.9286DrCIF0.11900.428624
    BeijingPM10Quality_disc0.71120.82420.8417STSF0.13050.126024
    Alzheimers0.41860.37210.5581MRHydra0.13950.302323
    PhotoStimulation0.41670.38890.5833ROCKET0.16670.388923
    FaceDetection0.50000.62510.6850H-InceptionTime0.18500.170524
    BeijingPM25Quality_disc0.69770.87510.8879ConvTran0.19020.128824
    AtrialFibrillation0.33330.26670.5333TS2Vec0.20000.466724
    HouseholdPowerConsumption1_disc0.77840.91110.9825STSF0.20410.864424
    LowCost0.50000.63170.7300TSF0.23000.248324
    BoneProbAgeGroup0.47640.64380.7124H-InceptionTime0.23600.184324
    StandWalkJump0.33330.40000.6000MRHydra0.26670.400024
    CrowdSourced0.50020.71900.7734LITETime-MV0.27320.176824
    BenzeneConcentration_disc0.68970.81070.9768STSF0.28700.577024
    FordChallenge0.62320.88430.9360QUANT0.31280.312824
    BIDMC32HR_disc0.65070.80120.9637RIST0.31300.635324
    BoneIntensitiesAgeGroup0.47640.79890.8202HC20.34380.256224
    STEW0.50000.74020.8448HC20.34480.216324
    PhonemeSpectra0.02560.27100.3746H-InceptionTime0.34890.291124
    KERAAL-RTK0.57140.78570.9286HC20.35710.571424
    LSST0.31510.62900.6752HC20.36010.452124
    HandMovementDirection0.20270.41220.6081TSF0.40540.351424
    DuckDuckGeese0.20000.46000.6400H-InceptionTime0.44000.480024
    SelfRegulationSCP10.50170.85320.9454MRHydra0.44370.392524
    IEEEPPG_disc0.26050.45480.7078ConvTran0.44730.438324
    AsphaltRegularityCoordinates0.50730.97870.9947H-InceptionTime0.48740.061324
    MindReading0.23120.52830.7243LITETime-MV0.49310.385924
    EthanolConcentration0.25100.43350.7490STC0.49810.513324
    UIPRMD-DS-C0.50000.83331.0000Catch220.50000.388924
    WISDM0.36640.86580.8965MRHydra0.53000.131024
    Blink0.44440.98671.0000Arsenal0.55560.428924
    EigenWorms0.41980.86260.9771MRHydra0.55730.557323
    KIMORE-PR-C0.14290.42860.7143LITETime-MV0.57140.571424
    AsphaltObstaclesCoordinates0.28390.81970.8670MRHydra0.58310.199524
    CounterMovementJump0.33520.75700.9274Arsenal0.59220.458124
    Handwriting0.03760.37290.6529H-InceptionTime0.61530.478824
    RacketSports0.28290.87830.9079RDST0.62500.125024
    UCDHE-Rowing-MC0.20450.73410.8295PatchMTSC0.62500.259124
    USCActivity0.11380.69240.7473HC20.63340.166422
    IRDS-SFL0.20690.79310.8966XCM0.68970.482824
    Skoda0.23560.94550.9646H-InceptionTime0.72900.119924
    Epilepsy0.26810.98191.0000HC20.73190.101424
    Tiselac0.06280.80860.8373STSF0.77450.204421
    MotionSenseHAR0.20380.98871.0000DrCIF0.79620.101924
    NATOPS0.16670.89170.9667LITETime-MV0.80000.155624
    UCIActivity0.19160.97690.9983LITETime-MV0.80670.174924
    UWaveGestureLibrary0.12500.90940.9406Arsenal0.81560.553124
    ERing0.16670.94070.9963MRHydra0.82960.692624
    PEMS-SF0.11560.86991.0000CIF0.88440.317924
    SpokenArabicDigits0.10000.98160.9945DisjointCNN0.89450.129124
    Libras0.06670.88890.9722RIST0.90560.338924
    JapaneseVowels0.08380.97030.9946LITETime-MV0.91080.208124
    Cricket0.08330.97921.0000Arsenal0.91670.069424
    CharacterTrajectories0.06480.98960.9958H-InceptionTime0.93110.044624
    TactileTextureRecognition0.05140.99851.0000H-InceptionTime0.94860.234924
    ArticularyWordRecognition0.04000.98000.9933Arsenal0.95330.050024

    One row per dataset. Dummy is the no-skill floor. Median, best and spread are over the other estimators, so the baseline cannot flatter them. Gain over dummy is best minus dummy, how much skill was found at all; spread is best minus worst, how much the choice of estimator mattered. The two answer different questions, and a single range would conflate them.

    3 of 64 datasets gained 0.05 or less over the baseline (shaded amber) and 14 have a best of 0.99 or more (shaded green). Both separate estimators poorly, for opposite reasons. Best is a maximum over many estimators, so it is optimistic by construction: read it as what the archive can currently do on a problem, not as what any one method delivers.