diff --git a/README.md b/README.md index 8ee3cdc..da32d64 100644 --- a/README.md +++ b/README.md @@ -40,35 +40,33 @@ The current paper version describes: | # | Estimator | Accuracy rank | Accuracy | Balanced accuracy | AUROC | F1 | Log loss ↓ | Sensitivity | Specificity | |---|---|---|---|---|---|---|---|---|---| -| 1 | HC2 | **8.40** | **0.7887** | 0.7541 | **0.9000** | 0.7346 | **0.5440** | 0.7547 | **0.7910** | -| 2 | MRHydra | 9.21 | 0.7810 | **0.7579** | 0.8130 | **0.7368** | 7.8942 | **0.7715** | 0.7718 | -| 3 | RDST | 10.10 | 0.7707 | 0.7372 | 0.7963 | 0.7105 | 8.2660 | 0.7236 | 0.7833 | -| 4 | RIST | 10.75 | 0.7693 | 0.7422 | 0.8755 | 0.7221 | 0.6294 | 0.7504 | 0.7613 | -| 5 | DrCIF | 10.87 | 0.7721 | 0.7454 | 0.8821 | 0.7248 | 0.6558 | 0.7490 | 0.7669 | -| 6 | CIF | 11.07 | 0.7756 | 0.7497 | 0.8920 | 0.7288 | 0.6497 | 0.7536 | 0.7714 | -| 7 | FreshPRINCE | 11.08 | 0.7717 | 0.7516 | 0.8752 | 0.7293 | 0.6075 | 0.7515 | 0.7731 | -| 8 | Arsenal | 11.42 | 0.7654 | 0.7340 | 0.8471 | 0.7092 | 3.9265 | 0.7337 | 0.7696 | -| 9 | QUANT | 11.65 | 0.7693 | 0.7486 | 0.8839 | 0.7262 | 0.6238 | 0.7616 | 0.7539 | -| 10 | LITETime-MV | 11.97 | 0.7476 | 0.7312 | 0.8518 | 0.6875 | 1.3300 | 0.7200 | 0.7600 | -| 11 | ROCKET | 12.01 | 0.7661 | 0.7345 | 0.7955 | 0.7080 | 8.4299 | 0.7282 | 0.7724 | -| 12 | STSF | 12.60 | 0.7698 | 0.7503 | 0.8813 | 0.7155 | 0.6493 | 0.7439 | 0.7790 | -| 13 | H-InceptionTime | 12.71 | 0.7375 | 0.7205 | 0.8506 | 0.6897 | 1.3334 | 0.7303 | 0.7333 | -| 14 | LiteTIME | 13.22 | 0.7308 | 0.7122 | 0.8402 | 0.6746 | 1.4921 | 0.7199 | 0.7291 | -| 15 | DisjointCNN | 13.63 | 0.7286 | 0.7061 | 0.8354 | 0.6688 | 1.9705 | 0.6889 | 0.7368 | -| 16 | ConvTran | 14.37 | 0.7430 | 0.7139 | 0.8606 | 0.6882 | 0.8300 | 0.7289 | 0.7295 | -| 17 | Catch22 | 14.50 | 0.7442 | 0.7203 | 0.8703 | 0.6996 | 0.7238 | 0.7337 | 0.7326 | -| 18 | PatchMTSC | 14.51 | 0.7395 | 0.6934 | 0.8288 | 0.6660 | 0.7748 | 0.6985 | 0.7300 | -| 19 | STC | 15.19 | 0.7516 | 0.7188 | 0.8748 | 0.7004 | 0.6447 | 0.7264 | 0.7496 | -| 20 | TSF | 15.36 | 0.7484 | 0.7257 | 0.8747 | 0.6952 | 0.7335 | 0.7179 | 0.7565 | -| 21 | TS2Vec | 15.87 | 0.7212 | 0.6849 | 0.8082 | 0.6588 | 0.7326 | 0.6980 | 0.7100 | -| 22 | TDE | 15.93 | 0.7230 | 0.6823 | 0.8383 | 0.6441 | 0.8859 | 0.6786 | 0.7301 | -| 23 | Summary | 18.54 | 0.6814 | 0.6586 | 0.8263 | 0.6294 | 0.9251 | 0.6661 | 0.6787 | -| 24 | TimesNet | 18.86 | 0.6971 | 0.6688 | 0.8280 | 0.6390 | 1.1785 | 0.6850 | 0.6838 | -| 25 | TimesURL | 18.95 | 0.6916 | 0.6563 | 0.7931 | 0.6084 | 1.0193 | 0.6379 | 0.6914 | -| 26 | 1NN-DTW | 20.47 | 0.6672 | 0.6457 | 0.7214 | 0.6193 | 11.9949 | 0.6584 | 0.6584 | -| 27 | Dummy | 24.77 | 0.3538 | 0.2991 | 0.5000 | 0.1537 | 1.4284 | 0.2911 | 0.3695 | - -Average over the 51 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. +| 1 | HC2 | **7.64** | **0.7917** | **0.7557** | **0.8935** | **0.7302** | **0.5350** | 0.7469 | **0.7998** | +| 2 | MRHydra | 9.08 | 0.7794 | 0.7520 | 0.8040 | 0.7266 | 7.9526 | **0.7577** | 0.7768 | +| 3 | RDST | 9.46 | 0.7729 | 0.7386 | 0.7928 | 0.7075 | 8.1867 | 0.7172 | 0.7902 | +| 4 | RIST | 9.85 | 0.7744 | 0.7451 | 0.8679 | 0.7174 | 0.6150 | 0.7416 | 0.7743 | +| 5 | CIF | 10.23 | 0.7770 | 0.7487 | 0.8842 | 0.7246 | 0.6442 | 0.7459 | 0.7780 | +| 6 | DrCIF | 10.25 | 0.7731 | 0.7433 | 0.8745 | 0.7189 | 0.6430 | 0.7400 | 0.7736 | +| 7 | QUANT | 10.70 | 0.7668 | 0.7404 | 0.8694 | 0.7166 | 0.7285 | 0.7491 | 0.7570 | +| 8 | LITETime-MV | 10.95 | 0.7511 | 0.7320 | 0.8503 | 0.6878 | 1.3004 | 0.7167 | 0.7660 | +| 9 | Arsenal | 11.04 | 0.7663 | 0.7335 | 0.8419 | 0.7061 | 3.6444 | 0.7266 | 0.7752 | +| 10 | ROCKET | 11.09 | 0.7690 | 0.7362 | 0.7925 | 0.7065 | 8.3274 | 0.7228 | 0.7798 | +| 11 | STSF | 11.29 | 0.7727 | 0.7508 | 0.8723 | 0.7164 | 0.6685 | 0.7412 | 0.7845 | +| 12 | H-InceptionTime | 11.61 | 0.7421 | 0.7208 | 0.8447 | 0.6853 | 1.3448 | 0.7227 | 0.7436 | +| 13 | ConvTran | 12.88 | 0.7490 | 0.7177 | 0.8529 | 0.6862 | 0.8826 | 0.7234 | 0.7419 | +| 14 | DisjointCNN | 13.08 | 0.7296 | 0.7057 | 0.8246 | 0.6641 | 2.1100 | 0.6815 | 0.7431 | +| 15 | PatchMTSC | 13.37 | 0.7454 | 0.6990 | 0.8250 | 0.6671 | 0.7818 | 0.6981 | 0.7397 | +| 16 | Catch22 | 13.47 | 0.7463 | 0.7177 | 0.8605 | 0.6929 | 0.7068 | 0.7229 | 0.7420 | +| 17 | TSF | 14.01 | 0.7426 | 0.7175 | 0.8571 | 0.6896 | 0.9987 | 0.7095 | 0.7521 | +| 18 | STC | 14.03 | 0.7507 | 0.7137 | 0.8624 | 0.6803 | 0.6468 | 0.7036 | 0.7611 | +| 19 | TDE | 14.99 | 0.7251 | 0.6834 | 0.8339 | 0.6446 | 0.8524 | 0.6759 | 0.7349 | +| 20 | TS2Vec | 15.21 | 0.7201 | 0.6809 | 0.7994 | 0.6527 | 0.7835 | 0.6879 | 0.7144 | +| 21 | XCM | 16.58 | 0.6706 | 0.6370 | 0.7893 | 0.5771 | 2.1798 | 0.6167 | 0.6875 | +| 22 | Summary | 16.92 | 0.6845 | 0.6570 | 0.8113 | 0.6299 | 0.9645 | 0.6621 | 0.6852 | +| 23 | TimesNet | 17.24 | 0.7020 | 0.6709 | 0.8218 | 0.6396 | 1.2889 | 0.6810 | 0.6934 | +| 24 | TimesURL | 17.36 | 0.6950 | 0.6562 | 0.7831 | 0.6052 | 0.9868 | 0.6312 | 0.7016 | +| 25 | Dummy | 22.69 | 0.3709 | 0.3067 | 0.5000 | 0.1626 | 1.3928 | 0.2987 | 0.3880 | + +Average over the 56 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. Rebuilt with `python -m multiverse.experiments.tables`, which also writes a sortable diff --git a/docs/classifiers.md b/docs/classifiers.md index 24dfecd..470ec09 100644 --- a/docs/classifiers.md +++ b/docs/classifiers.md @@ -114,14 +114,26 @@ Mathematics, 9(23), 2021. This is the only Keras port here, following the authors, so it needs `tensorflow` rather than `torch`. Both are in the `deep-learning` extra. -The XCM results in this repository follow the authors' tuning protocol. Section 4.3 sets +**The XCM results published here are the fixed-parameter run**: a single fit at window +0.8 with batch 32, not the per-dataset search. The search was run over the full core 66 +and did not pay for itself. Across the 65 shared datasets it was 0.017 mean accuracy +worse than the single fit, 31 wins to 31 with 3 ties, Wilcoxon p = 0.63. Worse, on the +14 datasets where the search happened to select 0.8 — the same window as the fixed run, +so the only difference is initialisation — the two runs still differed by 0.11 mean +absolute accuracy, and by 0.48 on HouseholdPowerConsumption2_disc and 0.44 on Libras. +At one resample XCM's run-to-run variance is larger than the effect the search is +tuning for, which makes the cross-validated selection largely a choice over noise: its +five per-window scores on Locust2022 run 0.674, 0.888, 0.253, 0.590, 0.707. The tuned +results are kept out of the tables rather than deleted. + +The search itself follows the authors' protocol. Section 4.3 sets `window_size` and `batch_size` per dataset "by grid search based on the best average accuracy following a stratified 5-fold cross-validation on the training set", over windows {0.2, 0.4, 0.6, 0.8, 1.0} and batches {1, 8, 32}. Selection never touches the test data. -The reported run searches the window on that grid and holds batch size at 32. That is -the one departure, and it is a cost decision rather than a modelling one: batch 1 takes +That search holds batch size at 32 rather than searching it. That is a cost decision +rather than a modelling one: batch 1 takes roughly 32 times the gradient steps, which would turn a day of GPU time into about 900 hours, for a value the published table selects on 4 of 30 datasets. diff --git a/docs/leaderboard.md b/docs/leaderboard.md index 2318f21..ce8ecf1 100644 --- a/docs/leaderboard.md +++ b/docs/leaderboard.md @@ -109,35 +109,33 @@ inferred from the ranking. | # | Estimator | Accuracy rank | Accuracy | Balanced accuracy | AUROC | F1 | Log loss ↓ | Sensitivity | Specificity | |---|---|---|---|---|---|---|---|---|---| -| 1 | HC2 | **7.37** | **0.7665** | **0.7452** | 0.8823 | 0.7411 | **0.6692** | 0.7470 | **0.7703** | -| 2 | RDST | 8.80 | 0.7459 | 0.7294 | 0.8179 | 0.7243 | 9.1587 | 0.7263 | 0.7560 | -| 3 | MRHydra | 9.20 | 0.7523 | 0.7388 | 0.8236 | **0.7432** | 8.9285 | 0.7599 | 0.7360 | -| 4 | Arsenal | 10.26 | 0.7321 | 0.7134 | 0.8439 | 0.7116 | 5.5624 | 0.7119 | 0.7417 | -| 5 | ROCKET | 10.28 | 0.7317 | 0.7146 | 0.8089 | 0.7130 | 9.6705 | 0.7133 | 0.7393 | -| 6 | RIST | 10.57 | 0.7433 | 0.7278 | 0.8755 | 0.7325 | 0.7983 | 0.7454 | 0.7322 | -| 7 | H-InceptionTime | 10.67 | 0.7223 | 0.7230 | 0.8653 | 0.6967 | 1.5030 | 0.7053 | 0.7345 | -| 8 | CIF | 11.33 | 0.7525 | 0.7378 | 0.8825 | 0.7400 | 0.8488 | **0.7604** | 0.7349 | -| 9 | FreshPRINCE | 11.74 | 0.7422 | 0.7281 | 0.8796 | 0.7239 | 0.7764 | 0.7313 | 0.7457 | -| 10 | DrCIF | 11.83 | 0.7386 | 0.7252 | 0.8734 | 0.7246 | 0.8458 | 0.7384 | 0.7303 | -| 11 | LITETime-MV | 12.00 | 0.7073 | 0.7064 | 0.8568 | 0.6779 | 1.4779 | 0.6905 | 0.7218 | -| 12 | LiteTIME | 12.50 | 0.7087 | 0.7019 | 0.8576 | 0.6751 | 1.6854 | 0.6985 | 0.7204 | -| 13 | DisjointCNN | 13.20 | 0.7011 | 0.7030 | 0.8510 | 0.6704 | 1.7943 | 0.6938 | 0.7070 | -| 14 | QUANT | 13.78 | 0.7285 | 0.7171 | **0.8888** | 0.7195 | 0.8041 | 0.7421 | 0.7074 | -| 15 | STSF | 14.24 | 0.7345 | 0.7223 | 0.8774 | 0.6934 | 0.8338 | 0.7007 | 0.7600 | -| 16 | TS2Vec | 14.78 | 0.7070 | 0.6913 | 0.8470 | 0.6917 | 0.8902 | 0.7150 | 0.6877 | -| 17 | TDE | 15.15 | 0.7079 | 0.6862 | 0.8484 | 0.6775 | 1.1475 | 0.6897 | 0.7095 | -| 18 | PatchMTSC | 15.37 | 0.7110 | 0.6986 | 0.8601 | 0.6899 | 0.7670 | 0.7192 | 0.6928 | -| 19 | ConvTran | 16.11 | 0.6931 | 0.6801 | 0.8552 | 0.6793 | 0.8155 | 0.7049 | 0.6736 | -| 20 | STC | 16.15 | 0.7265 | 0.7036 | 0.8803 | 0.7035 | 0.8124 | 0.7186 | 0.7184 | -| 21 | TSF | 16.17 | 0.7214 | 0.7076 | 0.8671 | 0.6917 | 0.9127 | 0.6977 | 0.7365 | -| 22 | Catch22 | 16.30 | 0.7006 | 0.6854 | 0.8557 | 0.6897 | 0.9814 | 0.7096 | 0.6802 | -| 23 | 1NN-DTW | 17.61 | 0.6848 | 0.6759 | 0.7785 | 0.6702 | 11.3600 | 0.6720 | 0.6879 | -| 24 | TimesURL | 18.24 | 0.6809 | 0.6658 | 0.8290 | 0.6539 | 1.3557 | 0.6698 | 0.6738 | -| 25 | Summary | 19.48 | 0.6477 | 0.6355 | 0.8295 | 0.6206 | 1.3000 | 0.6291 | 0.6589 | -| 26 | TimesNet | 20.09 | 0.6584 | 0.6504 | 0.8332 | 0.6386 | 1.1641 | 0.6628 | 0.6504 | -| 27 | Dummy | 24.78 | 0.2168 | 0.1980 | 0.5000 | 0.0800 | 1.9123 | 0.1853 | 0.2288 | - -Average over the 23 UEA datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. +| 1 | HC2 | **6.65** | **0.7617** | **0.7412** | 0.8752 | **0.7372** | **0.6681** | 0.7429 | **0.7655** | +| 2 | RDST | 8.27 | 0.7407 | 0.7250 | 0.8098 | 0.7197 | 9.3448 | 0.7212 | 0.7512 | +| 3 | MRHydra | 8.77 | 0.7462 | 0.7332 | 0.8145 | 0.7371 | 9.1480 | 0.7526 | 0.7315 | +| 4 | Arsenal | 9.38 | 0.7282 | 0.7103 | 0.8375 | 0.7084 | 5.3575 | 0.7084 | 0.7380 | +| 5 | H-InceptionTime | 9.56 | 0.7208 | 0.7214 | 0.8605 | 0.6962 | 1.5037 | 0.7046 | 0.7323 | +| 6 | ROCKET | 9.65 | 0.7265 | 0.7101 | 0.8004 | 0.7084 | 9.8595 | 0.7084 | 0.7341 | +| 7 | RIST | 10.04 | 0.7374 | 0.7226 | 0.8660 | 0.7261 | 0.7933 | 0.7370 | 0.7292 | +| 8 | CIF | 10.42 | 0.7475 | 0.7334 | 0.8742 | 0.7356 | 0.8410 | **0.7550** | 0.7307 | +| 9 | LITETime-MV | 10.96 | 0.7048 | 0.7040 | 0.8505 | 0.6769 | 1.4815 | 0.6894 | 0.7181 | +| 10 | DrCIF | 11.10 | 0.7328 | 0.7199 | 0.8639 | 0.7193 | 0.8386 | 0.7324 | 0.7250 | +| 11 | QUANT | 12.54 | 0.7245 | 0.7136 | **0.8803** | 0.7161 | 0.7978 | 0.7383 | 0.7035 | +| 12 | DisjointCNN | 12.62 | 0.6943 | 0.6961 | 0.8387 | 0.6627 | 1.8839 | 0.6830 | 0.7044 | +| 13 | STSF | 12.71 | 0.7309 | 0.7191 | 0.8703 | 0.6914 | 0.8255 | 0.6982 | 0.7556 | +| 14 | PatchMTSC | 13.71 | 0.7096 | 0.6977 | 0.8549 | 0.6906 | 0.7619 | 0.7220 | 0.6874 | +| 15 | TS2Vec | 13.96 | 0.6990 | 0.6839 | 0.8334 | 0.6853 | 0.8820 | 0.7088 | 0.6783 | +| 16 | TDE | 14.19 | 0.7026 | 0.6818 | 0.8386 | 0.6745 | 1.1277 | 0.6877 | 0.7016 | +| 17 | ConvTran | 14.23 | 0.6915 | 0.6791 | 0.8493 | 0.6784 | 0.8090 | 0.7032 | 0.6725 | +| 18 | TSF | 14.50 | 0.7183 | 0.7052 | 0.8600 | 0.6901 | 0.9018 | 0.6962 | 0.7324 | +| 19 | STC | 14.62 | 0.7224 | 0.7004 | 0.8722 | 0.7000 | 0.8052 | 0.7138 | 0.7157 | +| 20 | Catch22 | 14.98 | 0.6945 | 0.6800 | 0.8443 | 0.6836 | 0.9690 | 0.7021 | 0.6762 | +| 21 | XCM | 15.98 | 0.6356 | 0.6204 | 0.8184 | 0.5899 | 1.3507 | 0.6172 | 0.6410 | +| 22 | TimesURL | 17.04 | 0.6745 | 0.6600 | 0.8170 | 0.6473 | 1.3326 | 0.6612 | 0.6703 | +| 23 | TimesNet | 17.92 | 0.6582 | 0.6505 | 0.8275 | 0.6400 | 1.1485 | 0.6648 | 0.6481 | +| 24 | Summary | 18.06 | 0.6431 | 0.6314 | 0.8181 | 0.6179 | 1.2745 | 0.6269 | 0.6521 | +| 25 | Dummy | 23.15 | 0.2286 | 0.2106 | 0.5000 | 0.1044 | 1.8615 | 0.2193 | 0.2193 | + +Average over the 24 UEA datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. Sortable version with per-metric ranks: diff --git a/multiverse/classification/_rankscl.py b/multiverse/classification/_rankscl.py index d64d9ef..418210b 100644 --- a/multiverse/classification/_rankscl.py +++ b/multiverse/classification/_rankscl.py @@ -176,6 +176,20 @@ def _ranking_loss(embeddings, labels, distance): Returns None when the batch has no anchor with both a positive and a negative, which the authors' version would raise on. + + Evaluated as one dense expression rather than the authors' loop over anchors + and positives. The loop is what the paper describes, but at the archive + settings a batch holds ``batch_size * (2 * aug_positives + 1)`` embeddings, + 44 by default, and every case is repeated, so each anchor has around ten + positives. That is roughly 440 Python iterations per optimiser step, each + launching several small CUDA kernels, and launch latency then decides the + runtime rather than the arithmetic: measured at about 3.5 seconds a step for + a small encoder on an H200, which timed LSST out of a 60 hour job at epoch + 91 of 100. The value is unchanged; only the order of the reduction differs. + + Memory is cubic in the batch, ``n * n * n`` floats for the pairwise + differences, which is 85k elements at the defaults. A much larger + ``batch_size`` or ``aug_positives`` would need this chunked over anchors. """ if distance == "Cosine": matrix = -torch.cosine_similarity( @@ -185,20 +199,23 @@ def _ranking_loss(embeddings, labels, distance): matrix = torch.cdist(embeddings, embeddings, p=2) same = labels.reshape(1, -1) == labels.reshape(-1, 1) - violations = [] - for anchor in range(matrix.shape[0]): - negatives = matrix[anchor][~same[anchor]] - if negatives.numel() == 0: - continue - positives = same[anchor].nonzero().flatten() - for positive in positives[positives != anchor]: - gap = matrix[anchor, positive] - closer = negatives[negatives <= gap] - violations.append(torch.sigmoid(gap - closer).sum()) - - if not violations: + negative = ~same + # a positive is a same-class case other than the anchor, and an anchor with + # no negative is skipped entirely, as in the authors' loop + pairs = same & ~torch.eye( + matrix.shape[0], dtype=torch.bool, device=matrix.device + ) + pairs = pairs & negative.any(dim=1, keepdim=True) + if not pairs.any(): return None - return torch.atan(torch.stack(violations)).mean() + + # difference[anchor, positive, other] = d(anchor, positive) - d(anchor, other) + difference = matrix.unsqueeze(2) - matrix.unsqueeze(1) + # the wrongly ranked ones: a negative at least as close as the positive is + wrong = negative.unsqueeze(1) & (difference >= 0) + violations = (torch.sigmoid(difference) * wrong).sum(dim=2) + + return torch.atan(violations)[pairs].mean() class RankSCLClassifier(BaseClassifier): @@ -359,7 +376,15 @@ def _subsample(self, features, y): return features, y def _build_probe(self, n_cases: int, seed: int): - """Return the probe, following ``utils/_eval_protocols.py``.""" + """Return the probe, following ``utils/_eval_protocols.py``. + + The SVM is built with ``probability=False``, which is what the authors' + grid sets. Platt scaling is not free: libsvm fits it by an internal + five-fold cross-validation inside every ``fit``, so an estimator carrying + ``probability=True`` into a ten-value grid over five folds costs about + 300 SVC trainings rather than 50. ``_fit_probe`` turns it back on for a + single refit at the selected C, which is what ``predict_proba`` needs. + """ if self.probe == "logistic": return make_pipeline( StandardScaler(), @@ -367,7 +392,7 @@ def _build_probe(self, n_cases: int, seed: int): LogisticRegression(max_iter=1000000, random_state=seed) ), ) - svm = SVC(C=np.inf, gamma="scale", probability=True, random_state=seed) + svm = SVC(C=np.inf, gamma="scale", probability=False, random_state=seed) if n_cases // self.n_classes_ < 5 or n_cases < 50: return svm return GridSearchCV( @@ -378,6 +403,28 @@ def _build_probe(self, n_cases: int, seed: int): n_jobs=1, ) + def _fit_probe(self, features, y, seed): + """Select the probe's parameters, then refit it with probabilities on. + + The selection is the authors' own: their grid sets + ``probability=False``, and scoring uses ``predict``, which reads the + decision function either way, so the chosen C is unchanged. Only the + final estimator needs Platt scaling, because aeon classifiers must + implement ``predict_proba``. + """ + probe = self._build_probe(features.shape[0], seed) + if self.probe == "logistic": + return probe.fit(features, y) + if isinstance(probe, GridSearchCV): + probe.fit(features, y) + parameters = probe.best_params_ + else: + # the degenerate-case bypass: too few cases per class to select on + parameters = {"C": probe.C, "kernel": "rbf", "gamma": "scale"} + return SVC(probability=True, random_state=seed, **parameters).fit( + features, y + ) + def _fit(self, X: np.ndarray, y): self._validate_parameters() @@ -452,9 +499,7 @@ def _fit(self, X: np.ndarray, y): representations = self._encode(X) fit_features, fit_y = self._subsample(representations, encoded_y) self.probe_cases_ = int(fit_features.shape[0]) - self.probe_ = self._build_probe(self.probe_cases_, seed).fit( - fit_features, fit_y - ) + self.probe_ = self._fit_probe(fit_features, fit_y, seed) return self def _check_shape(self, X: np.ndarray) -> None: diff --git a/multiverse/classification/_ts2vec.py b/multiverse/classification/_ts2vec.py index 63f5e62..8acc2c2 100644 --- a/multiverse/classification/_ts2vec.py +++ b/multiverse/classification/_ts2vec.py @@ -17,9 +17,11 @@ by grid search over C on the training representations (``train.py`` passes ``eval_protocol='svm'``), so that is the default here. Their grid sets ``probability=False``, which leaves an ``SVC`` unable to produce probability -estimates, so this wrapper sets it True: aeon classifiers must implement -``predict_proba``. That adds Platt scaling, fitted by internal cross-validation -on the training data only. +estimates, and aeon classifiers must implement ``predict_proba``. The search +therefore runs with it off, as theirs does, and the selected C is refitted once +with it on. Platt scaling is not free: libsvm fits it by an internal five-fold +cross-validation on every fit, so carrying it through the grid would cost about +six times the search. This wrapper is designed for aeon and therefore assumes input X is a 3D NumPy array with shape (n_cases, n_channels, n_timepoints). The original expects @@ -251,7 +253,14 @@ def _build_probe(self, n_cases: int, seed: int): The SVM path mirrors ``tasks/_eval_protocols.py::fit_svm``: a plain SVC on very small or very unbalanced collections, otherwise a grid search - over C. ``probability=True`` is set so that ``predict_proba`` exists. + over C. + + The SVM is built with ``probability=False``, which is what the authors' + grid sets. Platt scaling is not free: libsvm fits it by an internal + five-fold cross-validation inside every ``fit``, so an estimator carrying + ``probability=True`` into a ten-value grid over five folds costs about + 300 SVC trainings rather than 50. ``_fit_probe`` turns it back on for a + single refit at the selected C, which is what ``predict_proba`` needs. """ if self.probe == "logistic": # One-vs-rest, matching the authors' fit_lr and the TimesURL probe. @@ -265,7 +274,7 @@ def _build_probe(self, n_cases: int, seed: int): ), ) - svm = SVC(C=np.inf, gamma="scale", probability=True, random_state=seed) + svm = SVC(C=np.inf, gamma="scale", probability=False, random_state=seed) if n_cases // self.n_classes_ < 5 or n_cases < 50: return svm return GridSearchCV( @@ -276,6 +285,28 @@ def _build_probe(self, n_cases: int, seed: int): n_jobs=1, ) + def _fit_probe(self, features, y, seed): + """Select the probe's parameters, then refit it with probabilities on. + + The selection is the authors' own: their grid sets + ``probability=False``, and scoring uses ``predict``, which reads the + decision function either way, so the chosen C is unchanged. Only the + final estimator needs Platt scaling, because aeon classifiers must + implement ``predict_proba``. + """ + probe = self._build_probe(features.shape[0], seed) + if self.probe == "logistic": + return probe.fit(features, y) + if isinstance(probe, GridSearchCV): + probe.fit(features, y) + parameters = probe.best_params_ + else: + # the degenerate-case bypass: too few cases per class to select on + parameters = {"C": probe.C, "kernel": "rbf", "gamma": "scale"} + return SVC(probability=True, random_state=seed, **parameters).fit( + features, y + ) + def _encode(self, X: np.ndarray) -> np.ndarray: """Embed a collection with the fitted encoder, one vector per case.""" return self.encoder_.encode( @@ -325,9 +356,7 @@ def _fit(self, X: np.ndarray, y): ) fit_features, fit_y = self._subsample(encoded, encoded_y, seed) self.probe_cases_ = int(fit_features.shape[0]) - self.probe_ = self._build_probe( - self.probe_cases_, seed - ).fit(fit_features, fit_y) + self.probe_ = self._fit_probe(fit_features, fit_y, seed) return self def _check_shape(self, X: np.ndarray) -> None: diff --git a/multiverse/classification/tests/test_original_equivalence.py b/multiverse/classification/tests/test_original_equivalence.py index 2b9c01e..3e4bb59 100644 --- a/multiverse/classification/tests/test_original_equivalence.py +++ b/multiverse/classification/tests/test_original_equivalence.py @@ -674,6 +674,23 @@ def test_ts2vec_svm_probe_matches_the_authors_grid(): } +def test_ts2vec_svm_grid_does_not_carry_platt_scaling(): + """The searched estimator must match the authors' probability=False. + + libsvm fits Platt scaling by an internal five-fold cross-validation inside + every ``fit``, so carrying probability=True into the grid multiplies a + ten-value search over five folds from about 50 SVC trainings to 300. The + authors set probability=False in their grid and we only need probabilities + on the final estimator. + """ + from multiverse.classification import TS2VecClassifier + + clf = TS2VecClassifier(probe="svm") + clf.n_classes_ = 2 + assert clf._build_probe(n_cases=100, seed=0).estimator.probability is False + assert clf._build_probe(n_cases=10, seed=0).probability is False + + # --------------------------------------------------------------------------- # XCM # --------------------------------------------------------------------------- diff --git a/multiverse/classification/tests/test_rankscl.py b/multiverse/classification/tests/test_rankscl.py index f37cf7a..bbe770d 100644 --- a/multiverse/classification/tests/test_rankscl.py +++ b/multiverse/classification/tests/test_rankscl.py @@ -65,6 +65,64 @@ def distance(i, j): assert float(_ranking_loss(embeddings, labels, "EU")) == pytest.approx(expected) +def _loop_ranking_loss(embeddings, labels, distance): + """The authors' loop, kept as the reference the vectorised form must match.""" + if distance == "Cosine": + matrix = -torch.cosine_similarity( + embeddings.unsqueeze(1), embeddings.unsqueeze(0), dim=2 + ) + else: + matrix = torch.cdist(embeddings, embeddings, p=2) + same = labels.reshape(1, -1) == labels.reshape(-1, 1) + violations = [] + for anchor in range(matrix.shape[0]): + negatives = matrix[anchor][~same[anchor]] + if negatives.numel() == 0: + continue + positives = same[anchor].nonzero().flatten() + for positive in positives[positives != anchor]: + gap = matrix[anchor, positive] + closer = negatives[negatives <= gap] + violations.append(torch.sigmoid(gap - closer).sum()) + if not violations: + return None + return torch.atan(torch.stack(violations)).mean() + + +@pytest.mark.parametrize("distance", ["EU", "Cosine"]) +@pytest.mark.parametrize("n_labels", [2, 5]) +def test_ranking_loss_matches_the_authors_loop(distance, n_labels): + """The dense form is a rewrite for speed, so it must equal the loop exactly. + + Shaped like a real batch: the archive settings give 4 * (2 * 5 + 1) = 44 + embeddings with every case repeated 11 times. + """ + generator = torch.Generator().manual_seed(3) + embeddings = torch.randn(44, 16, generator=generator) + labels = torch.randint(0, n_labels, (4,), generator=generator).repeat(11) + + expected = _loop_ranking_loss(embeddings, labels, distance) + actual = _ranking_loss(embeddings, labels, distance) + if expected is None: + assert actual is None + else: + assert float(actual) == pytest.approx(float(expected), rel=1e-6) + + +def test_ranking_loss_gradient_matches_the_authors_loop(): + """Equal values are not enough; the training signal must match too.""" + generator = torch.Generator().manual_seed(5) + base = torch.randn(22, 8, generator=generator) + labels = torch.tensor([0, 1]).repeat(11) + + grads = [] + for loss_fn in (_loop_ranking_loss, _ranking_loss): + embeddings = base.clone().requires_grad_(True) + loss_fn(embeddings, labels, "EU").backward() + grads.append(embeddings.grad) + assert torch.allclose(grads[0], grads[1], atol=1e-6) + + def test_ranking_loss_is_none_without_negatives(): """A single-class batch has nothing to rank, where the original raises.""" embeddings = torch.tensor([[0.0], [1.0]]) @@ -160,3 +218,27 @@ def test_shape_is_checked_at_predict(): clf.predict(np.random.random((4, 5, 40))) with pytest.raises(ValueError, match="length"): clf.predict(np.random.random((4, 2, 17))) + + +def test_svm_grid_does_not_carry_platt_scaling(): + """The searched estimator must match the authors' probability=False. + + libsvm fits Platt scaling by an internal five-fold cross-validation inside + every ``fit``, so carrying probability=True into the grid turns a ten-value + search over five folds from about 50 SVC trainings into 300. On the archive + collections that alone ran the probe past a 60 hour job. + """ + clf = RankSCLClassifier(probe="svm") + clf.n_classes_ = 2 + assert clf._build_probe(n_cases=100, seed=0).estimator.probability is False + assert clf._build_probe(n_cases=10, seed=0).probability is False + + +def test_svm_probe_ends_up_with_probabilities(): + """predict_proba is part of the aeon interface, so the fitted probe needs it.""" + X, y = _data(n_cases=60) + clf = RankSCLClassifier(**{**SMALL, "probe": "svm"}).fit(X, y) + assert clf.probe_.probability is True + proba = clf.predict_proba(X) + assert proba.shape == (len(y), clf.n_classes_) + assert np.allclose(proba.sum(axis=1), 1) diff --git a/multiverse/experiments/tables.py b/multiverse/experiments/tables.py index d00508b..26f0c1d 100644 --- a/multiverse/experiments/tables.py +++ b/multiverse/experiments/tables.py @@ -344,6 +344,118 @@ def _missing_by_estimator(frames, estimators, common, datasets): return missing +# Estimators held out of the published tables, with the reason each is held out. +# They are named on every page rather than quietly dropped, since an unexplained +# absence is the thing this archive exists to argue against. +WITHHELD_ESTIMATORS = { + "LiteTIME": ( + "LITE is a univariate architecture. The multivariate variant of the same " + "method is listed here as LITETime-MV" + ), + "FreshPRINCE": ( + "cannot complete the archive at the memory available: recorded OOM at " + "128 GB after eight attempts each on FaceDetection, FordChallenge and " + "Skoda, and 38 on Tiselac" + ), + "1NN-DTW": ( + "cannot complete the archive within the walltime available: exceeded the " + "limit on BIDMC32HR_disc, with no result recorded for BIDMC32SpO2_disc" + ), + "DisjointCNN-Aeon": ( + "aeon's implementation applies a Permute after the final block, so its " + "pooling reduces the wrong axes and the classifier head receives one " + "feature instead of 64 (aeon issue #3775). Held as evidence for that " + "issue; the port of the same method reports as DisjointCNN" + ), +} + + +# Datasets held out of the collection, and why. Distinct from the datasets an +# individual estimator is missing: these are ones no amount of scheduling will +# close, so leaving them in the denominator only makes the scored fraction look +# like a queue that is still draining. +DEFERRED_DATASETS = { + "AustraliaRainfall_disc": ( + "112186 cases, and three estimators fail on it for reasons compute cannot " + "fix. RDST and ROCKET hit LAPACK integer overflow in RidgeClassifierCV's " + "SVD, aeon issue 3738, after 14 and 12 attempts; MRHydra exhausted 128 GB " + "over 13" + ), + "PenDigits": ( + "the series are length 8 and MRHydra requires at least 9, so the dataset " + "cannot complete while MRHydra is a column" + ), +} + + +def _deferred_html() -> str: + """Name the datasets held out of the collection, and why.""" + if not DEFERRED_DATASETS: + return "" + items = "".join( + f"
  • {escape(name)} — {escape(reason)}.
    • " + for name, reason in DEFERRED_DATASETS.items() + ) + return ( + "

    Datasets not included

    " + f'' + '

    Held out of the collection rather than reported as ' + "missing, because no scheduling closes them. Results that do exist for " + "them remain in the repository.

    " + ) + + +# Estimators listed in the tables that carry a caveat a reader needs in order to +# read the row correctly. Kept beside WITHHELD_ESTIMATORS so both the inclusions +# and the exclusions state their reasoning in the same place. +ESTIMATOR_NOTES = { + "XCM": ( + "run at fixed parameters, a single fit at window 0.8 with batch 32, not " + "the per-dataset cross-validated search over window and batch size that " + "the paper describes. The search was run and did not pay: across the 65 " + "shared datasets it was 0.017 mean accuracy worse, 31 wins to 31 with 3 " + "ties, Wilcoxon p = 0.63. On the 14 datasets where the search selected " + "0.8, the window used here, the two runs still differed by 0.11 mean " + "absolute accuracy and by as much as 0.48, so at one resample XCM's " + "run-to-run variance is larger than the effect the search is tuning for" + ), +} + + +def _estimator_notes_html(listed) -> str: + """Render the caveats attached to estimators that are in the table.""" + items = "".join( + f"
  • {escape(name)} — {escape(reason)}.
    • " + for name, reason in ESTIMATOR_NOTES.items() + if name in listed + ) + if not items: + return "" + return ( + "

    Notes on listed estimators

    " + f'' + ) + + +def _withheld_html() -> str: + """Name the estimators kept out of the table, and why.""" + if not WITHHELD_ESTIMATORS: + return "" + items = "".join( + f"
  • {escape(name)} — {escape(reason)}.
    • " + for name, reason in WITHHELD_ESTIMATORS.items() + ) + return ( + "

    Estimators not listed

    " + f'' + '

    Their results remain in the repository under ' + "results/multiverse/. Removing an estimator that cannot " + "finish the archive returns the datasets it alone was missing to every " + "other estimator, which is why the scored count above is larger than " + "the number of datasets any single run completed.

    " + ) + + def _excluded_html(missing, reasons, common, dropped) -> str: """Render a one-line summary of what each estimator is missing.""" lookup = { @@ -711,6 +823,9 @@ def leaderboard( ) parts.append(_excluded_html(missing, reasons, common, dropped)) + parts.append(_estimator_notes_html(set(summary.index))) + parts.append(_deferred_html()) + parts.append(_withheld_html()) parts.append( _snippet_html( datasets_expr if datasets_expr is not None else _describe_datasets(datasets), @@ -1150,20 +1265,30 @@ def main() -> None: """Build the Multiverse-core leaderboard. Uses every estimator with results in the repository, including the Dummy - baseline, over the Multiverse-core datasets all of them have results for. - - DisjointCNN-Aeon is held back. Those results are around 20 accuracy points - below the authors' published numbers on all 23 shared datasets, because - aeon's network applies a Permute after the final block and its pooling then - reduces the wrong axes, leaving the classifier head one feature instead of - 64 (aeon issue #3775). They are kept as evidence for that issue rather than - deleted, but listing them would read as a claim about the method. The - Multiverse port of the same method reports under DisjointCNN. + baseline, over the Multiverse-core datasets all of them have results for, + except those named in WITHHELD_ESTIMATORS. That set is rendered onto every + page by _withheld_html, so an omission is stated rather than inferred. + + Three of the four are held back because they cannot finish the archive, and + scoring on the intersection makes an estimator's gaps everyone's: LiteTIME + is univariate, with LITETime-MV the multivariate variant of the same method, + while FreshPRINCE and 1NN-DTW exhaust the available memory and walltime + respectively. Removing them returns five datasets to the scored set. The + fourth, DisjointCNN-Aeon, completes the archive but scores around 20 + accuracy points below the published numbers because of aeon issue #3775; it + is kept as evidence for that issue, and the port reports as DisjointCNN. """ from aeon.datasets.tsc_datasets import UEA, multiverse_core - datasets = sorted(multiverse_core) - estimators = available_estimators(exclude=("DisjointCNN-Aeon",)) + # the same deferral applies to both tables: a dataset MRHydra cannot fit + # is no more completable inside the UEA 30 than inside Multiverse-core + uea_datasets = [name for name in sorted(UEA) if name not in DEFERRED_DATASETS] + + datasets = [ + name for name in sorted(multiverse_core) + if name not in DEFERRED_DATASETS + ] + estimators = available_estimators(exclude=tuple(WITHHELD_ESTIMATORS)) print(f"estimators: {', '.join(estimators)}") path = leaderboard( @@ -1187,7 +1312,7 @@ def main() -> None: # comparing against the literature actually needs. It is a subset view of # the same runs, not a separate experiment. uea_path = leaderboard( - sorted(UEA), + uea_datasets, estimators, sort_by="accuracy", title="UEA leaderboard", @@ -1197,7 +1322,7 @@ def main() -> None: docs = Path(__file__).resolve().parents[2] / "docs" / "leaderboard.md" uea_table = leaderboard_markdown( - sorted(UEA), estimators, sort_by="accuracy", collection="UEA" + uea_datasets, estimators, sort_by="accuracy", collection="UEA" ) if write_markdown_table(docs, uea_table, marker="UEA_LEADERBOARD"): print(f"updated the UEA table in {docs}") diff --git a/results/multiverse/ConvTran/ConvTran_accuracy.csv b/results/multiverse/ConvTran/ConvTran_accuracy.csv index e97675d..989ad31 100644 --- a/results/multiverse/ConvTran/ConvTran_accuracy.csv +++ b/results/multiverse/ConvTran/ConvTran_accuracy.csv @@ -20,6 +20,7 @@ Cricket,0.9722222222222222 CrowdSourced,0.7077303212142605 DuckDuckGeese,0.52 ERing,0.8703703703703703 +EmoPain,0.6816901408450704 Epilepsy,0.9565217391304348 EthanolConcentration,0.3155893536121673 EyesOpenShut,0.5 diff --git a/results/multiverse/ConvTran/ConvTran_auroc.csv b/results/multiverse/ConvTran/ConvTran_auroc.csv index 294de93..9974c6b 100644 --- a/results/multiverse/ConvTran/ConvTran_auroc.csv +++ b/results/multiverse/ConvTran/ConvTran_auroc.csv @@ -20,6 +20,7 @@ Cricket,0.9997895622895623 CrowdSourced,0.7640089171441217 DuckDuckGeese,0.8395 ERing,0.9809876543209877 +EmoPain,0.929164159088972 Epilepsy,0.9950667516584434 EthanolConcentration,0.5837762309747023 EyesOpenShut,0.6235827664399094 diff --git a/results/multiverse/ConvTran/ConvTran_balacc.csv b/results/multiverse/ConvTran/ConvTran_balacc.csv index 8aceeba..c49b804 100644 --- a/results/multiverse/ConvTran/ConvTran_balacc.csv +++ b/results/multiverse/ConvTran/ConvTran_balacc.csv @@ -20,6 +20,7 @@ Cricket,0.9722222222222223 CrowdSourced,0.7078297130485749 DuckDuckGeese,0.52 ERing,0.8703703703703703 +EmoPain,0.5033469998194992 Epilepsy,0.9576709062003179 EthanolConcentration,0.3152680652680653 EyesOpenShut,0.5 diff --git a/results/multiverse/ConvTran/ConvTran_f1.csv b/results/multiverse/ConvTran/ConvTran_f1.csv index 7b9544a..1a16121 100644 --- a/results/multiverse/ConvTran/ConvTran_f1.csv +++ b/results/multiverse/ConvTran/ConvTran_f1.csv @@ -20,6 +20,7 @@ Cricket,0.972125097125097 CrowdSourced,0.771900826446281 DuckDuckGeese,0.5148282828282827 ERing,0.866562408896808 +EmoPain,0.7463900201539311 Epilepsy,0.9563529167315983 EthanolConcentration,0.31328733858593627 EyesOpenShut,0.6557377049180327 diff --git a/results/multiverse/ConvTran/ConvTran_logloss.csv b/results/multiverse/ConvTran/ConvTran_logloss.csv index 428e14f..63343f4 100644 --- a/results/multiverse/ConvTran/ConvTran_logloss.csv +++ b/results/multiverse/ConvTran/ConvTran_logloss.csv @@ -20,6 +20,7 @@ Cricket,0.23399420852414396 CrowdSourced,6.732414904792953 DuckDuckGeese,1.228894854426434 ERing,0.8160938131974115 +EmoPain,1.2659174142349616 Epilepsy,0.15984163724218461 EthanolConcentration,1.4544173097170774 EyesOpenShut,0.6982935135605143 diff --git a/results/multiverse/ConvTran/ConvTran_sensitivity.csv b/results/multiverse/ConvTran/ConvTran_sensitivity.csv index d5acff0..274c138 100644 --- a/results/multiverse/ConvTran/ConvTran_sensitivity.csv +++ b/results/multiverse/ConvTran/ConvTran_sensitivity.csv @@ -20,6 +20,7 @@ Cricket,0.9722222222222222 CrowdSourced,0.989406779661017 DuckDuckGeese,0.52 ERing,0.8703703703703703 +EmoPain,0.6816901408450704 Epilepsy,0.9565217391304348 EthanolConcentration,0.3155893536121673 EyesOpenShut,0.9523809523809523 diff --git a/results/multiverse/ConvTran/ConvTran_specificity.csv b/results/multiverse/ConvTran/ConvTran_specificity.csv index 1ec5e1d..70a8817 100644 --- a/results/multiverse/ConvTran/ConvTran_specificity.csv +++ b/results/multiverse/ConvTran/ConvTran_specificity.csv @@ -20,6 +20,7 @@ Cricket,0.9722222222222222 CrowdSourced,0.4262526464361327 DuckDuckGeese,0.52 ERing,0.8703703703703703 +EmoPain,0.6816901408450704 Epilepsy,0.9565217391304348 EthanolConcentration,0.3155893536121673 EyesOpenShut,0.047619047619047616 diff --git a/results/multiverse/HC2/HC2_accuracy.csv b/results/multiverse/HC2/HC2_accuracy.csv index 505dce2..dc2bbce 100644 --- a/results/multiverse/HC2/HC2_accuracy.csv +++ b/results/multiverse/HC2/HC2_accuracy.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.8567774936061381 AsphaltRegularityCoordinates,0.9720372836218375 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.7653958944281525 AutomotiveRoadTrials,0.8311688311688312 BIDMC32HR_disc,0.9362234264276782 BIDMC32SpO2_disc,0.6627761567319717 @@ -59,5 +60,6 @@ TactileTextureRecognition,1.0 UCDHE-Rowing-MC,0.7886363636363637 UCIActivity,0.9862557267805081 UIPRMD-DS-C,1.0 +USCActivity,0.74726609963548 UWaveGestureLibrary,0.940625 WISDM,0.8810121692099672 diff --git a/results/multiverse/HC2/HC2_auroc.csv b/results/multiverse/HC2/HC2_auroc.csv index 55794f1..0abee34 100644 --- a/results/multiverse/HC2/HC2_auroc.csv +++ b/results/multiverse/HC2/HC2_auroc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9997337962962962 AsphaltObstaclesCoordinates,0.9734602114863985 AsphaltRegularityCoordinates,0.9971270483081507 AtrialFibrillation,0.4733333333333334 +AustraliaRainfall_disc,0.8493354969968424 AutomotiveRoadTrials,0.9346642468239564 BIDMC32HR_disc,0.9829130420375323 BIDMC32SpO2_disc,0.47559998566587147 @@ -59,5 +60,6 @@ TactileTextureRecognition,1.0 UCDHE-Rowing-MC,0.9547150195459019 UCIActivity,0.9998589849536378 UIPRMD-DS-C,1.0 +USCActivity,0.9729512907764547 UWaveGestureLibrary,0.9972767857142857 WISDM,0.9826859231636664 diff --git a/results/multiverse/HC2/HC2_balacc.csv b/results/multiverse/HC2/HC2_balacc.csv index f99d961..c7a3756 100644 --- a/results/multiverse/HC2/HC2_balacc.csv +++ b/results/multiverse/HC2/HC2_balacc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.8583356728366263 AsphaltRegularityCoordinates,0.9723238987018514 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.3586544279388035 AutomotiveRoadTrials,0.7286751361161524 BIDMC32HR_disc,0.9052044609665427 BIDMC32SpO2_disc,0.46813429371994525 @@ -59,5 +60,6 @@ TactileTextureRecognition,1.0 UCDHE-Rowing-MC,0.7924444444444444 UCIActivity,0.9866506989269137 UIPRMD-DS-C,1.0 +USCActivity,0.7693872599385568 UWaveGestureLibrary,0.940625 WISDM,0.5943050131981966 diff --git a/results/multiverse/HC2/HC2_f1.csv b/results/multiverse/HC2/HC2_f1.csv index 515f058..25b3f60 100644 --- a/results/multiverse/HC2/HC2_f1.csv +++ b/results/multiverse/HC2/HC2_f1.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933217391304349 AsphaltObstaclesCoordinates,0.8558648505702661 AsphaltRegularityCoordinates,0.9721854304635762 AtrialFibrillation,0.3206349206349206 +AustraliaRainfall_disc,0.7390104467553876 AutomotiveRoadTrials,0.6060606060606061 BIDMC32HR_disc,0.9324272452628007 BIDMC32SpO2_disc,0.026474127557160047 @@ -59,5 +60,6 @@ TactileTextureRecognition,1.0 UCDHE-Rowing-MC,0.7867549047609365 UCIActivity,0.9861188502098307 UIPRMD-DS-C,1.0 +USCActivity,0.730600284953102 UWaveGestureLibrary,0.9403506309738556 WISDM,0.8693085096317551 diff --git a/results/multiverse/HC2/HC2_logloss.csv b/results/multiverse/HC2/HC2_logloss.csv index 9cf67fd..cee6115 100644 --- a/results/multiverse/HC2/HC2_logloss.csv +++ b/results/multiverse/HC2/HC2_logloss.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.1862674632785058 AsphaltObstaclesCoordinates,0.42799098466560875 AsphaltRegularityCoordinates,0.0904744731571356 AtrialFibrillation,1.2255595169970868 +AustraliaRainfall_disc,0.5371241660153839 AutomotiveRoadTrials,0.33157119408461455 BIDMC32HR_disc,0.3343770378009419 BIDMC32SpO2_disc,0.6333093836657039 @@ -59,5 +60,6 @@ TactileTextureRecognition,0.08844470538136215 UCDHE-Rowing-MC,0.7059100588662615 UCIActivity,0.20648496301861372 UIPRMD-DS-C,0.22566949629880087 +USCActivity,0.845283723163391 UWaveGestureLibrary,0.3302529064972539 WISDM,0.4056622750938158 diff --git a/results/multiverse/HC2/HC2_sensitivity.csv b/results/multiverse/HC2/HC2_sensitivity.csv index 7365f41..f46f301 100644 --- a/results/multiverse/HC2/HC2_sensitivity.csv +++ b/results/multiverse/HC2/HC2_sensitivity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.8567774936061381 AsphaltRegularityCoordinates,0.9918918918918919 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.7653958944281525 AutomotiveRoadTrials,0.5263157894736842 BIDMC32HR_disc,0.9362234264276782 BIDMC32SpO2_disc,0.016105417276720352 @@ -59,5 +60,6 @@ TactileTextureRecognition,1.0 UCDHE-Rowing-MC,0.7886363636363637 UCIActivity,0.9862557267805081 UIPRMD-DS-C,1.0 +USCActivity,0.74726609963548 UWaveGestureLibrary,0.940625 WISDM,0.8810121692099672 diff --git a/results/multiverse/HC2/HC2_specificity.csv b/results/multiverse/HC2/HC2_specificity.csv index 87cdb20..1f1f8a5 100644 --- a/results/multiverse/HC2/HC2_specificity.csv +++ b/results/multiverse/HC2/HC2_specificity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.8567774936061381 AsphaltRegularityCoordinates,0.952755905511811 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.7653958944281525 AutomotiveRoadTrials,0.9310344827586207 BIDMC32HR_disc,0.9362234264276782 BIDMC32SpO2_disc,0.9201631701631702 @@ -59,5 +60,6 @@ TactileTextureRecognition,1.0 UCDHE-Rowing-MC,0.7886363636363637 UCIActivity,0.9862557267805081 UIPRMD-DS-C,1.0 +USCActivity,0.74726609963548 UWaveGestureLibrary,0.940625 WISDM,0.8810121692099672 diff --git a/results/multiverse/LiteTIME/LiteTIME_accuracy.csv b/results/multiverse/LiteTIME/LiteTIME_accuracy.csv index 952078f..46a0c64 100644 --- a/results/multiverse/LiteTIME/LiteTIME_accuracy.csv +++ b/results/multiverse/LiteTIME/LiteTIME_accuracy.csv @@ -60,5 +60,6 @@ Tiselac,0.8060161779575329 UCDHE-Rowing-MC,0.7204545454545455 UCIActivity,0.9954185755935027 UIPRMD-DS-C,0.6944444444444444 +USCActivity,0.68408262454435 UWaveGestureLibrary,0.9125 WISDM,0.8841027622175005 diff --git a/results/multiverse/LiteTIME/LiteTIME_auroc.csv b/results/multiverse/LiteTIME/LiteTIME_auroc.csv index ba41526..22ecbe1 100644 --- a/results/multiverse/LiteTIME/LiteTIME_auroc.csv +++ b/results/multiverse/LiteTIME/LiteTIME_auroc.csv @@ -60,5 +60,6 @@ Tiselac,0.9563846693108311 UCDHE-Rowing-MC,0.9370878584849173 UCIActivity,0.9999714242454316 UIPRMD-DS-C,0.904320987654321 +USCActivity,0.9626362091493008 UWaveGestureLibrary,0.9882924107142858 WISDM,0.9619285966824769 diff --git a/results/multiverse/LiteTIME/LiteTIME_balacc.csv b/results/multiverse/LiteTIME/LiteTIME_balacc.csv index 3674133..dfabe02 100644 --- a/results/multiverse/LiteTIME/LiteTIME_balacc.csv +++ b/results/multiverse/LiteTIME/LiteTIME_balacc.csv @@ -60,5 +60,6 @@ Tiselac,0.6281380688710096 UCDHE-Rowing-MC,0.7287539682539683 UCIActivity,0.9956138869182348 UIPRMD-DS-C,0.6944444444444444 +USCActivity,0.6918854089485373 UWaveGestureLibrary,0.9125000000000001 WISDM,0.5602390621754619 diff --git a/results/multiverse/LiteTIME/LiteTIME_f1.csv b/results/multiverse/LiteTIME/LiteTIME_f1.csv index aad7223..0c28c91 100644 --- a/results/multiverse/LiteTIME/LiteTIME_f1.csv +++ b/results/multiverse/LiteTIME/LiteTIME_f1.csv @@ -60,5 +60,6 @@ Tiselac,0.796096603811563 UCDHE-Rowing-MC,0.7193501397499977 UCIActivity,0.99540996175598 UIPRMD-DS-C,0.5925925925925926 +USCActivity,0.670932205831197 UWaveGestureLibrary,0.9106125154705872 WISDM,0.8634803950393592 diff --git a/results/multiverse/LiteTIME/LiteTIME_logloss.csv b/results/multiverse/LiteTIME/LiteTIME_logloss.csv index 9f5c8bf..2f29f2a 100644 --- a/results/multiverse/LiteTIME/LiteTIME_logloss.csv +++ b/results/multiverse/LiteTIME/LiteTIME_logloss.csv @@ -60,5 +60,6 @@ Tiselac,2.0719247088377974 UCDHE-Rowing-MC,1.058103035533595 UCIActivity,0.014448260333737011 UIPRMD-DS-C,1.8159978680400188 +USCActivity,1.960099187747739 UWaveGestureLibrary,0.3466050162423572 WISDM,1.6771960989756909 diff --git a/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv b/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv index 51aa807..3e52cd3 100644 --- a/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv +++ b/results/multiverse/LiteTIME/LiteTIME_sensitivity.csv @@ -60,5 +60,6 @@ Tiselac,0.8060161779575329 UCDHE-Rowing-MC,0.7204545454545455 UCIActivity,0.9954185755935027 UIPRMD-DS-C,0.4444444444444444 +USCActivity,0.68408262454435 UWaveGestureLibrary,0.9125 WISDM,0.8841027622175005 diff --git a/results/multiverse/LiteTIME/LiteTIME_specificity.csv b/results/multiverse/LiteTIME/LiteTIME_specificity.csv index c096e93..1c0cb2e 100644 --- a/results/multiverse/LiteTIME/LiteTIME_specificity.csv +++ b/results/multiverse/LiteTIME/LiteTIME_specificity.csv @@ -60,5 +60,6 @@ Tiselac,0.8060161779575329 UCDHE-Rowing-MC,0.7204545454545455 UCIActivity,0.9954185755935027 UIPRMD-DS-C,0.9444444444444444 +USCActivity,0.68408262454435 UWaveGestureLibrary,0.9125 WISDM,0.8841027622175005 diff --git a/results/multiverse/PatchMTSC/PatchMTSC_accuracy.csv b/results/multiverse/PatchMTSC/PatchMTSC_accuracy.csv index 6f9c357..8089385 100644 --- a/results/multiverse/PatchMTSC/PatchMTSC_accuracy.csv +++ b/results/multiverse/PatchMTSC/PatchMTSC_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.6925520649488175 DuckDuckGeese,0.44 ERing,0.9481481481481482 EigenWorms,0.4198473282442748 +EmoPain,0.847887323943662 Epilepsy,0.9565217391304348 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.5 diff --git a/results/multiverse/PatchMTSC/PatchMTSC_auroc.csv b/results/multiverse/PatchMTSC/PatchMTSC_auroc.csv index 7402a13..140b637 100644 --- a/results/multiverse/PatchMTSC/PatchMTSC_auroc.csv +++ b/results/multiverse/PatchMTSC/PatchMTSC_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7606923993955559 DuckDuckGeese,0.7930000000000001 ERing,0.9961152263374485 EigenWorms,0.7070538599343066 +EmoPain,0.9386338989936039 Epilepsy,0.9970363257061493 EthanolConcentration,0.5717449338495062 EyesOpenShut,0.6462585034013606 diff --git a/results/multiverse/PatchMTSC/PatchMTSC_balacc.csv b/results/multiverse/PatchMTSC/PatchMTSC_balacc.csv index 150da0f..77498ae 100644 --- a/results/multiverse/PatchMTSC/PatchMTSC_balacc.csv +++ b/results/multiverse/PatchMTSC/PatchMTSC_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.692643605293271 DuckDuckGeese,0.43999999999999995 ERing,0.9481481481481481 EigenWorms,0.2 +EmoPain,0.6540058275959877 Epilepsy,0.9576709062003179 EthanolConcentration,0.2770979020979021 EyesOpenShut,0.5 diff --git a/results/multiverse/PatchMTSC/PatchMTSC_f1.csv b/results/multiverse/PatchMTSC/PatchMTSC_f1.csv index 6d54631..6d4f204 100644 --- a/results/multiverse/PatchMTSC/PatchMTSC_f1.csv +++ b/results/multiverse/PatchMTSC/PatchMTSC_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7558172133445472 DuckDuckGeese,0.4236518273371055 ERing,0.947928149760099 EigenWorms,0.24963895192902827 +EmoPain,0.8586444160520196 Epilepsy,0.9565217391304348 EthanolConcentration,0.26018604446846005 EyesOpenShut,0.6666666666666666 diff --git a/results/multiverse/PatchMTSC/PatchMTSC_logloss.csv b/results/multiverse/PatchMTSC/PatchMTSC_logloss.csv index 921e356..aa0214f 100644 --- a/results/multiverse/PatchMTSC/PatchMTSC_logloss.csv +++ b/results/multiverse/PatchMTSC/PatchMTSC_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,3.604783115680626 DuckDuckGeese,1.4864029996306982 ERing,0.17473236669240275 EigenWorms,1.4752600925015948 +EmoPain,0.5201607916476759 Epilepsy,0.13011442553672609 EthanolConcentration,1.4148892335730348 EyesOpenShut,0.7119754317202979 diff --git a/results/multiverse/PatchMTSC/PatchMTSC_sensitivity.csv b/results/multiverse/PatchMTSC/PatchMTSC_sensitivity.csv index 548c8f8..7bfa757 100644 --- a/results/multiverse/PatchMTSC/PatchMTSC_sensitivity.csv +++ b/results/multiverse/PatchMTSC/PatchMTSC_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.9519774011299436 DuckDuckGeese,0.44 ERing,0.9481481481481482 EigenWorms,0.4198473282442748 +EmoPain,0.847887323943662 Epilepsy,0.9565217391304348 EthanolConcentration,0.27756653992395436 EyesOpenShut,1.0 diff --git a/results/multiverse/PatchMTSC/PatchMTSC_specificity.csv b/results/multiverse/PatchMTSC/PatchMTSC_specificity.csv index 997eee6..fc09126 100644 --- a/results/multiverse/PatchMTSC/PatchMTSC_specificity.csv +++ b/results/multiverse/PatchMTSC/PatchMTSC_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.43330980945659847 DuckDuckGeese,0.44 ERing,0.9481481481481482 EigenWorms,0.4198473282442748 +EmoPain,0.847887323943662 Epilepsy,0.9565217391304348 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.0 diff --git a/results/multiverse/STSF/STSF_accuracy.csv b/results/multiverse/STSF/STSF_accuracy.csv index fe3329b..7abe851 100644 --- a/results/multiverse/STSF/STSF_accuracy.csv +++ b/results/multiverse/STSF/STSF_accuracy.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.97 AsphaltObstaclesCoordinates,0.8005115089514067 AsphaltRegularityCoordinates,0.9866844207723036 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.777687652087103 AutomotiveRoadTrials,0.7662337662337663 BIDMC32HR_disc,0.7765735723218008 BIDMC32SpO2_disc,0.7061275531471446 diff --git a/results/multiverse/STSF/STSF_auroc.csv b/results/multiverse/STSF/STSF_auroc.csv index bd20131..bb5cc29 100644 --- a/results/multiverse/STSF/STSF_auroc.csv +++ b/results/multiverse/STSF/STSF_auroc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9941145833333334 AsphaltObstaclesCoordinates,0.9604597471111511 AsphaltRegularityCoordinates,0.9992551606724835 AtrialFibrillation,0.4466666666666666 +AustraliaRainfall_disc,0.8604056630583765 AutomotiveRoadTrials,0.8371143375680581 BIDMC32HR_disc,0.8338753179973307 BIDMC32SpO2_disc,0.39743717726880246 diff --git a/results/multiverse/STSF/STSF_balacc.csv b/results/multiverse/STSF/STSF_balacc.csv index 4fb9ae6..5058755 100644 --- a/results/multiverse/STSF/STSF_balacc.csv +++ b/results/multiverse/STSF/STSF_balacc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.97 AsphaltObstaclesCoordinates,0.7992539696963121 AsphaltRegularityCoordinates,0.9868376250266014 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.3857846346010465 AutomotiveRoadTrials,0.7386569872958257 BIDMC32HR_disc,0.673974445786005 BIDMC32SpO2_disc,0.5892192848635015 diff --git a/results/multiverse/STSF/STSF_f1.csv b/results/multiverse/STSF/STSF_f1.csv index 7c52cdf..bdd372c 100644 --- a/results/multiverse/STSF/STSF_f1.csv +++ b/results/multiverse/STSF/STSF_f1.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9699592928810322 AsphaltObstaclesCoordinates,0.7993387871560758 AsphaltRegularityCoordinates,0.9866310160427807 AtrialFibrillation,0.2222222222222222 +AustraliaRainfall_disc,0.7589204974912698 AutomotiveRoadTrials,0.5909090909090909 BIDMC32HR_disc,0.7450311536797996 BIDMC32SpO2_disc,0.3810359964881475 diff --git a/results/multiverse/STSF/STSF_logloss.csv b/results/multiverse/STSF/STSF_logloss.csv index b2950b7..d921132 100644 --- a/results/multiverse/STSF/STSF_logloss.csv +++ b/results/multiverse/STSF/STSF_logloss.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.5806244176556595 AsphaltObstaclesCoordinates,0.4807203932976841 AsphaltRegularityCoordinates,0.05028364452186321 AtrialFibrillation,1.2513045236837814 +AustraliaRainfall_disc,0.5288623833003138 AutomotiveRoadTrials,0.43863329796308087 BIDMC32HR_disc,0.6105505748824706 BIDMC32SpO2_disc,2.605049809014673 diff --git a/results/multiverse/STSF/STSF_sensitivity.csv b/results/multiverse/STSF/STSF_sensitivity.csv index 91abb58..7f24f61 100644 --- a/results/multiverse/STSF/STSF_sensitivity.csv +++ b/results/multiverse/STSF/STSF_sensitivity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.97 AsphaltObstaclesCoordinates,0.8005115089514067 AsphaltRegularityCoordinates,0.9972972972972973 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.777687652087103 AutomotiveRoadTrials,0.6842105263157895 BIDMC32HR_disc,0.7765735723218008 BIDMC32SpO2_disc,0.31771595900439237 diff --git a/results/multiverse/STSF/STSF_specificity.csv b/results/multiverse/STSF/STSF_specificity.csv index 554ce00..05a328f 100644 --- a/results/multiverse/STSF/STSF_specificity.csv +++ b/results/multiverse/STSF/STSF_specificity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.97 AsphaltObstaclesCoordinates,0.8005115089514067 AsphaltRegularityCoordinates,0.9763779527559056 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.777687652087103 AutomotiveRoadTrials,0.7931034482758621 BIDMC32HR_disc,0.7765735723218008 BIDMC32SpO2_disc,0.8607226107226107 diff --git a/results/multiverse/Summary/Summary_accuracy.csv b/results/multiverse/Summary/Summary_accuracy.csv index f793878..25b00c1 100644 --- a/results/multiverse/Summary/Summary_accuracy.csv +++ b/results/multiverse/Summary/Summary_accuracy.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9433333333333334 AsphaltObstaclesCoordinates,0.6675191815856778 AsphaltRegularityCoordinates,0.933422103861518 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.7398556602400116 AutomotiveRoadTrials,0.8311688311688312 BIDMC32HR_disc,0.45268862025844103 BIDMC32SpO2_disc,0.7144643601500625 diff --git a/results/multiverse/Summary/Summary_auroc.csv b/results/multiverse/Summary/Summary_auroc.csv index cb6462c..405eec3 100644 --- a/results/multiverse/Summary/Summary_auroc.csv +++ b/results/multiverse/Summary/Summary_auroc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9968344907407407 AsphaltObstaclesCoordinates,0.8889753047321435 AsphaltRegularityCoordinates,0.982499822657303 AtrialFibrillation,0.42 +AustraliaRainfall_disc,0.8075750991176159 AutomotiveRoadTrials,0.9142468239564429 BIDMC32HR_disc,0.4049085359059449 BIDMC32SpO2_disc,0.36119700211940325 diff --git a/results/multiverse/Summary/Summary_balacc.csv b/results/multiverse/Summary/Summary_balacc.csv index 793e03f..59ea13e 100644 --- a/results/multiverse/Summary/Summary_balacc.csv +++ b/results/multiverse/Summary/Summary_balacc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9433333333333332 AsphaltObstaclesCoordinates,0.6645937424276946 AsphaltRegularityCoordinates,0.9335638788394693 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.3453411893376324 AutomotiveRoadTrials,0.8348457350272231 BIDMC32HR_disc,0.23190262652146063 BIDMC32SpO2_disc,0.57918198200043 diff --git a/results/multiverse/Summary/Summary_f1.csv b/results/multiverse/Summary/Summary_f1.csv index 61271a1..b4a5cd8 100644 --- a/results/multiverse/Summary/Summary_f1.csv +++ b/results/multiverse/Summary/Summary_f1.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9432585791020576 AsphaltObstaclesCoordinates,0.6688875038207587 AsphaltRegularityCoordinates,0.9331550802139037 AtrialFibrillation,0.32255892255892255 +AustraliaRainfall_disc,0.7155256832569299 AutomotiveRoadTrials,0.7111111111111111 BIDMC32HR_disc,0.42492299231715364 BIDMC32SpO2_disc,0.3457497612225406 diff --git a/results/multiverse/Summary/Summary_logloss.csv b/results/multiverse/Summary/Summary_logloss.csv index 065580f..6bddf18 100644 --- a/results/multiverse/Summary/Summary_logloss.csv +++ b/results/multiverse/Summary/Summary_logloss.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,1.016944410988233 AsphaltObstaclesCoordinates,0.7842480946370547 AsphaltRegularityCoordinates,0.18681677354138634 AtrialFibrillation,1.471609282128226 +AustraliaRainfall_disc,0.6205058384526263 AutomotiveRoadTrials,0.39182235767704837 BIDMC32HR_disc,1.1257454612821087 BIDMC32SpO2_disc,4.524440233912232 diff --git a/results/multiverse/Summary/Summary_sensitivity.csv b/results/multiverse/Summary/Summary_sensitivity.csv index bc27a94..ac1d396 100644 --- a/results/multiverse/Summary/Summary_sensitivity.csv +++ b/results/multiverse/Summary/Summary_sensitivity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9433333333333334 AsphaltObstaclesCoordinates,0.6675191815856778 AsphaltRegularityCoordinates,0.9432432432432433 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.7398556602400116 AutomotiveRoadTrials,0.8421052631578947 BIDMC32HR_disc,0.45268862025844103 BIDMC32SpO2_disc,0.26500732064421667 diff --git a/results/multiverse/Summary/Summary_specificity.csv b/results/multiverse/Summary/Summary_specificity.csv index 3105da8..d7bbc3f 100644 --- a/results/multiverse/Summary/Summary_specificity.csv +++ b/results/multiverse/Summary/Summary_specificity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9433333333333334 AsphaltObstaclesCoordinates,0.6675191815856778 AsphaltRegularityCoordinates,0.9238845144356955 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.7398556602400116 AutomotiveRoadTrials,0.8275862068965517 BIDMC32HR_disc,0.45268862025844103 BIDMC32SpO2_disc,0.8933566433566433 diff --git a/results/multiverse/TDE/TDE_accuracy.csv b/results/multiverse/TDE/TDE_accuracy.csv index 69714c8..c734d40 100644 --- a/results/multiverse/TDE/TDE_accuracy.csv +++ b/results/multiverse/TDE/TDE_accuracy.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.8286445012787724 AsphaltRegularityCoordinates,0.9507323568575233 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.7201389322185479 AutomotiveRoadTrials,0.7792207792207793 BIDMC32HR_disc,0.704460191746561 BIDMC32SpO2_disc,0.6369320550229263 @@ -56,8 +57,10 @@ Skoda,0.9204103289706402 SpokenArabicDigits,0.9195088676671214 StandWalkJump,0.4 TactileTextureRecognition,0.9985315712187959 +Tiselac,0.6379170879676441 UCDHE-Rowing-MC,0.5704545454545454 UCIActivity,0.9546022490628905 UIPRMD-DS-C,0.8888888888888888 +USCActivity,0.5808778857837181 UWaveGestureLibrary,0.934375 WISDM,0.79582769943983 diff --git a/results/multiverse/TDE/TDE_auroc.csv b/results/multiverse/TDE/TDE_auroc.csv index 072c62e..5ce469b 100644 --- a/results/multiverse/TDE/TDE_auroc.csv +++ b/results/multiverse/TDE/TDE_auroc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9980787037037037 AsphaltObstaclesCoordinates,0.9526441031567158 AsphaltRegularityCoordinates,0.9883982407604455 AtrialFibrillation,0.46666666666666673 +AustraliaRainfall_disc,0.8074947826097345 AutomotiveRoadTrials,0.8729582577132485 BIDMC32HR_disc,0.847983903015853 BIDMC32SpO2_disc,0.541132976345275 @@ -56,8 +57,10 @@ Skoda,0.9952000827469243 SpokenArabicDigits,0.9933580427386813 StandWalkJump,0.5 TactileTextureRecognition,1.0 +Tiselac,0.8807714398090422 UCDHE-Rowing-MC,0.8429249759690937 UCIActivity,0.9969469318409124 UIPRMD-DS-C,1.0 +USCActivity,0.9364043932806045 UWaveGestureLibrary,0.9973158482142856 WISDM,0.9771380742801788 diff --git a/results/multiverse/TDE/TDE_balacc.csv b/results/multiverse/TDE/TDE_balacc.csv index 0fb1d99..d3a9758 100644 --- a/results/multiverse/TDE/TDE_balacc.csv +++ b/results/multiverse/TDE/TDE_balacc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.832030184791375 AsphaltRegularityCoordinates,0.9508583386536142 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.2991711989289402 AutomotiveRoadTrials,0.7118874773139746 BIDMC32HR_disc,0.6041218136945544 BIDMC32SpO2_disc,0.5060365452019918 @@ -56,8 +57,10 @@ Skoda,0.9120942331520815 SpokenArabicDigits,0.9195205479452054 StandWalkJump,0.4000000000000001 TactileTextureRecognition,0.9982363315696648 +Tiselac,0.4714800586271588 UCDHE-Rowing-MC,0.5843095238095237 UCIActivity,0.9559484189750713 UIPRMD-DS-C,0.8888888888888888 +USCActivity,0.5765274177228356 UWaveGestureLibrary,0.934375 WISDM,0.43332017717915633 diff --git a/results/multiverse/TDE/TDE_f1.csv b/results/multiverse/TDE/TDE_f1.csv index b9d451d..bb6086f 100644 --- a/results/multiverse/TDE/TDE_f1.csv +++ b/results/multiverse/TDE/TDE_f1.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933217391304349 AsphaltObstaclesCoordinates,0.82649385626469 AsphaltRegularityCoordinates,0.9504685408299867 AtrialFibrillation,0.2644688644688645 +AustraliaRainfall_disc,0.6478285217426618 AutomotiveRoadTrials,0.5641025641025641 BIDMC32HR_disc,0.6399175900096182 BIDMC32SpO2_disc,0.24062772449869224 @@ -56,8 +57,10 @@ Skoda,0.921771233546782 SpokenArabicDigits,0.9181939983167722 StandWalkJump,0.3945868945868946 TactileTextureRecognition,0.9985305990713076 +Tiselac,0.6179141273203642 UCDHE-Rowing-MC,0.5632377464073778 UCIActivity,0.9537560314714917 UIPRMD-DS-C,0.9 +USCActivity,0.5694408758179401 UWaveGestureLibrary,0.9339671545719854 WISDM,0.7459741780182929 diff --git a/results/multiverse/TDE/TDE_logloss.csv b/results/multiverse/TDE/TDE_logloss.csv index d393b59..eff47aa 100644 --- a/results/multiverse/TDE/TDE_logloss.csv +++ b/results/multiverse/TDE/TDE_logloss.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.2333438321835187 AsphaltObstaclesCoordinates,0.6077927802512004 AsphaltRegularityCoordinates,0.21716136144703443 AtrialFibrillation,1.3312925953981747 +AustraliaRainfall_disc,0.8103772152333866 AutomotiveRoadTrials,0.3562666561835708 BIDMC32HR_disc,0.5445802324380543 BIDMC32SpO2_disc,0.7849511334443995 @@ -56,8 +57,10 @@ Skoda,0.1875074463467951 SpokenArabicDigits,0.4570068697595056 StandWalkJump,1.1342721809601788 TactileTextureRecognition,0.009182027684064802 +Tiselac,2.3455434427386246 UCDHE-Rowing-MC,1.2487241911724993 UCIActivity,0.23102070354918536 UIPRMD-DS-C,0.1941595554944404 +USCActivity,1.2473973626067083 UWaveGestureLibrary,0.34179386795310895 WISDM,0.6539832593371383 diff --git a/results/multiverse/TDE/TDE_sensitivity.csv b/results/multiverse/TDE/TDE_sensitivity.csv index 8c61d30..d2eade9 100644 --- a/results/multiverse/TDE/TDE_sensitivity.csv +++ b/results/multiverse/TDE/TDE_sensitivity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.8286445012787724 AsphaltRegularityCoordinates,0.9594594594594594 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.7201389322185479 AutomotiveRoadTrials,0.5789473684210527 BIDMC32HR_disc,0.704460191746561 BIDMC32SpO2_disc,0.2020497803806735 @@ -56,8 +57,10 @@ Skoda,0.9204103289706402 SpokenArabicDigits,0.9195088676671214 StandWalkJump,0.4 TactileTextureRecognition,0.9985315712187959 +Tiselac,0.6379170879676441 UCDHE-Rowing-MC,0.5704545454545454 UCIActivity,0.9546022490628905 UIPRMD-DS-C,1.0 +USCActivity,0.5808778857837181 UWaveGestureLibrary,0.934375 WISDM,0.79582769943983 diff --git a/results/multiverse/TDE/TDE_specificity.csv b/results/multiverse/TDE/TDE_specificity.csv index 59481aa..5b5e9ec 100644 --- a/results/multiverse/TDE/TDE_specificity.csv +++ b/results/multiverse/TDE/TDE_specificity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9933333333333333 AsphaltObstaclesCoordinates,0.8286445012787724 AsphaltRegularityCoordinates,0.9422572178477691 AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.7201389322185479 AutomotiveRoadTrials,0.8448275862068966 BIDMC32HR_disc,0.704460191746561 BIDMC32SpO2_disc,0.8100233100233101 @@ -56,8 +57,10 @@ Skoda,0.9204103289706402 SpokenArabicDigits,0.9195088676671214 StandWalkJump,0.4 TactileTextureRecognition,0.9985315712187959 +Tiselac,0.6379170879676441 UCDHE-Rowing-MC,0.5704545454545454 UCIActivity,0.9546022490628905 UIPRMD-DS-C,0.7777777777777778 +USCActivity,0.5808778857837181 UWaveGestureLibrary,0.934375 WISDM,0.79582769943983 diff --git a/results/multiverse/TS2Vec/TS2Vec_accuracy.csv b/results/multiverse/TS2Vec/TS2Vec_accuracy.csv index 2bfa53d..6b612a2 100644 --- a/results/multiverse/TS2Vec/TS2Vec_accuracy.csv +++ b/results/multiverse/TS2Vec/TS2Vec_accuracy.csv @@ -21,6 +21,7 @@ CrowdSourced,0.7147899752912107 DuckDuckGeese,0.48 ERing,0.837037037037037 EigenWorms,0.8244274809160306 +EmoPain,0.7633802816901408 Epilepsy,0.9637681159420289 EthanolConcentration,0.2623574144486692 EyesOpenShut,0.5 diff --git a/results/multiverse/TS2Vec/TS2Vec_auroc.csv b/results/multiverse/TS2Vec/TS2Vec_auroc.csv index 216c662..ed9465e 100644 --- a/results/multiverse/TS2Vec/TS2Vec_auroc.csv +++ b/results/multiverse/TS2Vec/TS2Vec_auroc.csv @@ -21,6 +21,7 @@ CrowdSourced,0.7663625507856577 DuckDuckGeese,0.79 ERing,0.9865020576131686 EigenWorms,0.9599065200974137 +EmoPain,0.917405751146473 Epilepsy,0.993060614589887 EthanolConcentration,0.5203658830687841 EyesOpenShut,0.23129251700680273 diff --git a/results/multiverse/TS2Vec/TS2Vec_balacc.csv b/results/multiverse/TS2Vec/TS2Vec_balacc.csv index c4fb787..2563361 100644 --- a/results/multiverse/TS2Vec/TS2Vec_balacc.csv +++ b/results/multiverse/TS2Vec/TS2Vec_balacc.csv @@ -21,6 +21,7 @@ CrowdSourced,0.7148507429956661 DuckDuckGeese,0.4800000000000001 ERing,0.8370370370370371 EigenWorms,0.7367683524205263 +EmoPain,0.6102266573837705 Epilepsy,0.9644276629570747 EthanolConcentration,0.26223776223776224 EyesOpenShut,0.5 diff --git a/results/multiverse/TS2Vec/TS2Vec_f1.csv b/results/multiverse/TS2Vec/TS2Vec_f1.csv index 6731d96..75b14cf 100644 --- a/results/multiverse/TS2Vec/TS2Vec_f1.csv +++ b/results/multiverse/TS2Vec/TS2Vec_f1.csv @@ -21,6 +21,7 @@ CrowdSourced,0.7566265060240964 DuckDuckGeese,0.4276193991289584 ERing,0.8382191530713408 EigenWorms,0.8090221408586924 +EmoPain,0.7980260604157057 Epilepsy,0.9635825846846003 EthanolConcentration,0.25577488452660363 EyesOpenShut,0.6666666666666666 diff --git a/results/multiverse/TS2Vec/TS2Vec_logloss.csv b/results/multiverse/TS2Vec/TS2Vec_logloss.csv index 33f7f60..bec993b 100644 --- a/results/multiverse/TS2Vec/TS2Vec_logloss.csv +++ b/results/multiverse/TS2Vec/TS2Vec_logloss.csv @@ -21,6 +21,7 @@ CrowdSourced,0.6474338506785442 DuckDuckGeese,1.4348492613517514 ERing,0.7220637443397937 EigenWorms,0.5631454406550138 +EmoPain,0.7079228535637965 Epilepsy,0.17672386773869742 EthanolConcentration,1.3910358412111647 EyesOpenShut,0.7532299987952091 diff --git a/results/multiverse/TS2Vec/TS2Vec_sensitivity.csv b/results/multiverse/TS2Vec/TS2Vec_sensitivity.csv index 69dbdc9..afd1d62 100644 --- a/results/multiverse/TS2Vec/TS2Vec_sensitivity.csv +++ b/results/multiverse/TS2Vec/TS2Vec_sensitivity.csv @@ -21,6 +21,7 @@ CrowdSourced,0.8870056497175142 DuckDuckGeese,0.48 ERing,0.837037037037037 EigenWorms,0.8244274809160306 +EmoPain,0.7633802816901408 Epilepsy,0.9637681159420289 EthanolConcentration,0.2623574144486692 EyesOpenShut,1.0 diff --git a/results/multiverse/TS2Vec/TS2Vec_specificity.csv b/results/multiverse/TS2Vec/TS2Vec_specificity.csv index 03684bd..7c4df91 100644 --- a/results/multiverse/TS2Vec/TS2Vec_specificity.csv +++ b/results/multiverse/TS2Vec/TS2Vec_specificity.csv @@ -21,6 +21,7 @@ CrowdSourced,0.5426958362738179 DuckDuckGeese,0.48 ERing,0.837037037037037 EigenWorms,0.8244274809160306 +EmoPain,0.7633802816901408 Epilepsy,0.9637681159420289 EthanolConcentration,0.2623574144486692 EyesOpenShut,0.0 diff --git a/results/multiverse/TSF/TSF_accuracy.csv b/results/multiverse/TSF/TSF_accuracy.csv index e7a1137..8fdcf81 100644 --- a/results/multiverse/TSF/TSF_accuracy.csv +++ b/results/multiverse/TSF/TSF_accuracy.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.98 AsphaltObstaclesCoordinates,0.7723785166240409 AsphaltRegularityCoordinates,0.9573901464713716 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.767454919822799 AutomotiveRoadTrials,0.7402597402597403 BIDMC32HR_disc,0.32847019591496457 BIDMC32SpO2_disc,0.5489787411421425 diff --git a/results/multiverse/TSF/TSF_auroc.csv b/results/multiverse/TSF/TSF_auroc.csv index f376ed8..a3f40c6 100644 --- a/results/multiverse/TSF/TSF_auroc.csv +++ b/results/multiverse/TSF/TSF_auroc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9988368055555557 AsphaltObstaclesCoordinates,0.9426079057630901 AsphaltRegularityCoordinates,0.9825955877136979 AtrialFibrillation,0.51 +AustraliaRainfall_disc,0.8470335041915626 AutomotiveRoadTrials,0.8148820326678765 BIDMC32HR_disc,0.3655155326269203 BIDMC32SpO2_disc,0.35106883112007564 diff --git a/results/multiverse/TSF/TSF_balacc.csv b/results/multiverse/TSF/TSF_balacc.csv index 0458312..9c012fb 100644 --- a/results/multiverse/TSF/TSF_balacc.csv +++ b/results/multiverse/TSF/TSF_balacc.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.98 AsphaltObstaclesCoordinates,0.7709882039305465 AsphaltRegularityCoordinates,0.9574590338369866 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.37247336066256265 AutomotiveRoadTrials,0.721415607985481 BIDMC32HR_disc,0.16948632521173237 BIDMC32SpO2_disc,0.4771673543635476 diff --git a/results/multiverse/TSF/TSF_f1.csv b/results/multiverse/TSF/TSF_f1.csv index 0cf9790..4ddb469 100644 --- a/results/multiverse/TSF/TSF_f1.csv +++ b/results/multiverse/TSF/TSF_f1.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.9801626701800614 AsphaltObstaclesCoordinates,0.7717473191451875 AsphaltRegularityCoordinates,0.956989247311828 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.7450059736322683 AutomotiveRoadTrials,0.5652173913043478 BIDMC32HR_disc,0.3661794237694408 BIDMC32SpO2_disc,0.2815405046480744 diff --git a/results/multiverse/TSF/TSF_logloss.csv b/results/multiverse/TSF/TSF_logloss.csv index 4679a81..4273fa8 100644 --- a/results/multiverse/TSF/TSF_logloss.csv +++ b/results/multiverse/TSF/TSF_logloss.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.3577129123047125 AsphaltObstaclesCoordinates,0.6061286383001531 AsphaltRegularityCoordinates,0.20068202452981976 AtrialFibrillation,1.1949588120342045 +AustraliaRainfall_disc,0.5419462543562622 AutomotiveRoadTrials,0.47954273308282785 BIDMC32HR_disc,8.915091662504985 BIDMC32SpO2_disc,8.476640644543588 diff --git a/results/multiverse/TSF/TSF_sensitivity.csv b/results/multiverse/TSF/TSF_sensitivity.csv index 99614d6..c7d06df 100644 --- a/results/multiverse/TSF/TSF_sensitivity.csv +++ b/results/multiverse/TSF/TSF_sensitivity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.98 AsphaltObstaclesCoordinates,0.7723785166240409 AsphaltRegularityCoordinates,0.9621621621621622 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.767454919822799 AutomotiveRoadTrials,0.6842105263157895 BIDMC32HR_disc,0.3284701959149646 BIDMC32SpO2_disc,0.3103953147877013 diff --git a/results/multiverse/TSF/TSF_specificity.csv b/results/multiverse/TSF/TSF_specificity.csv index 043bbc9..5608e2f 100644 --- a/results/multiverse/TSF/TSF_specificity.csv +++ b/results/multiverse/TSF/TSF_specificity.csv @@ -5,6 +5,7 @@ ArticularyWordRecognition,0.98 AsphaltObstaclesCoordinates,0.7723785166240409 AsphaltRegularityCoordinates,0.952755905511811 AtrialFibrillation,0.3333333333333333 +AustraliaRainfall_disc,0.767454919822799 AutomotiveRoadTrials,0.7586206896551724 BIDMC32HR_disc,0.3284701959149646 BIDMC32SpO2_disc,0.6439393939393939 diff --git a/results/multiverse/TimesNet/TimesNet_accuracy.csv b/results/multiverse/TimesNet/TimesNet_accuracy.csv index 8dcefce..862ee6d 100644 --- a/results/multiverse/TimesNet/TimesNet_accuracy.csv +++ b/results/multiverse/TimesNet/TimesNet_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7116131309565831 DuckDuckGeese,0.42 ERing,0.7592592592592593 EigenWorms,0.4732824427480916 +EmoPain,0.7746478873239436 Epilepsy,0.8985507246376812 EthanolConcentration,0.2965779467680608 EyesOpenShut,0.5952380952380952 diff --git a/results/multiverse/TimesNet/TimesNet_auroc.csv b/results/multiverse/TimesNet/TimesNet_auroc.csv index 04d227f..f18d76e 100644 --- a/results/multiverse/TimesNet/TimesNet_auroc.csv +++ b/results/multiverse/TimesNet/TimesNet_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7412263913974378 DuckDuckGeese,0.7205000000000001 ERing,0.9686090534979424 EigenWorms,0.718388522476044 +EmoPain,0.9279579956422663 Epilepsy,0.9862550380827751 EthanolConcentration,0.5745624783473889 EyesOpenShut,0.7165532879818594 diff --git a/results/multiverse/TimesNet/TimesNet_balacc.csv b/results/multiverse/TimesNet/TimesNet_balacc.csv index 100692b..afc5185 100644 --- a/results/multiverse/TimesNet/TimesNet_balacc.csv +++ b/results/multiverse/TimesNet/TimesNet_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.711667045440953 DuckDuckGeese,0.42000000000000004 ERing,0.7592592592592592 EigenWorms,0.2979108813891423 +EmoPain,0.4675872205461437 Epilepsy,0.8982511923688394 EthanolConcentration,0.29545454545454547 EyesOpenShut,0.5952380952380952 diff --git a/results/multiverse/TimesNet/TimesNet_f1.csv b/results/multiverse/TimesNet/TimesNet_f1.csv index b55a80b..f953282 100644 --- a/results/multiverse/TimesNet/TimesNet_f1.csv +++ b/results/multiverse/TimesNet/TimesNet_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.749770290964778 DuckDuckGeese,0.41054342260513366 ERing,0.7404213649857618 EigenWorms,0.35942968120789176 +EmoPain,0.7992684823997005 Epilepsy,0.8924464174838986 EthanolConcentration,0.19858702091342648 EyesOpenShut,0.711864406779661 diff --git a/results/multiverse/TimesNet/TimesNet_logloss.csv b/results/multiverse/TimesNet/TimesNet_logloss.csv index 5cd770e..72e7688 100644 --- a/results/multiverse/TimesNet/TimesNet_logloss.csv +++ b/results/multiverse/TimesNet/TimesNet_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,3.7190997533835852 DuckDuckGeese,1.488468220036912 ERing,1.1844290967923863 EigenWorms,2.103077859528844 +EmoPain,0.7416951481591448 Epilepsy,0.28349653897231647 EthanolConcentration,1.8423970303040027 EyesOpenShut,0.665966461485736 diff --git a/results/multiverse/TimesNet/TimesNet_sensitivity.csv b/results/multiverse/TimesNet/TimesNet_sensitivity.csv index 824b7f9..5214a77 100644 --- a/results/multiverse/TimesNet/TimesNet_sensitivity.csv +++ b/results/multiverse/TimesNet/TimesNet_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.864406779661017 DuckDuckGeese,0.42 ERing,0.7592592592592593 EigenWorms,0.4732824427480916 +EmoPain,0.7746478873239436 Epilepsy,0.8985507246376812 EthanolConcentration,0.2965779467680608 EyesOpenShut,1.0 diff --git a/results/multiverse/TimesNet/TimesNet_specificity.csv b/results/multiverse/TimesNet/TimesNet_specificity.csv index c5032a7..9916e61 100644 --- a/results/multiverse/TimesNet/TimesNet_specificity.csv +++ b/results/multiverse/TimesNet/TimesNet_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.5589273112208892 DuckDuckGeese,0.42 ERing,0.7592592592592593 EigenWorms,0.4732824427480916 +EmoPain,0.7746478873239436 Epilepsy,0.8985507246376812 EthanolConcentration,0.2965779467680608 EyesOpenShut,0.19047619047619047 diff --git a/results/multiverse/XCM/XCM_accuracy.csv b/results/multiverse/XCM/XCM_accuracy.csv new file mode 100644 index 0000000..fcddf01 --- /dev/null +++ b/results/multiverse/XCM/XCM_accuracy.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.3023255813953488 +AppliancesEnergy_disc,0.8095238095238095 +ArticularyWordRecognition,0.9466666666666667 +AsphaltObstaclesCoordinates,0.7468030690537084 +AsphaltRegularityCoordinates,0.9733688415446072 +AtrialFibrillation,0.2 +AustraliaRainfall_disc,0.7725504877186414 +AutomotiveRoadTrials,0.7142857142857143 +BIDMC32HR_disc,0.6348478532721967 +BIDMC32SpO2_disc,0.5410587744893706 +BeijingPM10Quality_disc,0.715729001584786 +BeijingPM25Quality_disc,0.8332012678288431 +BenzeneConcentration_disc,0.7640906449738524 +Blink,0.9244444444444444 +BoneIntensitiesAgeGroup,0.7528089887640449 +BoneProbAgeGroup,0.6764044943820224 +CharacterTrajectories,0.9937325905292479 +CounterMovementJump,0.8212290502793296 +Cricket,0.9861111111111112 +CrowdSourced,0.7518531591951995 +DuckDuckGeese,0.54 +ERing,0.3037037037037037 +EigenWorms,0.5038167938931297 +Epilepsy,0.9130434782608695 +EthanolConcentration,0.2889733840304182 +EyesOpenShut,0.5 +FaceDetection,0.6174801362088536 +FordChallenge,0.6232423490488007 +HandMovementDirection,0.28378378378378377 +Handwriting,0.3611764705882353 +Heartbeat,0.7560975609756098 +HouseholdPowerConsumption1_disc,0.11807580174927114 +HouseholdPowerConsumption2_disc,0.6967930029154519 +IEEEPPG_disc,0.6453313253012049 +IRDS-SFL,0.896551724137931 +JapaneseVowels,0.9783783783783784 +KERAAL-RTK,0.5714285714285714 +KIMORE-PR-C,0.42857142857142855 +KINECAL-QSEO,0.29411764705882354 +LSST,0.45174371451743717 +Libras,0.8277777777777777 +Locust2022,0.8681418611700515 +LowCost,0.52 +MindReading,0.4119448698315467 +MotionSenseHAR,0.969811320754717 +MotorImagery,0.51 +NATOPS,0.9 +PEMS-SF,0.8323699421965318 +PenDigits,0.949685534591195 +PhonemeSpectra,0.1258574410975246 +PhotoStimulation,0.4166666666666667 +RacketSports,0.8618421052631579 +STEW,0.6557239057239057 +SelfRegulationSCP1,0.552901023890785 +Skoda,0.9405730456314114 +SpokenArabicDigits,0.9845384265575261 +StandWalkJump,0.2 +TactileTextureRecognition,0.7650513950073421 +Tiselac,0.7832153690596562 +UCDHE-Rowing-MC,0.6772727272727272 +UCIActivity,0.9837567680133278 +UIPRMD-DS-C,0.7222222222222222 +USCActivity,0.6830953827460511 +UWaveGestureLibrary,0.8375 +WISDM,0.873092524628163 diff --git a/results/multiverse/XCM/XCM_auroc.csv b/results/multiverse/XCM/XCM_auroc.csv new file mode 100644 index 0000000..e1ed21e --- /dev/null +++ b/results/multiverse/XCM/XCM_auroc.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.40807638331996793 +AppliancesEnergy_disc,0.7022058823529411 +ArticularyWordRecognition,0.9983680555555554 +AsphaltObstaclesCoordinates,0.9282288995088815 +AsphaltRegularityCoordinates,0.9947222813364545 +AtrialFibrillation,0.36000000000000004 +AustraliaRainfall_disc,0.8531741621798045 +AutomotiveRoadTrials,0.6814882032667877 +BIDMC32HR_disc,0.8257225160774645 +BIDMC32SpO2_disc,0.4466540048531264 +BeijingPM10Quality_disc,0.8228868866803458 +BeijingPM25Quality_disc,0.9209492402371762 +BenzeneConcentration_disc,0.7906529888748304 +Blink,0.94976 +BoneIntensitiesAgeGroup,0.8763879280693906 +BoneProbAgeGroup,0.8482297247796746 +CharacterTrajectories,0.999966482814142 +CounterMovementJump,0.962704411373488 +Cricket,0.9983164983164983 +CrowdSourced,0.7769353372486634 +DuckDuckGeese,0.8835 +ERing,0.9387489711934157 +EigenWorms,0.8730167577171467 +Epilepsy,0.9757065171400083 +EthanolConcentration,0.5558911771202598 +EyesOpenShut,0.2380952380952381 +FaceDetection,0.6725854880623994 +FordChallenge,0.5 +HandMovementDirection,0.5678910034842238 +Handwriting,0.8556383071775312 +Heartbeat,0.7316263632053106 +HouseholdPowerConsumption1_disc,0.6215145245466448 +HouseholdPowerConsumption2_disc,0.6832528326022078 +IEEEPPG_disc,0.8553718178469635 +IRDS-SFL,0.9637681159420289 +JapaneseVowels,0.9995787852093061 +KERAAL-RTK,0.5 +KIMORE-PR-C,0.5 +KINECAL-QSEO,0.25 +LSST,0.8052505045504954 +Libras,0.9869378306878308 +Locust2022,0.7439370405945788 +LowCost,0.5296222222222222 +MindReading,0.7188556235726715 +MotionSenseHAR,0.9927211095965354 +MotorImagery,0.5826 +NATOPS,0.9912222222222221 +PEMS-SF,0.9661758891337312 +PenDigits,0.9975886205572294 +PhonemeSpectra,0.8201195011847978 +PhotoStimulation,0.38638380138380135 +RacketSports,0.9724577717009012 +STEW,0.8272616975969951 +SelfRegulationSCP1,0.6061876805516726 +Skoda,0.9959321424656511 +SpokenArabicDigits,0.9998641964066783 +StandWalkJump,0.4 +TactileTextureRecognition,0.9863492753156763 +Tiselac,0.9526128233026999 +UCDHE-Rowing-MC,0.9200922262025203 +UCIActivity,0.9995001270676686 +UIPRMD-DS-C,0.8148148148148149 +USCActivity,0.9533753889936101 +UWaveGestureLibrary,0.9718861607142857 +WISDM,0.9644757073546718 diff --git a/results/multiverse/XCM/XCM_balacc.csv b/results/multiverse/XCM/XCM_balacc.csv new file mode 100644 index 0000000..9211a6b --- /dev/null +++ b/results/multiverse/XCM/XCM_balacc.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.367965367965368 +AppliancesEnergy_disc,0.5 +ArticularyWordRecognition,0.9466666666666665 +AsphaltObstaclesCoordinates,0.750158283785592 +AsphaltRegularityCoordinates,0.9735191884798184 +AtrialFibrillation,0.20000000000000004 +AustraliaRainfall_disc,0.38160086861129444 +AutomotiveRoadTrials,0.6333938294010889 +BIDMC32HR_disc,0.43818415153971185 +BIDMC32SpO2_disc,0.4103754347165767 +BeijingPM10Quality_disc,0.5304937507403205 +BeijingPM25Quality_disc,0.8490841689347542 +BenzeneConcentration_disc,0.6232841845755148 +Blink,0.9315 +BoneIntensitiesAgeGroup,0.791771432890703 +BoneProbAgeGroup,0.673252210372801 +CharacterTrajectories,0.9933931956260587 +CounterMovementJump,0.8216572504708098 +Cricket,0.9861111111111112 +CrowdSourced,0.7519220303099171 +DuckDuckGeese,0.54 +ERing,0.30370370370370375 +EigenWorms,0.36 +Epilepsy,0.9141494435612083 +EthanolConcentration,0.28805361305361304 +EyesOpenShut,0.5 +FaceDetection,0.6174801362088536 +FordChallenge,0.5 +HandMovementDirection,0.30238095238095236 +Handwriting,0.35229571326491305 +Heartbeat,0.599158368895211 +HouseholdPowerConsumption1_disc,0.3253012048192771 +HouseholdPowerConsumption2_disc,0.3894931045615977 +IEEEPPG_disc,0.604697162742921 +IRDS-SFL,0.9347826086956521 +JapaneseVowels,0.9757319741190709 +KERAAL-RTK,0.5 +KIMORE-PR-C,0.6666666666666666 +KINECAL-QSEO,0.15625 +LSST,0.22954875180846224 +Libras,0.8277777777777777 +Locust2022,0.5672494601241204 +LowCost,0.52 +MindReading,0.39793855343461887 +MotionSenseHAR,0.9599330551208486 +MotorImagery,0.51 +NATOPS,0.8999999999999999 +PEMS-SF,0.8300086387042909 +PenDigits,0.9503633306991516 +PhonemeSpectra,0.1258725314812866 +PhotoStimulation,0.3414141414141414 +RacketSports,0.8704734219269104 +STEW,0.6557239057239057 +SelfRegulationSCP1,0.554398471717454 +Skoda,0.9295284795279287 +SpokenArabicDigits,0.9845433789954339 +StandWalkJump,0.20000000000000004 +TactileTextureRecognition,0.7676131153558414 +Tiselac,0.5813305834575884 +UCDHE-Rowing-MC,0.697642857142857 +UCIActivity,0.9842652561389942 +UIPRMD-DS-C,0.7222222222222222 +USCActivity,0.6872296744351977 +UWaveGestureLibrary,0.8374999999999999 +WISDM,0.547020066421741 diff --git a/results/multiverse/XCM/XCM_f1.csv b/results/multiverse/XCM/XCM_f1.csv new file mode 100644 index 0000000..2576c12 --- /dev/null +++ b/results/multiverse/XCM/XCM_f1.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.21816168327796231 +AppliancesEnergy_disc,0.0 +ArticularyWordRecognition,0.9462743362917276 +AsphaltObstaclesCoordinates,0.7491477045617893 +AsphaltRegularityCoordinates,0.9732620320855615 +AtrialFibrillation,0.16190476190476188 +AustraliaRainfall_disc,0.75719188582855 +AutomotiveRoadTrials,0.45 +BIDMC32HR_disc,0.6518311866365318 +BIDMC32SpO2_disc,0.11708099438652766 +BeijingPM10Quality_disc,0.15736934820904286 +BeijingPM25Quality_disc,0.7632170978627671 +BenzeneConcentration_disc,0.39881539980256664 +Blink,0.9212962962962963 +BoneIntensitiesAgeGroup,0.7478544452020218 +BoneProbAgeGroup,0.6779403896390017 +CharacterTrajectories,0.9937101179249307 +CounterMovementJump,0.8167930322758735 +Cricket,0.9860139860139859 +CrowdSourced,0.7923190546528803 +DuckDuckGeese,0.5323809523809524 +ERing,0.1517735755778025 +EigenWorms,0.40159309658148024 +Epilepsy,0.9112318840579711 +EthanolConcentration,0.24591210249853798 +EyesOpenShut,0.6666666666666666 +FaceDetection,0.6516795865633075 +FordChallenge,0.0 +HandMovementDirection,0.2781612959451537 +Handwriting,0.3377079846984321 +Heartbeat,0.358974358974359 +HouseholdPowerConsumption1_disc,0.025756350972902773 +HouseholdPowerConsumption2_disc,0.6557375281190868 +IEEEPPG_disc,0.6225822295456521 +IRDS-SFL,0.8 +JapaneseVowels,0.9781367493415885 +KERAAL-RTK,0.0 +KIMORE-PR-C,0.3333333333333333 +KINECAL-QSEO,0.0 +LSST,0.3791312736547366 +Libras,0.819467577984202 +Locust2022,0.21338155515370705 +LowCost,0.6317135549872123 +MindReading,0.40190722407655577 +MotionSenseHAR,0.9728201579348978 +MotorImagery,0.07547169811320754 +NATOPS,0.9003106182590119 +PEMS-SF,0.8278898581282377 +PenDigits,0.9496829926785575 +PhonemeSpectra,0.09434448848932879 +PhotoStimulation,0.2851037851037851 +RacketSports,0.8588118410519711 +STEW,0.7288397790055249 +SelfRegulationSCP1,0.6888361045130641 +Skoda,0.9407162240986361 +SpokenArabicDigits,0.9845196910744067 +StandWalkJump,0.1574074074074074 +TactileTextureRecognition,0.7638645349615292 +Tiselac,0.7722507644925846 +UCDHE-Rowing-MC,0.6681854984389035 +UCIActivity,0.9837608130719198 +UIPRMD-DS-C,0.6153846153846154 +USCActivity,0.6796853915511324 +UWaveGestureLibrary,0.8385274008139714 +WISDM,0.8608499413848607 diff --git a/results/multiverse/XCM/XCM_logloss.csv b/results/multiverse/XCM/XCM_logloss.csv new file mode 100644 index 0000000..e9de0b4 --- /dev/null +++ b/results/multiverse/XCM/XCM_logloss.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,1.281995397783109 +AppliancesEnergy_disc,1.220613288803126 +ArticularyWordRecognition,0.20913287122187446 +AsphaltObstaclesCoordinates,0.9458313154759524 +AsphaltRegularityCoordinates,0.1266260204259907 +AtrialFibrillation,1.1886000723607923 +AustraliaRainfall_disc,0.5192454015724045 +AutomotiveRoadTrials,1.531994513704828 +BIDMC32HR_disc,2.6153268183918863 +BIDMC32SpO2_disc,3.2864579433764276 +BeijingPM10Quality_disc,1.192763567851187 +BeijingPM25Quality_disc,0.5700050251191676 +BenzeneConcentration_disc,3.346307447421879 +Blink,0.6552198963655975 +BoneIntensitiesAgeGroup,0.7577387737965597 +BoneProbAgeGroup,1.3781207487249014 +CharacterTrajectories,0.025087426830150132 +CounterMovementJump,0.4458332415980918 +Cricket,0.23899100869077017 +CrowdSourced,5.681748011476673 +DuckDuckGeese,1.4476585961906254 +ERing,1.5511330924068825 +EigenWorms,2.7894721126602673 +Epilepsy,0.3715004381698749 +EthanolConcentration,1.529273939296933 +EyesOpenShut,4.248528533555482 +FaceDetection,4.314418813211354 +FordChallenge,13.571439293456832 +HandMovementDirection,1.5513074783505933 +Handwriting,2.480634334028785 +Heartbeat,0.8112859994036347 +HouseholdPowerConsumption1_disc,16.637074689936 +HouseholdPowerConsumption2_disc,1.9238153847023434 +IEEEPPG_disc,1.4806723442802172 +IRDS-SFL,0.5406823350671576 +JapaneseVowels,0.07008161110194283 +KERAAL-RTK,5.5033371337501675 +KIMORE-PR-C,8.192123345424639 +KINECAL-QSEO,1.509511533283483 +LSST,2.6029082409706916 +Libras,0.5640605291689049 +Locust2022,0.33809055946445105 +LowCost,1.1380963668204618 +MindReading,2.8786347532333054 +MotionSenseHAR,0.19610327002131758 +MotorImagery,2.977766816238155 +NATOPS,0.2578673378413515 +PEMS-SF,0.7632564329767687 +PenDigits,0.2884284956829766 +PhonemeSpectra,4.032035001666196 +PhotoStimulation,1.1993444304140304 +RacketSports,0.39196803556227966 +STEW,3.4722974016081136 +SelfRegulationSCP1,2.4810720946580234 +Skoda,0.24444171624728298 +SpokenArabicDigits,0.06498082123670214 +StandWalkJump,1.838150270556138 +TactileTextureRecognition,1.3949109714966077 +Tiselac,0.9638490690786189 +UCDHE-Rowing-MC,1.7944030264704292 +UCIActivity,0.06438419313786149 +UIPRMD-DS-C,2.579849189540059 +USCActivity,1.7029310258175516 +UWaveGestureLibrary,0.6536157560205249 +WISDM,1.999502230004768 diff --git a/results/multiverse/XCM/XCM_sensitivity.csv b/results/multiverse/XCM/XCM_sensitivity.csv new file mode 100644 index 0000000..e9242f0 --- /dev/null +++ b/results/multiverse/XCM/XCM_sensitivity.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.3023255813953488 +AppliancesEnergy_disc,0.0 +ArticularyWordRecognition,0.9466666666666667 +AsphaltObstaclesCoordinates,0.7468030690537084 +AsphaltRegularityCoordinates,0.9837837837837838 +AtrialFibrillation,0.2 +AustraliaRainfall_disc,0.7725504877186414 +AutomotiveRoadTrials,0.47368421052631576 +BIDMC32HR_disc,0.6348478532721967 +BIDMC32SpO2_disc,0.10688140556368961 +BeijingPM10Quality_disc,0.09190672153635117 +BeijingPM25Quality_disc,0.8892529488859764 +BenzeneConcentration_disc,0.25218476903870163 +Blink,0.995 +BoneIntensitiesAgeGroup,0.7528089887640449 +BoneProbAgeGroup,0.6764044943820224 +CharacterTrajectories,0.9937325905292479 +CounterMovementJump,0.8212290502793296 +Cricket,0.9861111111111112 +CrowdSourced,0.9470338983050848 +DuckDuckGeese,0.54 +ERing,0.3037037037037037 +EigenWorms,0.5038167938931297 +Epilepsy,0.9130434782608695 +EthanolConcentration,0.2889733840304182 +EyesOpenShut,1.0 +FaceDetection,0.7156640181611805 +FordChallenge,0.0 +HandMovementDirection,0.28378378378378377 +Handwriting,0.3611764705882353 +Heartbeat,0.24561403508771928 +HouseholdPowerConsumption1_disc,0.11807580174927114 +HouseholdPowerConsumption2_disc,0.6967930029154519 +IEEEPPG_disc,0.6453313253012049 +IRDS-SFL,1.0 +JapaneseVowels,0.9783783783783784 +KERAAL-RTK,0.0 +KIMORE-PR-C,1.0 +KINECAL-QSEO,0.0 +LSST,0.45174371451743717 +Libras,0.8277777777777777 +Locust2022,0.20136518771331058 +LowCost,0.8233333333333334 +MindReading,0.4119448698315467 +MotionSenseHAR,0.969811320754717 +MotorImagery,0.04 +NATOPS,0.9 +PEMS-SF,0.8323699421965318 +PenDigits,0.949685534591195 +PhonemeSpectra,0.1258574410975246 +PhotoStimulation,0.4166666666666667 +RacketSports,0.8618421052631579 +STEW,0.925364758698092 +SelfRegulationSCP1,0.9931506849315068 +Skoda,0.9405730456314114 +SpokenArabicDigits,0.9845384265575261 +StandWalkJump,0.2 +TactileTextureRecognition,0.7650513950073421 +Tiselac,0.7832153690596562 +UCDHE-Rowing-MC,0.6772727272727272 +UCIActivity,0.9837567680133278 +UIPRMD-DS-C,0.4444444444444444 +USCActivity,0.6830953827460511 +UWaveGestureLibrary,0.8375 +WISDM,0.873092524628163 diff --git a/results/multiverse/XCM/XCM_specificity.csv b/results/multiverse/XCM/XCM_specificity.csv new file mode 100644 index 0000000..bc472f0 --- /dev/null +++ b/results/multiverse/XCM/XCM_specificity.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.3023255813953488 +AppliancesEnergy_disc,1.0 +ArticularyWordRecognition,0.9466666666666667 +AsphaltObstaclesCoordinates,0.7468030690537084 +AsphaltRegularityCoordinates,0.963254593175853 +AtrialFibrillation,0.2 +AustraliaRainfall_disc,0.7725504877186414 +AutomotiveRoadTrials,0.7931034482758621 +BIDMC32HR_disc,0.6348478532721967 +BIDMC32SpO2_disc,0.7138694638694638 +BeijingPM10Quality_disc,0.9690807799442896 +BeijingPM25Quality_disc,0.8089153889835321 +BenzeneConcentration_disc,0.994383600112328 +Blink,0.868 +BoneIntensitiesAgeGroup,0.7528089887640449 +BoneProbAgeGroup,0.6764044943820224 +CharacterTrajectories,0.9937325905292479 +CounterMovementJump,0.8212290502793296 +Cricket,0.9861111111111112 +CrowdSourced,0.5568101623147494 +DuckDuckGeese,0.54 +ERing,0.3037037037037037 +EigenWorms,0.5038167938931297 +Epilepsy,0.9130434782608695 +EthanolConcentration,0.2889733840304182 +EyesOpenShut,0.0 +FaceDetection,0.5192962542565267 +FordChallenge,1.0 +HandMovementDirection,0.28378378378378377 +Handwriting,0.3611764705882353 +Heartbeat,0.9527027027027027 +HouseholdPowerConsumption1_disc,0.11807580174927114 +HouseholdPowerConsumption2_disc,0.6967930029154519 +IEEEPPG_disc,0.6453313253012049 +IRDS-SFL,0.8695652173913043 +JapaneseVowels,0.9783783783783784 +KERAAL-RTK,1.0 +KIMORE-PR-C,0.3333333333333333 +KINECAL-QSEO,0.3125 +LSST,0.45174371451743717 +Libras,0.8277777777777777 +Locust2022,0.9331337325349301 +LowCost,0.21666666666666667 +MindReading,0.4119448698315467 +MotionSenseHAR,0.969811320754717 +MotorImagery,0.98 +NATOPS,0.9 +PEMS-SF,0.8323699421965318 +PenDigits,0.949685534591195 +PhonemeSpectra,0.1258574410975246 +PhotoStimulation,0.4166666666666667 +RacketSports,0.8618421052631579 +STEW,0.38608305274971944 +SelfRegulationSCP1,0.11564625850340136 +Skoda,0.9405730456314114 +SpokenArabicDigits,0.9845384265575261 +StandWalkJump,0.2 +TactileTextureRecognition,0.7650513950073421 +Tiselac,0.7832153690596562 +UCDHE-Rowing-MC,0.6772727272727272 +UCIActivity,0.9837567680133278 +UIPRMD-DS-C,1.0 +USCActivity,0.6830953827460511 +UWaveGestureLibrary,0.8375 +WISDM,0.873092524628163 diff --git a/results/multiverse/datasets.html b/results/multiverse/datasets.html index f024c1e..9ede74a 100644 --- a/results/multiverse/datasets.html +++ b/results/multiverse/datasets.html @@ -60,7 +60,7 @@ details { margin-top: .6rem; } summary { cursor: pointer; color: var(--accent); } code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: .9em; } -tr.nosignal td { background: rgba(214, 158, 46, .16); }tr.saturated td { background: rgba(56, 161, 105, .14); }

    Multiverse-core datasets: accuracy

    66 datasets · accuracy · best of up to 26 estimators against the Dummy baseline · built 2026-09-05

    DatasetDummyMedianBestBest estimatorGain over dummySpreadEstimators
    KINECAL-QSEO0.94120.94120.9412Arsenal0.00000.117626
    BIDMC32SpO2_disc0.71530.66190.7203ROCKET0.00500.190124
    Locust20220.91120.90820.9206MRHydra0.00940.045225
    Heartbeat0.72200.74390.7854CIF0.06340.126826
    HouseholdPowerConsumption2_disc0.72160.76750.7872DisjointCNN0.06560.141426
    AutomotiveRoadTrials0.75320.79220.8442CIF0.09090.233826
    AustraliaRainfall_disc0.68600.77310.7808LITETime-MV0.09480.088117
    EyesOpenShut0.50000.50000.5952STSF0.09520.190526
    MotorImagery0.50000.50500.6000FreshPRINCE0.10000.140026
    BeijingPM10Quality_disc0.71120.82420.8417FreshPRINCE0.13050.107026
    Alzheimers0.41860.37210.5581MRHydra0.13950.302325
    AppliancesEnergy_disc0.80950.82140.9524FreshPRINCE0.14290.452426
    EmoPain0.78310.84080.92681NN-DTW0.14370.242320
    PhotoStimulation0.41670.38890.5833ROCKET0.16670.388925
    FaceDetection0.50000.62830.6850H-InceptionTime0.18500.170525
    BeijingPM25Quality_disc0.69770.87690.8879ConvTran0.19020.128826
    AtrialFibrillation0.33330.26670.5333TS2Vec0.20000.466726
    HouseholdPowerConsumption1_disc0.77840.91180.9825FreshPRINCE0.20410.218726
    LowCost0.50000.63170.7300TSF0.23000.248326
    BoneProbAgeGroup0.47640.64380.7124H-InceptionTime0.23600.193326
    StandWalkJump0.33330.40000.6000MRHydra0.26670.400026
    CrowdSourced0.50020.71430.7734LITETime-MV0.27320.176826
    BenzeneConcentration_disc0.68970.82010.9768STSF0.28700.577026
    FordChallenge0.62320.87900.9360QUANT0.31280.312825
    BIDMC32HR_disc0.65070.80120.9637RIST0.31300.635324
    STEW0.50000.73600.8385Arsenal0.33850.210024
    BoneIntensitiesAgeGroup0.47640.79100.8202HC20.34380.296626
    PhonemeSpectra0.02560.27890.3746H-InceptionTime0.34890.291126
    KERAAL-RTK0.57140.78570.9286HC20.35710.571426
    LSST0.31510.62900.7040FreshPRINCE0.38890.480926
    HandMovementDirection0.20270.41220.6081TSF0.40540.418926
    DuckDuckGeese0.20000.47000.6400H-InceptionTime0.44000.480026
    SelfRegulationSCP10.50170.85320.9454MRHydra0.44370.208226
    IEEEPPG_disc0.26050.43980.7078ConvTran0.44730.438326
    AsphaltRegularityCoordinates0.50730.98140.9947H-InceptionTime0.48740.291626
    MindReading0.23120.52830.7243LITETime-MV0.49310.385926
    EthanolConcentration0.25100.42970.7490STC0.49810.532326
    UIPRMD-DS-C0.50000.83331.0000Catch220.50000.388926
    WISDM0.36640.86580.8965MRHydra0.53000.131026
    Blink0.44440.98671.0000Arsenal0.55560.428926
    EigenWorms0.41980.86260.9771MRHydra0.55730.557325
    KIMORE-PR-C0.14290.42860.7143LITETime-MV0.57140.571426
    AsphaltObstaclesCoordinates0.28390.82100.8670MRHydra0.58310.289026
    CounterMovementJump0.33520.74860.9274Arsenal0.59220.458126
    Handwriting0.03760.38940.6529H-InceptionTime0.61530.478826
    USCActivity0.11380.69360.7354LITETime-MV0.62160.137921
    RacketSports0.28290.87830.9079RDST0.62500.125026
    UCDHE-Rowing-MC0.20450.73410.8295PatchMTSC0.62500.338626
    IRDS-SFL0.20690.79310.8621RDST0.65520.448326
    Skoda0.23560.94660.9646H-InceptionTime0.72900.119925
    Epilepsy0.26810.98191.0000HC20.73190.101426
    Tiselac0.06280.81360.8373STSF0.77450.204420
    MotionSenseHAR0.20380.98681.0000DrCIF0.79620.101926
    NATOPS0.16670.89170.9667LITETime-MV0.80000.155626
    UCIActivity0.19160.97610.9983LITETime-MV0.80670.174926
    UWaveGestureLibrary0.12500.90940.9406Arsenal0.81560.553126
    ERing0.16670.93330.9963MRHydra0.82960.237026
    PEMS-SF0.11560.86991.0000CIF0.88440.317926
    PenDigits0.10380.97780.9911H-InceptionTime0.88740.234424
    SpokenArabicDigits0.10000.98020.9945DisjointCNN0.89450.129126
    Libras0.06670.88890.9722RIST0.90560.338926
    JapaneseVowels0.08380.96620.9946LiteTIME0.91080.208126
    Cricket0.08330.97921.00001NN-DTW0.91670.069426
    CharacterTrajectories0.06480.98960.9958H-InceptionTime0.93110.044626
    TactileTextureRecognition0.05140.99851.0000H-InceptionTime0.94860.168926
    ArticularyWordRecognition0.04000.98170.9933Arsenal0.95330.050026

    One row per dataset. Dummy is the no-skill floor. Median, best and spread are over the other estimators, so the baseline cannot flatter them. Gain over dummy is best minus dummy, how much skill was found at all; spread is best minus worst, how much the choice of estimator mattered. The two answer different questions, and a single range would conflate them.

    3 of 66 datasets gained 0.05 or less over the baseline (shaded amber) and 15 have a best of 0.99 or more (shaded green). Both separate estimators poorly, for opposite reasons. Best is a maximum over many estimators, so it is optimistic by construction: read it as what the archive can currently do on a problem, not as what any one method delivers.