diff --git a/README.md b/README.md index c2153f2..6af806d 100644 --- a/README.md +++ b/README.md @@ -40,33 +40,34 @@ The current paper version describes: | # | Estimator | Accuracy rank | Accuracy | Balanced accuracy | AUROC | F1 | Log loss ↓ | Sensitivity | Specificity | |---|---|---|---|---|---|---|---|---|---| -| 1 | HC2 | **8.06** | **0.7909** | 0.7518 | **0.8990** | 0.7273 | **0.5383** | 0.7459 | **0.7943** | -| 2 | MRHydra | 8.51 | 0.7837 | **0.7564** | 0.8105 | **0.7316** | 7.7974 | **0.7642** | 0.7757 | -| 3 | RDST | 9.35 | 0.7734 | 0.7333 | 0.7912 | 0.6991 | 8.1667 | 0.7109 | 0.7874 | -| 4 | RIST | 9.97 | 0.7720 | 0.7397 | 0.8748 | 0.7147 | 0.6218 | 0.7408 | 0.7655 | -| 5 | DrCIF | 10.25 | 0.7747 | 0.7429 | 0.8813 | 0.7173 | 0.6484 | 0.7397 | 0.7708 | -| 6 | FreshPRINCE | 10.30 | 0.7743 | 0.7487 | 0.8745 | 0.7211 | 0.6007 | 0.7414 | 0.7770 | -| 7 | CIF | 10.37 | 0.7781 | 0.7471 | 0.8908 | 0.7212 | 0.6430 | 0.7441 | 0.7753 | -| 8 | QUANT | 10.81 | 0.7720 | 0.7462 | 0.8831 | 0.7189 | 0.6175 | 0.7521 | 0.7581 | -| 9 | Arsenal | 10.96 | 0.7680 | 0.7321 | 0.8457 | 0.7024 | 3.8631 | 0.7257 | 0.7732 | -| 10 | ROCKET | 11.09 | 0.7690 | 0.7326 | 0.7925 | 0.7019 | 8.3249 | 0.7200 | 0.7764 | -| 11 | LITETime-MV | 11.50 | 0.7506 | 0.7299 | 0.8513 | 0.6820 | 1.3206 | 0.7132 | 0.7637 | -| 12 | STSF | 11.79 | 0.7724 | 0.7477 | 0.8804 | 0.7080 | 0.6432 | 0.7345 | 0.7826 | -| 13 | H-InceptionTime | 11.93 | 0.7408 | 0.7190 | 0.8496 | 0.6838 | 1.3227 | 0.7223 | 0.7378 | -| 14 | LiteTIME | 12.68 | 0.7341 | 0.7104 | 0.8394 | 0.6680 | 1.4776 | 0.7113 | 0.7336 | -| 15 | PatchMTSC | 13.38 | 0.7428 | 0.6897 | 0.8261 | 0.6533 | 0.7655 | 0.6852 | 0.7352 | -| 16 | ConvTran | 13.39 | 0.7462 | 0.7102 | 0.8592 | 0.6767 | 0.8190 | 0.7159 | 0.7345 | -| 17 | Catch22 | 13.42 | 0.7475 | 0.7181 | 0.8697 | 0.6922 | 0.7147 | 0.7240 | 0.7374 | -| 18 | STC | 14.25 | 0.7545 | 0.7172 | 0.8744 | 0.6940 | 0.6391 | 0.7185 | 0.7537 | -| 19 | TSF | 14.26 | 0.7515 | 0.7236 | 0.8740 | 0.6883 | 0.7252 | 0.7093 | 0.7606 | -| 20 | TDE | 15.07 | 0.7262 | 0.6813 | 0.8374 | 0.6382 | 0.8869 | 0.6714 | 0.7344 | -| 21 | Summary | 17.18 | 0.6858 | 0.6574 | 0.8268 | 0.6230 | 0.9123 | 0.6574 | 0.6844 | -| 22 | TimesNet | 17.26 | 0.7013 | 0.6659 | 0.8275 | 0.6281 | 1.1613 | 0.6726 | 0.6898 | -| 23 | TimesURL | 17.46 | 0.6958 | 0.6533 | 0.7906 | 0.5967 | 1.0055 | 0.6257 | 0.6973 | -| 24 | 1NN-DTW | 19.08 | 0.6712 | 0.6454 | 0.7197 | 0.6136 | 11.8506 | 0.6521 | 0.6636 | -| 25 | Dummy | 22.68 | 0.3645 | 0.3029 | 0.5000 | 0.1507 | 1.4067 | 0.2855 | 0.3816 | - -Average over the 52 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. +| 1 | HC2 | **8.11** | **0.7887** | 0.7541 | **0.9000** | 0.7346 | **0.5440** | 0.7547 | **0.7910** | +| 2 | MRHydra | 8.92 | 0.7810 | **0.7579** | 0.8130 | **0.7368** | 7.8942 | **0.7715** | 0.7718 | +| 3 | RDST | 9.76 | 0.7707 | 0.7372 | 0.7963 | 0.7105 | 8.2660 | 0.7236 | 0.7833 | +| 4 | RIST | 10.38 | 0.7693 | 0.7422 | 0.8755 | 0.7221 | 0.6294 | 0.7504 | 0.7613 | +| 5 | DrCIF | 10.44 | 0.7721 | 0.7454 | 0.8821 | 0.7248 | 0.6558 | 0.7490 | 0.7669 | +| 6 | CIF | 10.61 | 0.7756 | 0.7497 | 0.8920 | 0.7288 | 0.6497 | 0.7536 | 0.7714 | +| 7 | FreshPRINCE | 10.63 | 0.7717 | 0.7516 | 0.8752 | 0.7293 | 0.6075 | 0.7515 | 0.7731 | +| 8 | Arsenal | 11.04 | 0.7654 | 0.7340 | 0.8471 | 0.7092 | 3.9265 | 0.7337 | 0.7696 | +| 9 | QUANT | 11.19 | 0.7693 | 0.7486 | 0.8839 | 0.7262 | 0.6238 | 0.7616 | 0.7539 | +| 10 | LITETime-MV | 11.60 | 0.7476 | 0.7312 | 0.8518 | 0.6875 | 1.3300 | 0.7200 | 0.7600 | +| 11 | ROCKET | 11.61 | 0.7661 | 0.7345 | 0.7955 | 0.7080 | 8.4299 | 0.7282 | 0.7724 | +| 12 | STSF | 12.08 | 0.7698 | 0.7503 | 0.8813 | 0.7155 | 0.6493 | 0.7439 | 0.7790 | +| 13 | H-InceptionTime | 12.28 | 0.7375 | 0.7205 | 0.8506 | 0.6897 | 1.3334 | 0.7303 | 0.7333 | +| 14 | LiteTIME | 12.79 | 0.7308 | 0.7122 | 0.8402 | 0.6746 | 1.4921 | 0.7199 | 0.7291 | +| 15 | ConvTran | 13.81 | 0.7430 | 0.7139 | 0.8606 | 0.6882 | 0.8300 | 0.7289 | 0.7295 | +| 16 | PatchMTSC | 13.92 | 0.7395 | 0.6934 | 0.8288 | 0.6660 | 0.7748 | 0.6985 | 0.7300 | +| 17 | Catch22 | 13.95 | 0.7442 | 0.7203 | 0.8703 | 0.6996 | 0.7238 | 0.7337 | 0.7326 | +| 18 | STC | 14.61 | 0.7516 | 0.7188 | 0.8748 | 0.7004 | 0.6447 | 0.7264 | 0.7496 | +| 19 | TSF | 14.73 | 0.7484 | 0.7257 | 0.8747 | 0.6952 | 0.7335 | 0.7179 | 0.7565 | +| 20 | TS2Vec | 15.26 | 0.7212 | 0.6849 | 0.8082 | 0.6588 | 0.7326 | 0.6980 | 0.7100 | +| 21 | TDE | 15.39 | 0.7230 | 0.6823 | 0.8383 | 0.6441 | 0.8859 | 0.6786 | 0.7301 | +| 22 | Summary | 17.85 | 0.6814 | 0.6586 | 0.8263 | 0.6294 | 0.9251 | 0.6661 | 0.6787 | +| 23 | TimesNet | 18.16 | 0.6971 | 0.6688 | 0.8280 | 0.6390 | 1.1785 | 0.6850 | 0.6838 | +| 24 | TimesURL | 18.25 | 0.6916 | 0.6563 | 0.7931 | 0.6084 | 1.0193 | 0.6379 | 0.6914 | +| 25 | 1NN-DTW | 19.71 | 0.6672 | 0.6457 | 0.7214 | 0.6193 | 11.9949 | 0.6584 | 0.6584 | +| 26 | Dummy | 23.91 | 0.3538 | 0.2991 | 0.5000 | 0.1537 | 1.4284 | 0.2911 | 0.3695 | + +Average over the 51 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. Rebuilt with `python -m multiverse.experiments.tables`, which also writes a sortable diff --git a/multiverse/classification/_ts2vec.py b/multiverse/classification/_ts2vec.py index a26a421..63f5e62 100644 --- a/multiverse/classification/_ts2vec.py +++ b/multiverse/classification/_ts2vec.py @@ -58,7 +58,7 @@ import numpy as np from aeon.classification import BaseClassifier from sklearn.linear_model import LogisticRegression -from sklearn.model_selection import GridSearchCV +from sklearn.model_selection import GridSearchCV, train_test_split from sklearn.multiclass import OneVsRestClassifier from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler @@ -101,6 +101,12 @@ class TS2VecClassifier(BaseClassifier): in the original. temporal_unit : int, default=0 Minimum unit for temporal contrast, as in the original. + probe_max_samples : int or None, default=None + Cap on the number of cases the probe is fitted on. None takes the + authors' values, 10000 for the SVM probe and 100000 for the logistic + one, and the collection is subsampled with stratification when it is + larger. Without this the SVM grid search is unaffordable on the large + collections. probe : {"svm", "logistic"}, default="svm" Classifier fitted on the representations. ``"svm"`` reproduces the authors' UEA protocol, an ``SVC`` chosen by grid search over C on the @@ -120,6 +126,8 @@ class TS2VecClassifier(BaseClassifier): Pretrained TS2Vec encoder. probe_ : object Classifier fitted on the encoded training collection. + probe_cases_ : int + Number of cases the probe was fitted on, after any subsampling. device_ : str Resolved device. n_channels_ : int @@ -166,6 +174,7 @@ def __init__( max_train_length: int = 3000, temporal_unit: int = 0, probe: str = "svm", + probe_max_samples: int | None = None, device: str = "auto", verbose: bool = False, random_state=1234, @@ -180,6 +189,7 @@ def __init__( self.max_train_length = max_train_length self.temporal_unit = temporal_unit self.probe = probe + self.probe_max_samples = probe_max_samples self.device = device self.verbose = verbose self.random_state = random_state @@ -217,6 +227,25 @@ def _to_original_layout(X: np.ndarray) -> np.ndarray: """Convert an aeon collection to the authors' (case, time, channel).""" return np.transpose(np.asarray(X, dtype=np.float32), (0, 2, 1)) + def _subsample(self, features, y, seed): + """Cap the collection the probe is fitted on, as the authors do. + + ``fit_svm`` takes ``MAX_SAMPLES=10000`` and ``fit_lr`` 100000, and both + subsample with stratification before fitting. This is not a detail: + the SVM grid is ten values of C over five folds, and an RBF SVC is + between quadratic and cubic in the number of cases, so without the cap + the probe alone can outrun a 60 hour job on the larger collections. + """ + limit = self.probe_max_samples + if limit is None: + limit = 10_000 if self.probe == "svm" else 100_000 + if features.shape[0] <= limit: + return features, y + features, _, y, _ = train_test_split( + features, y, train_size=limit, random_state=0, stratify=y + ) + return features, y + def _build_probe(self, n_cases: int, seed: int): """Return the probe, following the authors' evaluation protocols. @@ -294,7 +323,11 @@ def _fit(self, X: np.ndarray, y): encoded_y = np.asarray( [self._class_dictionary[label] for label in y], dtype=np.int64 ) - self.probe_ = self._build_probe(len(encoded_y), seed).fit(encoded, encoded_y) + fit_features, fit_y = self._subsample(encoded, encoded_y, seed) + self.probe_cases_ = int(fit_features.shape[0]) + self.probe_ = self._build_probe( + self.probe_cases_, seed + ).fit(fit_features, fit_y) return self def _check_shape(self, X: np.ndarray) -> None: diff --git a/results/multiverse/TS2Vec/TS2Vec_accuracy.csv b/results/multiverse/TS2Vec/TS2Vec_accuracy.csv new file mode 100644 index 0000000..2bfa53d --- /dev/null +++ b/results/multiverse/TS2Vec/TS2Vec_accuracy.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4186046511627907 +AppliancesEnergy_disc,0.8095238095238095 +ArticularyWordRecognition,0.98 +AsphaltObstaclesCoordinates,0.7851662404092071 +AsphaltRegularityCoordinates,0.9893475366178429 +AtrialFibrillation,0.5333333333333333 +AutomotiveRoadTrials,0.7662337662337663 +BIDMC32HR_disc,0.7661525635681534 +BIDMC32SpO2_disc,0.5360566902876198 +BeijingPM10Quality_disc,0.8248811410459588 +BeijingPM25Quality_disc,0.8817353407290016 +BenzeneConcentration_disc,0.7848150300213055 +Blink,0.6511111111111111 +BoneIntensitiesAgeGroup,0.5887640449438202 +BoneProbAgeGroup,0.5280898876404494 +CharacterTrajectories,0.9937325905292479 +CounterMovementJump,0.6759776536312849 +Cricket,0.9861111111111112 +CrowdSourced,0.7147899752912107 +DuckDuckGeese,0.48 +ERing,0.837037037037037 +EigenWorms,0.8244274809160306 +Epilepsy,0.9637681159420289 +EthanolConcentration,0.2623574144486692 +EyesOpenShut,0.5 +FaceDetection,0.514472190692395 +FordChallenge,0.7882547559966915 +HandMovementDirection,0.25675675675675674 +Handwriting,0.40941176470588236 +Heartbeat,0.7804878048780488 +HouseholdPowerConsumption1_disc,0.9052478134110787 +HouseholdPowerConsumption2_disc,0.7128279883381924 +IEEEPPG_disc,0.39231927710843373 +IRDS-SFL,0.8620689655172413 +JapaneseVowels,0.972972972972973 +KERAAL-RTK,0.6428571428571429 +KIMORE-PR-C,0.2857142857142857 +KINECAL-QSEO,0.9411764705882353 +LSST,0.5766423357664233 +Libras,0.8888888888888888 +LowCost,0.53 +MindReading,0.33843797856049007 +MotionSenseHAR,0.9660377358490566 +MotorImagery,0.5 +NATOPS,0.9277777777777778 +PEMS-SF,0.8265895953757225 +PenDigits,0.9868496283590623 +PhonemeSpectra,0.23560990158067402 +PhotoStimulation,0.3888888888888889 +RacketSports,0.868421052631579 +STEW,0.6877104377104377 +SelfRegulationSCP1,0.757679180887372 +Skoda,0.9405730456314114 +SpokenArabicDigits,0.9818099135970896 +StandWalkJump,0.3333333333333333 +TactileTextureRecognition,0.9926578560939795 +UCDHE-Rowing-MC,0.6545454545454545 +UCIActivity,0.9912536443148688 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.909375 +WISDM,0.8049063163994592 diff --git a/results/multiverse/TS2Vec/TS2Vec_auroc.csv b/results/multiverse/TS2Vec/TS2Vec_auroc.csv new file mode 100644 index 0000000..216c662 --- /dev/null +++ b/results/multiverse/TS2Vec/TS2Vec_auroc.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.5646872493985565 +AppliancesEnergy_disc,0.4411764705882353 +ArticularyWordRecognition,0.9978587962962961 +AsphaltObstaclesCoordinates,0.9428780736122715 +AsphaltRegularityCoordinates,0.9982620415691282 +AtrialFibrillation,0.62 +AutomotiveRoadTrials,0.7704174228675136 +BIDMC32HR_disc,0.8476332651663594 +BIDMC32SpO2_disc,0.3377351053046514 +BeijingPM10Quality_disc,0.8753902205104104 +BeijingPM25Quality_disc,0.9305178161163344 +BenzeneConcentration_disc,0.8549342982883301 +Blink,0.79414 +BoneIntensitiesAgeGroup,0.7540972473907201 +BoneProbAgeGroup,0.7108969782840555 +CharacterTrajectories,0.9999629286634228 +CounterMovementJump,0.897687901976433 +Cricket,0.9995791245791246 +CrowdSourced,0.7663625507856577 +DuckDuckGeese,0.79 +ERing,0.9865020576131686 +EigenWorms,0.9599065200974137 +Epilepsy,0.993060614589887 +EthanolConcentration,0.5203658830687841 +EyesOpenShut,0.23129251700680273 +FaceDetection,0.5197674838081274 +FordChallenge,0.8475737447710174 +HandMovementDirection,0.5015481197684587 +Handwriting,0.8724933352072651 +Heartbeat,0.7336415362731151 +HouseholdPowerConsumption1_disc,0.9622873349469153 +HouseholdPowerConsumption2_disc,0.635829188983623 +IEEEPPG_disc,0.5898166629567041 +IRDS-SFL,0.9551630434782609 +JapaneseVowels,0.9991795362939342 +KERAAL-RTK,0.9583333333333334 +KIMORE-PR-C,0.5 +KINECAL-QSEO,0.125 +LSST,0.871763658336968 +Libras,0.9939153439153439 +LowCost,0.5648222222222222 +MindReading,0.6572383950269114 +MotionSenseHAR,0.9987337932160057 +MotorImagery,0.4576 +NATOPS,0.9905925925925926 +PEMS-SF,0.9610942578494267 +PenDigits,0.9996544727334365 +PhonemeSpectra,0.8732441261763364 +PhotoStimulation,0.38888685388685396 +RacketSports,0.9622508105125198 +STEW,0.7450874643051038 +SelfRegulationSCP1,0.859658932065977 +Skoda,0.9942989727741186 +SpokenArabicDigits,0.9998471945840186 +StandWalkJump,0.5133333333333334 +TactileTextureRecognition,0.999977297010771 +UCDHE-Rowing-MC,0.8770285580653228 +UCIActivity,0.9996600092798027 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.9842299107142857 +WISDM,0.9455932026252636 diff --git a/results/multiverse/TS2Vec/TS2Vec_balacc.csv b/results/multiverse/TS2Vec/TS2Vec_balacc.csv new file mode 100644 index 0000000..c4fb787 --- /dev/null +++ b/results/multiverse/TS2Vec/TS2Vec_balacc.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.3333333333333333 +AppliancesEnergy_disc,0.5 +ArticularyWordRecognition,0.98 +AsphaltObstaclesCoordinates,0.7829203036336215 +AsphaltRegularityCoordinates,0.9894622969426119 +AtrialFibrillation,0.5333333333333333 +AutomotiveRoadTrials,0.6147912885662432 +BIDMC32HR_disc,0.6099722691239187 +BIDMC32SpO2_disc,0.38175965079332574 +BeijingPM10Quality_disc,0.7709771465471456 +BeijingPM25Quality_disc,0.8565692300707852 +BenzeneConcentration_disc,0.7251881862078469 +Blink,0.673 +BoneIntensitiesAgeGroup,0.6180629197596325 +BoneProbAgeGroup,0.5133856219612237 +CharacterTrajectories,0.9936616128047581 +CounterMovementJump,0.6761770244821091 +Cricket,0.9861111111111112 +CrowdSourced,0.7148507429956661 +DuckDuckGeese,0.4800000000000001 +ERing,0.8370370370370371 +EigenWorms,0.7367683524205263 +Epilepsy,0.9644276629570747 +EthanolConcentration,0.26223776223776224 +EyesOpenShut,0.5 +FaceDetection,0.514472190692395 +FordChallenge,0.7562525023484745 +HandMovementDirection,0.2511904761904762 +Handwriting,0.41205945477865136 +Heartbeat,0.648411569464201 +HouseholdPowerConsumption1_disc,0.8050103033858279 +HouseholdPowerConsumption2_disc,0.34276094276094277 +IEEEPPG_disc,0.4025443119105505 +IRDS-SFL,0.8976449275362319 +JapaneseVowels,0.9735600081540015 +KERAAL-RTK,0.5833333333333334 +KIMORE-PR-C,0.5833333333333334 +KINECAL-QSEO,0.5 +LSST,0.33480027802363316 +Libras,0.8888888888888888 +LowCost,0.53 +MindReading,0.3330829762368523 +MotionSenseHAR,0.9726266887422225 +MotorImagery,0.5 +NATOPS,0.9277777777777777 +PEMS-SF,0.828463163245772 +PenDigits,0.9868256827975281 +PhonemeSpectra,0.23568697604265318 +PhotoStimulation,0.3191919191919192 +RacketSports,0.8789244186046512 +STEW,0.6877104377104377 +SelfRegulationSCP1,0.7580607585499954 +Skoda,0.9360292747896445 +SpokenArabicDigits,0.9818140307181403 +StandWalkJump,0.3333333333333333 +TactileTextureRecognition,0.9924961273745948 +UCDHE-Rowing-MC,0.6597063492063493 +UCIActivity,0.9915942256095708 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.909375 +WISDM,0.5025863399079105 diff --git a/results/multiverse/TS2Vec/TS2Vec_f1.csv b/results/multiverse/TS2Vec/TS2Vec_f1.csv new file mode 100644 index 0000000..6731d96 --- /dev/null +++ b/results/multiverse/TS2Vec/TS2Vec_f1.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.24704536789935186 +AppliancesEnergy_disc,0.0 +ArticularyWordRecognition,0.9802171683389075 +AsphaltObstaclesCoordinates,0.7857148794795134 +AsphaltRegularityCoordinates,0.9892761394101877 +AtrialFibrillation,0.5075757575757576 +AutomotiveRoadTrials,0.4 +BIDMC32HR_disc,0.751819920016781 +BIDMC32SpO2_disc,0.02794759825327511 +BeijingPM10Quality_disc,0.6797101449275362 +BeijingPM25Quality_disc,0.8021213125621478 +BenzeneConcentration_disc,0.620948481746844 +Blink,0.689108910891089 +BoneIntensitiesAgeGroup,0.5856665137516777 +BoneProbAgeGroup,0.533632015343325 +CharacterTrajectories,0.9937505422521032 +CounterMovementJump,0.6658438126498493 +Cricket,0.9860139860139859 +CrowdSourced,0.7566265060240964 +DuckDuckGeese,0.4276193991289584 +ERing,0.8382191530713408 +EigenWorms,0.8090221408586924 +Epilepsy,0.9635825846846003 +EthanolConcentration,0.25577488452660363 +EyesOpenShut,0.6666666666666666 +FaceDetection,0.538191632928475 +FordChallenge,0.6903225806451613 +HandMovementDirection,0.25935254772052646 +Handwriting,0.34020295702854986 +Heartbeat,0.47058823529411764 +HouseholdPowerConsumption1_disc,0.9034660695388977 +HouseholdPowerConsumption2_disc,0.6066867263035853 +IEEEPPG_disc,0.39102139602942654 +IRDS-SFL,0.7419354838709677 +JapaneseVowels,0.9731909275283056 +KERAAL-RTK,0.2857142857142857 +KIMORE-PR-C,0.2857142857142857 +KINECAL-QSEO,0.0 +LSST,0.5261253925694681 +Libras,0.8890296477542855 +LowCost,0.5688073394495413 +MindReading,0.33680667083617616 +MotionSenseHAR,0.9663060132611815 +MotorImagery,0.6666666666666666 +NATOPS,0.9255380245641962 +PEMS-SF,0.8243029470599211 +PenDigits,0.9868406552005395 +PhonemeSpectra,0.2311725963438298 +PhotoStimulation,0.2693452380952381 +RacketSports,0.8682293583609373 +STEW,0.6763594068043036 +SelfRegulationSCP1,0.7815384615384615 +Skoda,0.940509648539942 +SpokenArabicDigits,0.9818409944516879 +StandWalkJump,0.31111111111111106 +TactileTextureRecognition,0.9926548515104918 +UCDHE-Rowing-MC,0.6585683842542325 +UCIActivity,0.9912527097593792 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.9069046125356246 +WISDM,0.7869545005215961 diff --git a/results/multiverse/TS2Vec/TS2Vec_logloss.csv b/results/multiverse/TS2Vec/TS2Vec_logloss.csv new file mode 100644 index 0000000..33f7f60 --- /dev/null +++ b/results/multiverse/TS2Vec/TS2Vec_logloss.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,1.0760321520266076 +AppliancesEnergy_disc,0.530784133490714 +ArticularyWordRecognition,1.084094354880307 +AsphaltObstaclesCoordinates,0.5668770111674879 +AsphaltRegularityCoordinates,0.04304921126851031 +AtrialFibrillation,1.061000597141431 +AutomotiveRoadTrials,0.5171696769666839 +BIDMC32HR_disc,0.6932010780958163 +BIDMC32SpO2_disc,4.434789207916943 +BeijingPM10Quality_disc,0.40432530768767094 +BeijingPM25Quality_disc,0.30559026750964513 +BenzeneConcentration_disc,0.4238285347729799 +Blink,0.8733051237412777 +BoneIntensitiesAgeGroup,0.7724344269245791 +BoneProbAgeGroup,0.9254839346745701 +CharacterTrajectories,0.1525771065809532 +CounterMovementJump,0.7987977031088543 +Cricket,0.7052087128638784 +CrowdSourced,0.6474338506785442 +DuckDuckGeese,1.4348492613517514 +ERing,0.7220637443397937 +EigenWorms,0.5631454406550138 +Epilepsy,0.17672386773869742 +EthanolConcentration,1.3910358412111647 +EyesOpenShut,0.7532299987952091 +FaceDetection,0.6927979005965101 +FordChallenge,0.48185296241650205 +HandMovementDirection,1.407638722478598 +Handwriting,2.5158978716425833 +Heartbeat,0.5065109879876131 +HouseholdPowerConsumption1_disc,0.2211112028806409 +HouseholdPowerConsumption2_disc,0.7491583090000944 +IEEEPPG_disc,1.3543375055451639 +IRDS-SFL,0.338983558061043 +JapaneseVowels,0.2433010144883182 +KERAAL-RTK,0.5799940497328644 +KIMORE-PR-C,0.9919845752093838 +KINECAL-QSEO,0.3096902617711121 +LSST,1.3171550310515487 +Libras,0.8432433071323391 +LowCost,0.6891334928764153 +MindReading,1.5870207119189175 +MotionSenseHAR,0.11844062054005312 +MotorImagery,0.6933294346111208 +NATOPS,0.2755090185088572 +PEMS-SF,0.6444782995925598 +PenDigits,0.0528409795516652 +PhonemeSpectra,2.656924792671446 +PhotoStimulation,1.1235672541984034 +RacketSports,0.35677548185544333 +STEW,0.6927531526795336 +SelfRegulationSCP1,0.4977438518895362 +Skoda,0.20732022762624142 +SpokenArabicDigits,0.07094699936544402 +StandWalkJump,1.1230437593080356 +TactileTextureRecognition,0.12347820086172717 +UCDHE-Rowing-MC,1.2212241779837336 +UCIActivity,0.040364467063487786 +UIPRMD-DS-C,0.23108895372215368 +UWaveGestureLibrary,0.5952183940967828 +WISDM,0.7714030350002307 diff --git a/results/multiverse/TS2Vec/TS2Vec_sensitivity.csv b/results/multiverse/TS2Vec/TS2Vec_sensitivity.csv new file mode 100644 index 0000000..69dbdc9 --- /dev/null +++ b/results/multiverse/TS2Vec/TS2Vec_sensitivity.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4186046511627907 +AppliancesEnergy_disc,0.0 +ArticularyWordRecognition,0.98 +AsphaltObstaclesCoordinates,0.7851662404092071 +AsphaltRegularityCoordinates,0.9972972972972973 +AtrialFibrillation,0.5333333333333333 +AutomotiveRoadTrials,0.3157894736842105 +BIDMC32HR_disc,0.7661525635681534 +BIDMC32SpO2_disc,0.02342606149341142 +BeijingPM10Quality_disc,0.6433470507544582 +BeijingPM25Quality_disc,0.7929226736566186 +BenzeneConcentration_disc,0.568039950062422 +Blink,0.87 +BoneIntensitiesAgeGroup,0.5887640449438202 +BoneProbAgeGroup,0.5280898876404494 +CharacterTrajectories,0.9937325905292479 +CounterMovementJump,0.6759776536312849 +Cricket,0.9861111111111112 +CrowdSourced,0.8870056497175142 +DuckDuckGeese,0.48 +ERing,0.837037037037037 +EigenWorms,0.8244274809160306 +Epilepsy,0.9637681159420289 +EthanolConcentration,0.2623574144486692 +EyesOpenShut,1.0 +FaceDetection,0.5658342792281499 +FordChallenge,0.6264178558360776 +HandMovementDirection,0.25675675675675674 +Handwriting,0.40941176470588236 +Heartbeat,0.3508771929824561 +HouseholdPowerConsumption1_disc,0.9052478134110787 +HouseholdPowerConsumption2_disc,0.7128279883381924 +IEEEPPG_disc,0.39231927710843373 +IRDS-SFL,0.9583333333333334 +JapaneseVowels,0.972972972972973 +KERAAL-RTK,0.16666666666666666 +KIMORE-PR-C,1.0 +KINECAL-QSEO,0.0 +LSST,0.5766423357664233 +Libras,0.8888888888888888 +LowCost,0.62 +MindReading,0.33843797856049007 +MotionSenseHAR,0.9660377358490566 +MotorImagery,1.0 +NATOPS,0.9277777777777778 +PEMS-SF,0.8265895953757225 +PenDigits,0.9868496283590623 +PhonemeSpectra,0.23560990158067402 +PhotoStimulation,0.3888888888888889 +RacketSports,0.868421052631579 +STEW,0.6526374859708193 +SelfRegulationSCP1,0.8698630136986302 +Skoda,0.9405730456314114 +SpokenArabicDigits,0.9818099135970896 +StandWalkJump,0.3333333333333333 +TactileTextureRecognition,0.9926578560939795 +UCDHE-Rowing-MC,0.6545454545454545 +UCIActivity,0.9912536443148688 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.909375 +WISDM,0.8049063163994592 diff --git a/results/multiverse/TS2Vec/TS2Vec_specificity.csv b/results/multiverse/TS2Vec/TS2Vec_specificity.csv new file mode 100644 index 0000000..03684bd --- /dev/null +++ b/results/multiverse/TS2Vec/TS2Vec_specificity.csv @@ -0,0 +1,62 @@ +Resamples:,0 +Alzheimers,0.4186046511627907 +AppliancesEnergy_disc,1.0 +ArticularyWordRecognition,0.98 +AsphaltObstaclesCoordinates,0.7851662404092071 +AsphaltRegularityCoordinates,0.9816272965879265 +AtrialFibrillation,0.5333333333333333 +AutomotiveRoadTrials,0.9137931034482759 +BIDMC32HR_disc,0.7661525635681534 +BIDMC32SpO2_disc,0.7400932400932401 +BeijingPM10Quality_disc,0.8986072423398329 +BeijingPM25Quality_disc,0.9202157864849517 +BenzeneConcentration_disc,0.8823364223532716 +Blink,0.476 +BoneIntensitiesAgeGroup,0.5887640449438202 +BoneProbAgeGroup,0.5280898876404494 +CharacterTrajectories,0.9937325905292479 +CounterMovementJump,0.6759776536312849 +Cricket,0.9861111111111112 +CrowdSourced,0.5426958362738179 +DuckDuckGeese,0.48 +ERing,0.837037037037037 +EigenWorms,0.8244274809160306 +Epilepsy,0.9637681159420289 +EthanolConcentration,0.2623574144486692 +EyesOpenShut,0.0 +FaceDetection,0.46311010215664017 +FordChallenge,0.8860871488608715 +HandMovementDirection,0.25675675675675674 +Handwriting,0.40941176470588236 +Heartbeat,0.9459459459459459 +HouseholdPowerConsumption1_disc,0.9052478134110787 +HouseholdPowerConsumption2_disc,0.7128279883381924 +IEEEPPG_disc,0.39231927710843373 +IRDS-SFL,0.8369565217391305 +JapaneseVowels,0.972972972972973 +KERAAL-RTK,1.0 +KIMORE-PR-C,0.16666666666666666 +KINECAL-QSEO,1.0 +LSST,0.5766423357664233 +Libras,0.8888888888888888 +LowCost,0.44 +MindReading,0.33843797856049007 +MotionSenseHAR,0.9660377358490566 +MotorImagery,0.0 +NATOPS,0.9277777777777778 +PEMS-SF,0.8265895953757225 +PenDigits,0.9868496283590623 +PhonemeSpectra,0.23560990158067402 +PhotoStimulation,0.3888888888888889 +RacketSports,0.868421052631579 +STEW,0.7227833894500562 +SelfRegulationSCP1,0.6462585034013606 +Skoda,0.9405730456314114 +SpokenArabicDigits,0.9818099135970896 +StandWalkJump,0.3333333333333333 +TactileTextureRecognition,0.9926578560939795 +UCDHE-Rowing-MC,0.6545454545454545 +UCIActivity,0.9912536443148688 +UIPRMD-DS-C,1.0 +UWaveGestureLibrary,0.909375 +WISDM,0.8049063163994592 diff --git a/results/multiverse/datasets.html b/results/multiverse/datasets.html index 5302b7b..fde92bf 100644 --- a/results/multiverse/datasets.html +++ b/results/multiverse/datasets.html @@ -60,7 +60,7 @@ details { margin-top: .6rem; } summary { cursor: pointer; color: var(--accent); } code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: .9em; } -tr.nosignal td { background: rgba(214, 158, 46, .16); }tr.saturated td { background: rgba(56, 161, 105, .14); }

Multiverse-core datasets: accuracy

66 datasets · accuracy · best of up to 24 estimators against the Dummy baseline · built 2026-09-03

DatasetDummyMedianBestBest estimatorGain over dummySpreadEstimators
AtrialFibrillation0.33330.23330.3333CIF0.00000.266724
KINECAL-QSEO0.94120.94120.9412Arsenal0.00000.117624
BIDMC32SpO2_disc0.71530.67030.7203ROCKET0.00500.171322
Locust20220.91120.90850.9206MRHydra0.00940.045224
Heartbeat0.72200.74390.7854CIF0.06340.126824
HouseholdPowerConsumption2_disc0.72160.76750.7872HC20.06560.141424
AutomotiveRoadTrials0.75320.79870.8442CIF0.09090.233824
AustraliaRainfall_disc0.68600.77460.7808LITETime-MV0.09480.088116
EyesOpenShut0.50000.50000.5952STSF0.09520.190524
MotorImagery0.50000.51000.6000FreshPRINCE0.10000.140024
BeijingPM10Quality_disc0.71120.82420.8417FreshPRINCE0.13050.107024
Alzheimers0.41860.37210.5581MRHydra0.13950.302323
AppliancesEnergy_disc0.80950.83330.9524FreshPRINCE0.14290.452424
EmoPain0.78310.84080.92681NN-DTW0.14370.242320
PhotoStimulation0.41670.38890.5833ROCKET0.16670.388923
FaceDetection0.50000.63250.6850H-InceptionTime0.18500.158623
BeijingPM25Quality_disc0.69770.87510.8879ConvTran0.19020.128824
HouseholdPowerConsumption1_disc0.77840.91470.9825FreshPRINCE0.20410.218724
LowCost0.50000.63750.7300TSF0.23000.248324
BoneProbAgeGroup0.47640.65170.7124H-InceptionTime0.23600.193324
StandWalkJump0.33330.43330.6000MRHydra0.26670.400024
CrowdSourced0.50020.71270.7734LITETime-MV0.27320.176824
BenzeneConcentration_disc0.68970.82010.9768STSF0.28700.577024
FordChallenge0.62320.88960.9360QUANT0.31280.312823
BIDMC32HR_disc0.65070.80830.9637RIST0.31300.635322
STEW0.50000.74020.8385Arsenal0.33850.210022
BoneIntensitiesAgeGroup0.47640.79890.8202HC20.34380.296624
PhonemeSpectra0.02560.27890.3746H-InceptionTime0.34890.291124
KERAAL-RTK0.57140.78570.9286HC20.35710.571424
LSST0.31510.63080.7040FreshPRINCE0.38890.480924
HandMovementDirection0.20270.41890.6081TSF0.40540.418924
DuckDuckGeese0.20000.46000.6400H-InceptionTime0.44000.480024
SelfRegulationSCP10.50170.85320.9454MRHydra0.44370.208224
IEEEPPG_disc0.26050.43980.7078ConvTran0.44730.438324
AsphaltRegularityCoordinates0.50730.97870.9947H-InceptionTime0.48740.291624
MindReading0.23120.52830.7243LITETime-MV0.49310.332324
EthanolConcentration0.25100.43350.7490STC0.49810.532324
UIPRMD-DS-C0.50000.83331.0000Catch220.50000.388924
WISDM0.36640.86580.8965MRHydra0.53000.131024
Blink0.44440.99221.0000Arsenal0.55560.415624
EigenWorms0.41980.89310.9771MRHydra0.55730.557323
KIMORE-PR-C0.14290.42860.7143LITETime-MV0.57140.571424
AsphaltObstaclesCoordinates0.28390.82100.8670MRHydra0.58310.289024
CounterMovementJump0.33520.74860.9274Arsenal0.59220.458124
Handwriting0.03760.37290.6529H-InceptionTime0.61530.478824
USCActivity0.11380.69240.7354LITETime-MV0.62160.137920
RacketSports0.28290.88160.9079RDST0.62500.125024
UCDHE-Rowing-MC0.20450.73410.8295PatchMTSC0.62500.338624
IRDS-SFL0.20690.78880.8621RDST0.65520.448324
Skoda0.23560.94730.9646H-InceptionTime0.72900.119923
Epilepsy0.26810.98551.0000HC20.73190.101424
Tiselac0.06280.81860.8373STSF0.77450.204419
MotionSenseHAR0.20380.98681.0000DrCIF0.79620.101924
NATOPS0.16670.88610.9667LITETime-MV0.80000.155624
UCIActivity0.19160.97520.9983LITETime-MV0.80670.174924
UWaveGestureLibrary0.12500.90620.9406Arsenal0.81560.553124
ERing0.16670.93330.9963MRHydra0.82960.237024
PEMS-SF0.11560.93061.0000CIF0.88440.317924
PenDigits0.10380.97780.9911H-InceptionTime0.88740.234422
SpokenArabicDigits0.10000.97820.9941RDST0.89400.128724
Libras0.06670.88890.9722RIST0.90560.338924
JapaneseVowels0.08380.96080.9946LiteTIME0.91080.208124
Cricket0.08330.97921.00001NN-DTW0.91670.069424
CharacterTrajectories0.06480.98890.9958H-InceptionTime0.93110.044624
TactileTextureRecognition0.05140.99851.0000H-InceptionTime0.94860.168924
ArticularyWordRecognition0.04000.98170.9933Arsenal0.95330.050024

One row per dataset. Dummy is the no-skill floor. Median, best and spread are over the other estimators, so the baseline cannot flatter them. Gain over dummy is best minus dummy, how much skill was found at all; spread is best minus worst, how much the choice of estimator mattered. The two answer different questions, and a single range would conflate them.

4 of 66 datasets gained 0.05 or less over the baseline (shaded amber) and 15 have a best of 0.99 or more (shaded green). Both separate estimators poorly, for opposite reasons. Best is a maximum over many estimators, so it is optimistic by construction: read it as what the archive can currently do on a problem, not as what any one method delivers.