Ingest TS2Vec, and cap the probe as the authors do

README.md

@@ -40,33 +40,34 @@ The current paper version describes:
 <!-- LEADERBOARD:START -->
 | # | Estimator | Accuracy rank | Accuracy | Balanced accuracy | AUROC | F1 | Log loss &darr; | Sensitivity | Specificity |
 |---|---|---|---|---|---|---|---|---|---|
-| 1 | HC2 | **8.06** | **0.7909** | 0.7518 | **0.8990** | 0.7273 | **0.5383** | 0.7459 | **0.7943** |
-| 2 | MRHydra | 8.51 | 0.7837 | **0.7564** | 0.8105 | **0.7316** | 7.7974 | **0.7642** | 0.7757 |
-| 3 | RDST | 9.35 | 0.7734 | 0.7333 | 0.7912 | 0.6991 | 8.1667 | 0.7109 | 0.7874 |
-| 4 | RIST | 9.97 | 0.7720 | 0.7397 | 0.8748 | 0.7147 | 0.6218 | 0.7408 | 0.7655 |
-| 5 | DrCIF | 10.25 | 0.7747 | 0.7429 | 0.8813 | 0.7173 | 0.6484 | 0.7397 | 0.7708 |
-| 6 | FreshPRINCE | 10.30 | 0.7743 | 0.7487 | 0.8745 | 0.7211 | 0.6007 | 0.7414 | 0.7770 |
-| 7 | CIF | 10.37 | 0.7781 | 0.7471 | 0.8908 | 0.7212 | 0.6430 | 0.7441 | 0.7753 |
-| 8 | QUANT | 10.81 | 0.7720 | 0.7462 | 0.8831 | 0.7189 | 0.6175 | 0.7521 | 0.7581 |
-| 9 | Arsenal | 10.96 | 0.7680 | 0.7321 | 0.8457 | 0.7024 | 3.8631 | 0.7257 | 0.7732 |
-| 10 | ROCKET | 11.09 | 0.7690 | 0.7326 | 0.7925 | 0.7019 | 8.3249 | 0.7200 | 0.7764 |
-| 11 | LITETime-MV | 11.50 | 0.7506 | 0.7299 | 0.8513 | 0.6820 | 1.3206 | 0.7132 | 0.7637 |
-| 12 | STSF | 11.79 | 0.7724 | 0.7477 | 0.8804 | 0.7080 | 0.6432 | 0.7345 | 0.7826 |
-| 13 | H-InceptionTime | 11.93 | 0.7408 | 0.7190 | 0.8496 | 0.6838 | 1.3227 | 0.7223 | 0.7378 |
-| 14 | LiteTIME | 12.68 | 0.7341 | 0.7104 | 0.8394 | 0.6680 | 1.4776 | 0.7113 | 0.7336 |
-| 15 | PatchMTSC | 13.38 | 0.7428 | 0.6897 | 0.8261 | 0.6533 | 0.7655 | 0.6852 | 0.7352 |
-| 16 | ConvTran | 13.39 | 0.7462 | 0.7102 | 0.8592 | 0.6767 | 0.8190 | 0.7159 | 0.7345 |
-| 17 | Catch22 | 13.42 | 0.7475 | 0.7181 | 0.8697 | 0.6922 | 0.7147 | 0.7240 | 0.7374 |
-| 18 | STC | 14.25 | 0.7545 | 0.7172 | 0.8744 | 0.6940 | 0.6391 | 0.7185 | 0.7537 |
-| 19 | TSF | 14.26 | 0.7515 | 0.7236 | 0.8740 | 0.6883 | 0.7252 | 0.7093 | 0.7606 |
-| 20 | TDE | 15.07 | 0.7262 | 0.6813 | 0.8374 | 0.6382 | 0.8869 | 0.6714 | 0.7344 |
-| 21 | Summary | 17.18 | 0.6858 | 0.6574 | 0.8268 | 0.6230 | 0.9123 | 0.6574 | 0.6844 |
-| 22 | TimesNet | 17.26 | 0.7013 | 0.6659 | 0.8275 | 0.6281 | 1.1613 | 0.6726 | 0.6898 |
-| 23 | TimesURL | 17.46 | 0.6958 | 0.6533 | 0.7906 | 0.5967 | 1.0055 | 0.6257 | 0.6973 |
-| 24 | 1NN-DTW | 19.08 | 0.6712 | 0.6454 | 0.7197 | 0.6136 | 11.8506 | 0.6521 | 0.6636 |
-| 25 | Dummy | 22.68 | 0.3645 | 0.3029 | 0.5000 | 0.1507 | 1.4067 | 0.2855 | 0.3816 |
-
-Average over the 52 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold.
+| 1 | HC2 | **8.11** | **0.7887** | 0.7541 | **0.9000** | 0.7346 | **0.5440** | 0.7547 | **0.7910** |
+| 2 | MRHydra | 8.92 | 0.7810 | **0.7579** | 0.8130 | **0.7368** | 7.8942 | **0.7715** | 0.7718 |
+| 3 | RDST | 9.76 | 0.7707 | 0.7372 | 0.7963 | 0.7105 | 8.2660 | 0.7236 | 0.7833 |
+| 4 | RIST | 10.38 | 0.7693 | 0.7422 | 0.8755 | 0.7221 | 0.6294 | 0.7504 | 0.7613 |
+| 5 | DrCIF | 10.44 | 0.7721 | 0.7454 | 0.8821 | 0.7248 | 0.6558 | 0.7490 | 0.7669 |
+| 6 | CIF | 10.61 | 0.7756 | 0.7497 | 0.8920 | 0.7288 | 0.6497 | 0.7536 | 0.7714 |
+| 7 | FreshPRINCE | 10.63 | 0.7717 | 0.7516 | 0.8752 | 0.7293 | 0.6075 | 0.7515 | 0.7731 |
+| 8 | Arsenal | 11.04 | 0.7654 | 0.7340 | 0.8471 | 0.7092 | 3.9265 | 0.7337 | 0.7696 |
+| 9 | QUANT | 11.19 | 0.7693 | 0.7486 | 0.8839 | 0.7262 | 0.6238 | 0.7616 | 0.7539 |
+| 10 | LITETime-MV | 11.60 | 0.7476 | 0.7312 | 0.8518 | 0.6875 | 1.3300 | 0.7200 | 0.7600 |
+| 11 | ROCKET | 11.61 | 0.7661 | 0.7345 | 0.7955 | 0.7080 | 8.4299 | 0.7282 | 0.7724 |
+| 12 | STSF | 12.08 | 0.7698 | 0.7503 | 0.8813 | 0.7155 | 0.6493 | 0.7439 | 0.7790 |
+| 13 | H-InceptionTime | 12.28 | 0.7375 | 0.7205 | 0.8506 | 0.6897 | 1.3334 | 0.7303 | 0.7333 |
+| 14 | LiteTIME | 12.79 | 0.7308 | 0.7122 | 0.8402 | 0.6746 | 1.4921 | 0.7199 | 0.7291 |
+| 15 | ConvTran | 13.81 | 0.7430 | 0.7139 | 0.8606 | 0.6882 | 0.8300 | 0.7289 | 0.7295 |
+| 16 | PatchMTSC | 13.92 | 0.7395 | 0.6934 | 0.8288 | 0.6660 | 0.7748 | 0.6985 | 0.7300 |
+| 17 | Catch22 | 13.95 | 0.7442 | 0.7203 | 0.8703 | 0.6996 | 0.7238 | 0.7337 | 0.7326 |
+| 18 | STC | 14.61 | 0.7516 | 0.7188 | 0.8748 | 0.7004 | 0.6447 | 0.7264 | 0.7496 |
+| 19 | TSF | 14.73 | 0.7484 | 0.7257 | 0.8747 | 0.6952 | 0.7335 | 0.7179 | 0.7565 |
+| 20 | TS2Vec | 15.26 | 0.7212 | 0.6849 | 0.8082 | 0.6588 | 0.7326 | 0.6980 | 0.7100 |
+| 21 | TDE | 15.39 | 0.7230 | 0.6823 | 0.8383 | 0.6441 | 0.8859 | 0.6786 | 0.7301 |
+| 22 | Summary | 17.85 | 0.6814 | 0.6586 | 0.8263 | 0.6294 | 0.9251 | 0.6661 | 0.6787 |
+| 23 | TimesNet | 18.16 | 0.6971 | 0.6688 | 0.8280 | 0.6390 | 1.1785 | 0.6850 | 0.6838 |
+| 24 | TimesURL | 18.25 | 0.6916 | 0.6563 | 0.7931 | 0.6084 | 1.0193 | 0.6379 | 0.6914 |
+| 25 | 1NN-DTW | 19.71 | 0.6672 | 0.6457 | 0.7214 | 0.6193 | 11.9949 | 0.6584 | 0.6584 |
+| 26 | Dummy | 23.91 | 0.3538 | 0.2991 | 0.5000 | 0.1537 | 1.4284 | 0.2911 | 0.3695 |
+
+Average over the 51 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold.
 <!-- LEADERBOARD:END -->
 
 Rebuilt with `python -m multiverse.experiments.tables`, which also writes a sortable

multiverse/classification/_ts2vec.py

@@ -58,7 +58,7 @@
 import numpy as np
 from aeon.classification import BaseClassifier
 from sklearn.linear_model import LogisticRegression
-from sklearn.model_selection import GridSearchCV
+from sklearn.model_selection import GridSearchCV, train_test_split
 from sklearn.multiclass import OneVsRestClassifier
 from sklearn.pipeline import make_pipeline
 from sklearn.preprocessing import StandardScaler
@@ -101,6 +101,12 @@ class TS2VecClassifier(BaseClassifier):
         in the original.
     temporal_unit : int, default=0
         Minimum unit for temporal contrast, as in the original.
+    probe_max_samples : int or None, default=None
+        Cap on the number of cases the probe is fitted on. None takes the
+        authors' values, 10000 for the SVM probe and 100000 for the logistic
+        one, and the collection is subsampled with stratification when it is
+        larger. Without this the SVM grid search is unaffordable on the large
+        collections.
     probe : {"svm", "logistic"}, default="svm"
         Classifier fitted on the representations. ``"svm"`` reproduces the
         authors' UEA protocol, an ``SVC`` chosen by grid search over C on the
@@ -120,6 +126,8 @@ class TS2VecClassifier(BaseClassifier):
         Pretrained TS2Vec encoder.
     probe_ : object
         Classifier fitted on the encoded training collection.
+    probe_cases_ : int
+        Number of cases the probe was fitted on, after any subsampling.
     device_ : str
         Resolved device.
     n_channels_ : int
@@ -166,6 +174,7 @@ def __init__(
         max_train_length: int = 3000,
         temporal_unit: int = 0,
         probe: str = "svm",
+        probe_max_samples: int | None = None,
         device: str = "auto",
         verbose: bool = False,
         random_state=1234,
@@ -180,6 +189,7 @@ def __init__(
         self.max_train_length = max_train_length
         self.temporal_unit = temporal_unit
         self.probe = probe
+        self.probe_max_samples = probe_max_samples
         self.device = device
         self.verbose = verbose
         self.random_state = random_state
@@ -217,6 +227,25 @@ def _to_original_layout(X: np.ndarray) -> np.ndarray:
         """Convert an aeon collection to the authors' (case, time, channel)."""
         return np.transpose(np.asarray(X, dtype=np.float32), (0, 2, 1))
 
+    def _subsample(self, features, y, seed):
+        """Cap the collection the probe is fitted on, as the authors do.
+
+        ``fit_svm`` takes ``MAX_SAMPLES=10000`` and ``fit_lr`` 100000, and both
+        subsample with stratification before fitting. This is not a detail:
+        the SVM grid is ten values of C over five folds, and an RBF SVC is
+        between quadratic and cubic in the number of cases, so without the cap
+        the probe alone can outrun a 60 hour job on the larger collections.
+        """
+        limit = self.probe_max_samples
+        if limit is None:
+            limit = 10_000 if self.probe == "svm" else 100_000
+        if features.shape[0] <= limit:
+            return features, y
+        features, _, y, _ = train_test_split(
+            features, y, train_size=limit, random_state=0, stratify=y
+        )
+        return features, y
+
     def _build_probe(self, n_cases: int, seed: int):
         """Return the probe, following the authors' evaluation protocols.
 
@@ -294,7 +323,11 @@ def _fit(self, X: np.ndarray, y):
         encoded_y = np.asarray(
             [self._class_dictionary[label] for label in y], dtype=np.int64
         )
-        self.probe_ = self._build_probe(len(encoded_y), seed).fit(encoded, encoded_y)
+        fit_features, fit_y = self._subsample(encoded, encoded_y, seed)
+        self.probe_cases_ = int(fit_features.shape[0])
+        self.probe_ = self._build_probe(
+            self.probe_cases_, seed
+        ).fit(fit_features, fit_y)
         return self
 
     def _check_shape(self, X: np.ndarray) -> None:

results/multiverse/TS2Vec/TS2Vec_accuracy.csv

@@ -0,0 +1,62 @@
+Resamples:,0
+Alzheimers,0.4186046511627907
+AppliancesEnergy_disc,0.8095238095238095
+ArticularyWordRecognition,0.98
+AsphaltObstaclesCoordinates,0.7851662404092071
+AsphaltRegularityCoordinates,0.9893475366178429
+AtrialFibrillation,0.5333333333333333
+AutomotiveRoadTrials,0.7662337662337663
+BIDMC32HR_disc,0.7661525635681534
+BIDMC32SpO2_disc,0.5360566902876198
+BeijingPM10Quality_disc,0.8248811410459588
+BeijingPM25Quality_disc,0.8817353407290016
+BenzeneConcentration_disc,0.7848150300213055
+Blink,0.6511111111111111
+BoneIntensitiesAgeGroup,0.5887640449438202
+BoneProbAgeGroup,0.5280898876404494
+CharacterTrajectories,0.9937325905292479
+CounterMovementJump,0.6759776536312849
+Cricket,0.9861111111111112
+CrowdSourced,0.7147899752912107
+DuckDuckGeese,0.48
+ERing,0.837037037037037
+EigenWorms,0.8244274809160306
+Epilepsy,0.9637681159420289
+EthanolConcentration,0.2623574144486692
+EyesOpenShut,0.5
+FaceDetection,0.514472190692395
+FordChallenge,0.7882547559966915
+HandMovementDirection,0.25675675675675674
+Handwriting,0.40941176470588236
+Heartbeat,0.7804878048780488
+HouseholdPowerConsumption1_disc,0.9052478134110787
+HouseholdPowerConsumption2_disc,0.7128279883381924
+IEEEPPG_disc,0.39231927710843373
+IRDS-SFL,0.8620689655172413
+JapaneseVowels,0.972972972972973
+KERAAL-RTK,0.6428571428571429
+KIMORE-PR-C,0.2857142857142857
+KINECAL-QSEO,0.9411764705882353
+LSST,0.5766423357664233
+Libras,0.8888888888888888
+LowCost,0.53
+MindReading,0.33843797856049007
+MotionSenseHAR,0.9660377358490566
+MotorImagery,0.5
+NATOPS,0.9277777777777778
+PEMS-SF,0.8265895953757225
+PenDigits,0.9868496283590623
+PhonemeSpectra,0.23560990158067402
+PhotoStimulation,0.3888888888888889
+RacketSports,0.868421052631579
+STEW,0.6877104377104377
+SelfRegulationSCP1,0.757679180887372
+Skoda,0.9405730456314114
+SpokenArabicDigits,0.9818099135970896
+StandWalkJump,0.3333333333333333
+TactileTextureRecognition,0.9926578560939795
+UCDHE-Rowing-MC,0.6545454545454545
+UCIActivity,0.9912536443148688
+UIPRMD-DS-C,1.0
+UWaveGestureLibrary,0.909375
+WISDM,0.8049063163994592

results/multiverse/TS2Vec/TS2Vec_auroc.csv

@@ -0,0 +1,62 @@
+Resamples:,0
+Alzheimers,0.5646872493985565
+AppliancesEnergy_disc,0.4411764705882353
+ArticularyWordRecognition,0.9978587962962961
+AsphaltObstaclesCoordinates,0.9428780736122715
+AsphaltRegularityCoordinates,0.9982620415691282
+AtrialFibrillation,0.62
+AutomotiveRoadTrials,0.7704174228675136
+BIDMC32HR_disc,0.8476332651663594
+BIDMC32SpO2_disc,0.3377351053046514
+BeijingPM10Quality_disc,0.8753902205104104
+BeijingPM25Quality_disc,0.9305178161163344
+BenzeneConcentration_disc,0.8549342982883301
+Blink,0.79414
+BoneIntensitiesAgeGroup,0.7540972473907201
+BoneProbAgeGroup,0.7108969782840555
+CharacterTrajectories,0.9999629286634228
+CounterMovementJump,0.897687901976433
+Cricket,0.9995791245791246
+CrowdSourced,0.7663625507856577
+DuckDuckGeese,0.79
+ERing,0.9865020576131686
+EigenWorms,0.9599065200974137
+Epilepsy,0.993060614589887
+EthanolConcentration,0.5203658830687841
+EyesOpenShut,0.23129251700680273
+FaceDetection,0.5197674838081274
+FordChallenge,0.8475737447710174
+HandMovementDirection,0.5015481197684587
+Handwriting,0.8724933352072651
+Heartbeat,0.7336415362731151
+HouseholdPowerConsumption1_disc,0.9622873349469153
+HouseholdPowerConsumption2_disc,0.635829188983623
+IEEEPPG_disc,0.5898166629567041
+IRDS-SFL,0.9551630434782609
+JapaneseVowels,0.9991795362939342
+KERAAL-RTK,0.9583333333333334
+KIMORE-PR-C,0.5
+KINECAL-QSEO,0.125
+LSST,0.871763658336968
+Libras,0.9939153439153439
+LowCost,0.5648222222222222
+MindReading,0.6572383950269114
+MotionSenseHAR,0.9987337932160057
+MotorImagery,0.4576
+NATOPS,0.9905925925925926
+PEMS-SF,0.9610942578494267
+PenDigits,0.9996544727334365
+PhonemeSpectra,0.8732441261763364
+PhotoStimulation,0.38888685388685396
+RacketSports,0.9622508105125198
+STEW,0.7450874643051038
+SelfRegulationSCP1,0.859658932065977
+Skoda,0.9942989727741186
+SpokenArabicDigits,0.9998471945840186
+StandWalkJump,0.5133333333333334
+TactileTextureRecognition,0.999977297010771
+UCDHE-Rowing-MC,0.8770285580653228
+UCIActivity,0.9996600092798027
+UIPRMD-DS-C,1.0
+UWaveGestureLibrary,0.9842299107142857
+WISDM,0.9455932026252636

results/multiverse/TS2Vec/TS2Vec_balacc.csv

@@ -0,0 +1,62 @@
+Resamples:,0
+Alzheimers,0.3333333333333333
+AppliancesEnergy_disc,0.5
+ArticularyWordRecognition,0.98
+AsphaltObstaclesCoordinates,0.7829203036336215
+AsphaltRegularityCoordinates,0.9894622969426119
+AtrialFibrillation,0.5333333333333333
+AutomotiveRoadTrials,0.6147912885662432
+BIDMC32HR_disc,0.6099722691239187
+BIDMC32SpO2_disc,0.38175965079332574
+BeijingPM10Quality_disc,0.7709771465471456
+BeijingPM25Quality_disc,0.8565692300707852
+BenzeneConcentration_disc,0.7251881862078469
+Blink,0.673
+BoneIntensitiesAgeGroup,0.6180629197596325
+BoneProbAgeGroup,0.5133856219612237
+CharacterTrajectories,0.9936616128047581
+CounterMovementJump,0.6761770244821091
+Cricket,0.9861111111111112
+CrowdSourced,0.7148507429956661
+DuckDuckGeese,0.4800000000000001
+ERing,0.8370370370370371
+EigenWorms,0.7367683524205263
+Epilepsy,0.9644276629570747
+EthanolConcentration,0.26223776223776224
+EyesOpenShut,0.5
+FaceDetection,0.514472190692395
+FordChallenge,0.7562525023484745
+HandMovementDirection,0.2511904761904762
+Handwriting,0.41205945477865136
+Heartbeat,0.648411569464201
+HouseholdPowerConsumption1_disc,0.8050103033858279
+HouseholdPowerConsumption2_disc,0.34276094276094277
+IEEEPPG_disc,0.4025443119105505
+IRDS-SFL,0.8976449275362319
+JapaneseVowels,0.9735600081540015
+KERAAL-RTK,0.5833333333333334
+KIMORE-PR-C,0.5833333333333334
+KINECAL-QSEO,0.5
+LSST,0.33480027802363316
+Libras,0.8888888888888888
+LowCost,0.53
+MindReading,0.3330829762368523
+MotionSenseHAR,0.9726266887422225
+MotorImagery,0.5
+NATOPS,0.9277777777777777
+PEMS-SF,0.828463163245772
+PenDigits,0.9868256827975281
+PhonemeSpectra,0.23568697604265318
+PhotoStimulation,0.3191919191919192
+RacketSports,0.8789244186046512
+STEW,0.6877104377104377
+SelfRegulationSCP1,0.7580607585499954
+Skoda,0.9360292747896445
+SpokenArabicDigits,0.9818140307181403
+StandWalkJump,0.3333333333333333
+TactileTextureRecognition,0.9924961273745948
+UCDHE-Rowing-MC,0.6597063492063493
+UCIActivity,0.9915942256095708
+UIPRMD-DS-C,1.0
+UWaveGestureLibrary,0.909375
+WISDM,0.5025863399079105

results/multiverse/TS2Vec/TS2Vec_f1.csv

@@ -0,0 +1,62 @@
+Resamples:,0
+Alzheimers,0.24704536789935186
+AppliancesEnergy_disc,0.0
+ArticularyWordRecognition,0.9802171683389075
+AsphaltObstaclesCoordinates,0.7857148794795134
+AsphaltRegularityCoordinates,0.9892761394101877
+AtrialFibrillation,0.5075757575757576
+AutomotiveRoadTrials,0.4
+BIDMC32HR_disc,0.751819920016781
+BIDMC32SpO2_disc,0.02794759825327511
+BeijingPM10Quality_disc,0.6797101449275362
+BeijingPM25Quality_disc,0.8021213125621478
+BenzeneConcentration_disc,0.620948481746844
+Blink,0.689108910891089
+BoneIntensitiesAgeGroup,0.5856665137516777
+BoneProbAgeGroup,0.533632015343325
+CharacterTrajectories,0.9937505422521032
+CounterMovementJump,0.6658438126498493
+Cricket,0.9860139860139859
+CrowdSourced,0.7566265060240964
+DuckDuckGeese,0.4276193991289584
+ERing,0.8382191530713408
+EigenWorms,0.8090221408586924
+Epilepsy,0.9635825846846003
+EthanolConcentration,0.25577488452660363
+EyesOpenShut,0.6666666666666666
+FaceDetection,0.538191632928475
+FordChallenge,0.6903225806451613
+HandMovementDirection,0.25935254772052646
+Handwriting,0.34020295702854986
+Heartbeat,0.47058823529411764
+HouseholdPowerConsumption1_disc,0.9034660695388977
+HouseholdPowerConsumption2_disc,0.6066867263035853
+IEEEPPG_disc,0.39102139602942654
+IRDS-SFL,0.7419354838709677
+JapaneseVowels,0.9731909275283056
+KERAAL-RTK,0.2857142857142857
+KIMORE-PR-C,0.2857142857142857
+KINECAL-QSEO,0.0
+LSST,0.5261253925694681
+Libras,0.8890296477542855
+LowCost,0.5688073394495413
+MindReading,0.33680667083617616
+MotionSenseHAR,0.9663060132611815
+MotorImagery,0.6666666666666666
+NATOPS,0.9255380245641962
+PEMS-SF,0.8243029470599211
+PenDigits,0.9868406552005395
+PhonemeSpectra,0.2311725963438298
+PhotoStimulation,0.2693452380952381
+RacketSports,0.8682293583609373
+STEW,0.6763594068043036
+SelfRegulationSCP1,0.7815384615384615
+Skoda,0.940509648539942
+SpokenArabicDigits,0.9818409944516879
+StandWalkJump,0.31111111111111106
+TactileTextureRecognition,0.9926548515104918
+UCDHE-Rowing-MC,0.6585683842542325
+UCIActivity,0.9912527097593792
+UIPRMD-DS-C,1.0
+UWaveGestureLibrary,0.9069046125356246
+WISDM,0.7869545005215961