Feat/cross validation and statistical tests

[andyreus17]

Summary

Functionality has been added to evaluate models using cross-validation and statistical tests within cross-validation sessions. Frontend components were modified to accommodate the new integrated configurations and to display the resulting detailed data. Also, the backend data model was updated to store per-fold metrics, session configurations, statistical test results, and others.

Additionally, the model_job was refactored to abstract the process from evaluation strategies and data partitioning schemes, handling only the orchestration. The design pattern employed is the Strategy pattern, with the model_job acting as the context (capable of managing multiple strategies, all sharing the same objective).

---

Type of Change

Check all that apply like this [x]:

• [ X ] Backend change
• [ X ] Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

Below are the most relevant modifications/additions to files introduced.

Data model

• DashAI/back/dependencies/database/models.py:
  • ModelSession: The evaluation_strategy field is added to be able to store the session's evaluation strategy (holdout or cross-validation).
  • Run: The nested field is added to be able to store the inner splitter configuration when performing nested cross-validation.
  • Metric: The std_value field is added to attach the standard deviation associated with a final metric, useful in the case of cross-validation so it doesn't have to be recomputed each time, saving computation; the fold_index and inner_fold_index fields are added, representing the outer and inner fold index, respectively, of a metric obtained in nested cross-validation, for a complete identification of the metric's origin.
  • StatisticalTest: This model is created to store the complete information of a statistical test executed and saved by a user.

Cross validation

Backend

• DashAI/back/api/api_v1/endpoints/metrics.py: The retrieval of metrics for the live metrics websocket is modified to specify that these are trial-level metrics, so they are not mistakenly mixed with other metrics.
• DashAI/back/api/api_v1/endpoints/model_sessions.py: The selected evaluation strategy is added when creating the session.
• DashAI/back/api/api_v1/endpoints/runs.py:
  • get_metrics_for_run is modified to allow selecting the metrics level to be obtained. In addition, the standard deviation of the metrics is added when available (available for metrics with LAST level obtained in cross-validation) in the response and in the runs, in the same way it was already done previously with the metric values.
  • The nested field is added when uploading or updating a run.
  • Specific endpoints are added to obtain the metrics obtained in the cross-validation folds (both normal folds and outer folds, and by partition), and also to obtain their averaged values.
• DashAI/back/core/enums/metrics.py: New levels are added to store metrics:
  • FOLD: metric obtained in a normal cross-validation fold.
  • OUTER_FOLD: metric obtained belonging to an outer fold in nested cross-validation.
  • LAST_OUTER: Equivalent to LAST, but for outer folds. In cross-validation, the LAST level represents the summary metric of the process (average of the folds for each metric).
• DashAI/back/dataloaders/classes/dashai_dataset.py: split_dataset_cv is added, which fulfills a function analogous to the existing split_dataset, but adapted to the context of partitioning data in cross-validation and obtaining multiple partitions.
• DashAI/back/initial_components.py: The newly created components are added.
• DashAI/back/job/model_job.py: The data partitioning, training, and model evaluation process is refactored. The Strategy design pattern is used to be able to handle different data partitioning schemes and evaluation strategies. Now the logic is not contained in the ModelJob; it only orchestrates and delegates the data partitioning and model evaluation process.
• DashAI/back/metrics/classification/f1.py, precision.py, recall.py: A division-by-zero edge case is added so it doesn't raise an error and the value returned is zero.
• DashAI/back/models/base_model.py: Create a function similar to calculate_metrics that returns the scores instead of saving them to the database, to be used in the CV evaluation loop.
• DashAI/back/optimizers/base_optimizer.py, optuna_optimizer.py, hyperopt_optimizer.py: The optimize method now receives the target metric and the evaluation strategy to be used, implemented in each evaluation strategy. In addition, the signature is changed so that the optimizer is limited to delivering the model with the hyperparameters and the best values found, and the final training is performed in the evaluation strategy used by the optimizer, properly separating their responsibilities.

Furthermore, the architecture and abstraction devised for the evaluation strategies in DashAI/back/evaluation/* can be observed in this diagram:

On the other hand, the following applies to the various data partitioning schemes, created in DashAI/back/splitters/*:

Frontend

• DashAI/front/src/api/run.ts: The types are modified according to the changes explained in the DashAI/back/api/api_v1/endpoints/runs.py file.
• DashAI/front/src/components/models/AddModelDialog.jsx: An option is added to enable and configure the inner splitter of a nested cross-validation when the session is cross-validation.
• DashAI/front/src/components/models/CreateSessionSteps.jsx: The selection of the evaluation strategy (holdout or cross-validation) is added.
• DashAI/front/src/components/models/FoldMetricsChart.jsx: Component that collects a set of metrics obtained in cross-validation and shows different charts to visualize the metrics obtained in the cross-validation process.
• DashAI/front/src/components/models/LiveMetricsChart.jsx: Modified to not show the validation split when performing cross-validation, since it does not exist in that context.
• DashAI/front/src/components/models/ModelComparisonTable.jsx: Color is added to the table rows according to whether the run does HPO, does not do HPO, or does nested cross-validation, along with a small legend for the colors. Additionally, visualization of the standard deviation associated with the average value of each metric obtained in the execution of a run in a cross-validation session is added.
• DashAI/front/src/components/models/SessionVisualization.jsx: A tab is added to switch between viewing the general charts (as previously existed) or viewing the saved statistical tests.
• DashAI/front/src/components/models/modelSession/PrepareDatasetStep.jsx: Handling of the session configuration for cross-validation is added for its creation, similarly to what already existed for holdout and split sizes, but in this case with the number of folds, repetitions, grouping, etc.
• DashAI/front/src/components/models/modelSession/SplitDatasetRows.jsx: Support is added for configuring the session in the case of cross-validation, showing the available partitioning schemes for the task to be performed and adding configuration fields such as the number of folds, number of repetitions, among others, depending on the availability declared in the backend for each partitioning scheme. Message handling is also added for the values entered in these fields.
• DashAI/front/src/components/models/runResults/ResultsTabsHeader.jsx: The header is modified to be able to show and select the fold chart tabs and nested cross-validation metrics summary tab, in the metrics section, in an orderly manner.
• DashAI/front/src/pages/results/components/ResultsTabMetricsToggle.jsx: The validation partition selection toggle is disabled for the cross-validation case.
• DashAI/front/src/utils/i18n/locales/*/experiments.json, models.json, common.json: Full translation support is added for the components created/modified to incorporate cross-validation, for all languages.

Statistical tests

Backend

• DashAI/back/api/api_v1/api.py: The API created to handle statistical tests is added.
• DashAI/back/api/api_v1/endpoints/statistical_tests.py: Endpoints are added to handle statistical tests, both to execute them and for CRUD operations.
• DashAI/back/api/api_v1/schemas/statistical_tests_params.py: A schema is created for the expected data types that will travel through the statistical test endpoints.
• DashAI/back/statistical_tests/utils.py: Created to implement useful tools complementary to statistical tests. In this case, the correction of p-values through correction methods that are optionally used in the corresponding statistical test classes.
• pyproject.toml: statsmodels is added as a project dependency, used to run some statistical tests.

The implementation of the various statistical tests as concrete classes in DashAI/back/statistical_tests/* can be visualized using the following diagram:

Frontend

• DashAI/front/src/api/statisticalTests.ts: A type interface is created for the endpoints created in DashAI/back/api/api_v1/endpoints/statistical_tests.py.
• DashAI/front/src/components/models/ModelsRightBar.jsx: A catalog of statistical tests is added in a cross-validation session, visually similar to the catalog of available models.
• DashAI/front/src/components/models/OuterFoldMetricsTable.jsx: Component responsible for showing a summary table of the metrics obtained in nested cross-validation.
• DashAI/front/src/components/models/PerRunResults.jsx: Component to show the result of statistical tests that execute one test per run (for example, the Shapiro-Wilk normality test).
• DashAI/front/src/components/models/RunResults.jsx: New tabs are added to be able to select the fold charts in cross-validation and the nested cross-validation summary metrics, if applicable.
• DashAI/front/src/components/models/SingleTestResult.jsx: Component to show the result of statistical tests that are executed only once on a set of runs, not once per run as in the case of PerRunResults.
• DashAI/front/src/components/models/SavedStatisticalTestResults.jsx: Modal that shows the results of a saved statistical test, rendering PerRunResults or SingleTestResult, as appropriate for that statistical test.
• DashAI/front/src/components/models/StatisticalTestTable.jsx: Component responsible for listing the saved statistical tests along with their relevant information (name, date, etc.), and able to delete each saved test.
• DashAI/front/src/components/models/StatisticalTestsList.jsx: Lists the catalog of statistical tests available to run.
• DashAI/front/src/components/models/StatisticalTestsModal.jsx: Modal that opens when a statistical test is selected, where the metric to compare, the finished runs that participate, and specific configurations of each statistical test can be selected, such as the significance value, hypothesis direction, correction method, etc.
• DashAI/front/src/components/models/TechnicalDetails.jsx: Shows the technical details obtained from the backend in more detail in JSON form, such as the values used in the statistical test, the name of the runs that participated, creation date, etc.
• DashAI/front/src/components/models/modelSession/NestedCVSelector.jsx: Component that makes it possible to select the inner splitter configuration when enabling nested cross-validation.
• DashAI/front/src/types/statisticalTests.ts: A typed interface is added for the expected data that will travel through the created statistical tests API.
• DashAI/front/src/utils/i18n/locales/*/experiments.json, models.json, common.json: Full translation support is added for the statistical test components created, for all languages.

Testing

• tests/back/api/test_experiments_api.py, test_explainer_jobs.py, test_explanations_api.py, test_jobs.py, test_predict_api.py, test_run_download_gate.py, test_runs_api.py: Minimal changes are made to add the evaluation strategy of the runs involved in testing, since it is a non-nullable field in the modified data model.
• tests/back/api/test_statistical_tests_api.py: A test is added to test the statistical tests API, such as execution, listing, and deletion.
• tests/back/api/test_runs_api.py: A test is added that creates a session with cross-validation as the evaluation strategy and verifies that it exists correctly. It then creates a run associated with that session, verifies that its assigned data is correct and consistent, and finally deletes the session.