Skip to content

README.md

Latest commit

 

History

History
102 lines (57 loc) · 4.37 KB

README.md

File metadata and controls

102 lines (57 loc) · 4.37 KB
Raw

ATC-138 Python Numerical Validation Testing Against MATLAB

1. Overview

This documentation provides instructions on how to validate ATC-138 Python implementation against the original MATLAB framework for numerical consistency and correct reproduction of calculations.

The focus is on the standalone use of the comparison engine (compare_runs.py) for users who need to validate their own Python and MATLAB models.

The goal is to:

  • Explain the testing metrics used
  • Outline the testing workflow

This workflow is independent of the integration tests and can be applied to any model. Two example models are included here for demonstration purposes.

2. Example Models Used

Model Description
ICSB 6-story special moment frame
haseltonRCMF_4story 4-story reinforced concrete moment frame building

3. Validation Workflow

  1. Run the MATLAB ATC-138 implementation using the provided example inputs.
  2. Run the Python ATC-138 implementation using the same demand/data.
  3. Extract both output files recovery_outputs.
  4. Compare Python and MATLAB outputs using quantitative metrics using compare_runs.py.
  5. Evaluate whether results fall within acceptable tolerance thresholds.

4. Comparison Metrics

Four quantitative metrics are used to evaluate agreement between MATLAB and Python results:


Metric 1 — Absolute difference of mean functional recovery time

| mean_FR_python − mean_FR_matlab |

Units: days

Threshold: ≤ 2 days


Metric 2 — Relative difference of mean functional recovery time

| mean_FR_python − mean_FR_matlab | / mean_FR_matlab

Units: percent

Threshold: ≤ 4 %


Metric 3 — Worst system mean absolute error (MAE)

System-level wise, the highest MAE of the fraction of unresolved damage over time, across all functional-recovery and reoccupancy systems.

Threshold: ≤ 0.02 (no more than 2% unresolved fraction across all systems)


Metric 4 — Worst system 95th percentile absolute error

System-level wise, the maximum 95th percentile absolute error of the fraction of unresolved damage over time, across all functional-recovery and reoccupancy systems.

Threshold: ≤ 0.04 (no more than of 4% unresolved fraction across all systems)


5. Testing thresholds

Both absolute and relative differences are reported to provide a more complete assessment: absolute difference reflects the magnitude of discrepancies, while relative difference normalizes them across models with different recovery scales. Using both avoids misleading interpretations. For example, a small absolute difference may appear large in relative terms for short recovery durations, while a small relative difference may still correspond to a large absolute difference for long recovery durations.

In the porting process from MATLAB to Python, the output Python results were compared against the reference MATLAB values. The test case is ideally averaged across multiple random seeds, to minimize variance between each realization. The test case is considered to adequately match if:

  1. At the high level, either Metric 1 or Metric 2 pass their thresholds. That is, either an absolute difference in mean recovery time $\leq$ 2 days or a relative difference in mean recovery time of $\leq$ 4% is considered passing.
  2. At the system level, both Metric 3 and Metric 4 pass their thresholds. That is, for all systems at each assessed point in time, the average difference in unresolved fraction is no more than 2%, and the 95th-percentile worst difference in unresolved fraction is no more than 4%.

6. Recovery Trajectory and System-Level Recovery Comparison

For each model, plots of recovery trajectories and system-level recovery curves are saved:

  • recovery_trajectory.jpg
  • system_breakdowns_functional.jpg
  • system_breakdowns_reoccupancy.jpg

7. Validation Results

The comparison results for the two example models are summarized below. Twenty batch runs were carried out for each model, and the mean values are compared against the defined metrics.

Model Absolute difference (days) Relative difference (%) Worst system MAE Worst system P95
ICSB 1.38 0.241 0.015 0.030
haseltonRCMF_4story 1.74 1.61 0.007 0.027

All comparison metrics here fall within the defined tolerance thresholds.