MMA (Momentum Mamba) is a multimodal Human Activity Recognition (HAR) framework built on top of the Mamba selective state-space model (SSM). The key innovation is a second-order momentum mechanism injected into Mamba's SSM scan, enabling richer temporal dynamics with minimal parameter overhead.
The framework fuses RGB-D video and 6-axis IMU sensor data for 27-class action recognition on the UTD-MHAD dataset, achieving 90.93% accuracy with state-of-the-art missing-modality robustness.
- Momentum SSM — Second-order exponential momentum buffer smooths noisy sensor inputs and captures sustained motion patterns
- Cross-Mamba Fusion — Deep cross-modal interaction through a shared selective SSM, preserving full temporal alignment
- Feature-level Modality Dropout (MD-Drop) — Asymmetric dropout makes missing-modality inference in-distribution
- Cross-Modal Alignment Regularization (CMAR) — Prevents weaker modality collapse via soft representation alignment
The project also includes a complete real-time deployment system (MMA-Deploy) with custom ESP32-C3 IMU hardware, a FastAPI inference server, and a browser-based visualization dashboard.
| Method | Full (%) | RGBD-only (%) | IMU-only (%) | Avg (%) | Degradation (%) |
|---|---|---|---|---|---|
| R4 baseline (no dropout) | 0.8977 | 0.1419 | 0.7302 | 0.5900 | 0.7558 |
| MD4 (dataset-level dropout) | 0.8930 | 0.3558 | 0.8767 | 0.7085 | 0.5372 |
| MDdrop1 (feature-level, imu_p=0.20) | 0.8860 | 0.3465 | 0.8488 | 0.6938 | 0.5395 |
| CMAR1 (MD-Drop + CMAR) | 0.9093 | 0.4163 | 0.8628 | 0.7295 | 0.4930 |
| Method | Clean | Gaussian Noise | Per-Channel Missing | Per-Sensor Missing | Combined | Avg Corrupt | Ratio |
|---|---|---|---|---|---|---|---|
| GRUFusion | 66.7% | 34.9% | 43.8% | 51.7% | 27.3% | 39.1% | 0.585 |
| MuMu | 61.2% | 60.4% | 40.1% | 44.5% | 40.9% | 47.5% | 0.777 |
| TransFusion | 74.7% | 52.4% | 47.0% | 59.7% | 39.4% | 49.8% | 0.668 |
| MMA-CMAR1 | 90.9% | 87.7% | 68.5% | 65.1% | 69.9% | 74.0% | 0.813 |
| Pipeline | Modalities | Best Accuracy | Parameters | Checkpoint |
|---|---|---|---|---|
rgbd_imu (CMAR1) |
RGBD + IMU | 90.93% | 12.4M | rgbd_imu_CMAR1.pt |
skel_imu (v4) |
Skeleton + IMU | 97.02% | 1.06M | mma_skel_imu_v4.pt |
tri_modal (CMAR34) |
RGB / Depth / IMU | 96.73% | 35M | tri_modal_cmar34.pt |
utdmad |
IMU-only | 88%+ | 321K | mma_best.pt |
Standard Mamba uses a first-order recurrence:
MMA extends this with second-order exponential momentum:
The momentum buffer
[RGBD_seq + mod_embed_rgbd || IMU_seq + mod_embed_imu]
↓ shared MomentumMambaBlock (cross-modal SSM scan)
↓ AttentionPool (learned temporal weighting)
↓ Classification Head
Both modality sequences are concatenated with learnable modality embeddings and passed through a shared MambaBlock, enabling deep bidirectional cross-modal interaction through the SSM's input-dependent gating. Unlike late fusion, Cross-Mamba preserves full temporal structure.
MD-Drop (training only): Randomly zero out one modality's encoded features before fusion.
md_drop_imu = 0.35: 35% of batches train on RGBD-only pathmd_drop_rgbd = 0.10: 10% of batches train on IMU-only path
CMAR: Soft representation alignment loss preventing the weaker modality (RGBD) from collapsing:
Combined total loss:
MMA/
├── model/ # Neural network architectures | 模型架构
│ ├── mamba.py # MomentumSSM, MomentumMambaBlock (core SSM)
│ ├── encoders.py # RGBDEncoder, IMUEncoder, SkeletonEncoder
│ ├── fusion.py # Cross-Mamba, Attention, Gated, Concat fusion
│ ├── layers.py # RMSNorm, GatedLinear
│ ├── mma_rgbd_imu.py # MultimodalMMA with MD-Drop + CMAR ★
│ ├── mma_skel_imu.py # Skeleton + IMU variant (deploy-ready)
│ ├── mma_tri_modal.py # RGB / Depth / IMU (3-way split)
│ ├── mma_utdmad.py # IMU-only baseline
│ ├── mma_mmtsa.py # MMTSA (Depth + IMU)
│ ├── mma_rgbd_imu_placement.py # Placement experiment variant
│ ├── mma_rgbd_imu_streaming.py # Streaming inference variant
│ ├── cleaning_dae.py # Denoising autoencoder
│ └── backbones/
│ ├── pretrained_cnn.py # ResNet18 (4-ch RGBD, partial freeze)
│ ├── convnextv2.py # ConvNeXt-V2 (via timm)
│ ├── spatial_cnn.py # Lightweight spatial CNN
│ └── light_cnn.py # SE attention CNN
│
├── datasets/ # Data loaders | 数据加载器
│ ├── utd_rgbd_imu.py # RGBD + IMU (main pipeline)
│ ├── utd_rgbd_imu_wrist.py # Wrist-only subset (actions 1-21)
│ ├── utd_rgbd_imu_masked.py # With foreground masking
│ ├── utd_rgbd_imu_maskcrop.py # Mask + crop variant
│ ├── utd_rgb_depth_imu.py # RGB / Depth / IMU (tri-modal)
│ ├── utd_rgb_depth_imu_wrist.py # Tri-modal wrist subset
│ ├── utd_skel_imu.py # Skeleton + IMU
│ ├── utd_inertial.py # IMU-only
│ ├── utd_depth_imu.py # Depth + IMU (MMTSA)
│ ├── utd_mhad.py # Base dataset wrapper
│ ├── groups.py # Cross-subject train/test splits
│ └── transforms.py # Augmentation, GAF encoding
│
├── train/ # Training pipeline | 训练流程
│ ├── run_train.py # Unified entry point (pipeline registry)
│ ├── default_train.py # Training loop (CE + aux + CMAR loss)
│ ├── staged_train.py # Progressive unfreezing
│ ├── mumu_train.py # MuMu baseline trainer
│ └── train_dae.py # Denoising autoencoder trainer
│
├── infer/ # Inference & evaluation | 推理评估
│ └── run_infer.py # Evaluation with --eval_missing support
│
├── baselines/ # Baseline implementations | 基线实现
│ ├── MuMu/ # MuMu collaborative multi-task fusion
│ ├── GRUFusion/ # GRU-based fusion baseline
│ ├── TransFusion/ # Transformer fusion baseline
│ ├── BASELINES.md # Baseline results
│ └── RUNNING_BASELINES.md # Reproduction guide
│
├── vis/ # Visualization toolkit | 可视化工具
│ ├── training.py # Training curves, LR schedules
│ ├── evaluation.py # Confusion matrix, per-class metrics
│ ├── data_inspection.py # IMU/depth/RGBD data inspection
│ ├── model.py # Gradient flow, t-SNE, attention maps
│ └── preview_dataset.py # Dataset sample viewer
│
├── script/ # Utility scripts | 工具脚本
│ ├── extract_skel_mediapipe.py # MediaPipe skeleton extraction
│ ├── build_comparison_table.py # Aggregate experiment results
│ ├── viz_depth_mask.py # Depth mask visualization
│ ├── run_train.ps1 # PowerShell training launcher
│ └── run_infer.ps1 # PowerShell inference launcher
│
├── figures/ # Architecture diagrams | 架构图
│ ├── fig1_arch.png # Main MMA architecture
│ ├── fig2_trimodal.png # Tri-modal pipeline
│ ├── fig3_skel_pipeline.png # Skeleton extraction pipeline
│ └── fig4_deploy.png # Deployment architecture
│
├── checkpoints/ # Saved model weights (gitignored)
├── datasets/UTD-MHAD/ # Dataset files (gitignored)
├── runs/ # TensorBoard logs (gitignored)
│
├── requirements.txt # Python dependencies
├── environment.yml # Conda environment
├── REPO_GUIDE.md # Detailed developer guide (Chinese)
├── FINAL_REPORT.md # Full technical report
├── experiment_report.md # Ablation study & experiment log
└── LICENSE # MIT License
- Python 3.10+
- CUDA 13.0 (for GPU training)
- Conda (recommended) or pip
# Option 1: Conda (recommended)
conda env create -f environment.yml
conda activate mma
# Option 2: pip
pip install -r requirements.txtDownload UTD-MHAD and extract to:
datasets/UTD-MHAD/
├── Depth/ # a{action}_s{subject}_t{trial}_depth.mat
├── Inertial/ # a{action}_s{subject}_t{trial}_inertial.mat
├── RGB/ # a{action}_s{subject}_t{trial}_color.avi
└── Skeleton/ # a{action}_s{subject}_t{trial}_skeleton.mat
# Best configuration: RGBD + IMU with CMAR (90.93%)
python train/run_train.py --pipeline rgbd_imu --data_root datasets/UTD-MHAD \
--epochs 100 --batch_size 16 --lr 3e-4 --weight_decay 0.008 \
--scheduler cosine_warmup --warmup_epochs 5 --patience 40 --seed 42 \
--label_smoothing 0.1 --mixup_alpha 0.15 \
--save_name rgbd_imu_CMAR1.pt --tb_dir runs \
--dataset_kwargs '{"augment":true}' \
--model_kwargs '{"d_model":160,"fusion":"cross_mamba","aux_weight":0.1,"encoder":"pretrained","freeze":"partial","temporal_velocity":true,"md_drop_imu":0.35,"md_drop_rgbd":0.10,"cmar_weight":0.1,"cmar_proj_dim":64}'
# IMU-only baseline
python train/run_train.py --pipeline utdmad --data_root datasets/UTD-MHAD/Inertial
# Skeleton + IMU (deployment-ready, 97.02%)
python train/run_train.py --pipeline skel_imu --data_root datasets/UTD-MHAD \
--model_kwargs '{"d_model":128,"n_layers":2}'# Full evaluation with missing-modality robustness test
python infer/run_infer.py --pipeline rgbd_imu --data_root datasets/UTD-MHAD \
--checkpoint checkpoints/rgbd_imu_CMAR1.pt --eval_missing all
# Generate confusion matrix and per-class metrics
python infer/run_infer.py --pipeline rgbd_imu --data_root datasets/UTD-MHAD \
--checkpoint checkpoints/rgbd_imu_CMAR1.pt --vis_dir vis_resultstensorboard --logdir runsTracked metrics: Loss/train, Loss/val, Accuracy/train, Accuracy/val, F1/val, LR, hyperparameter correlation.
All pipelines are managed through a unified entry point. Use --pipeline to select:
| Pipeline | Model Class | Dataset | Modalities | Description |
|---|---|---|---|---|
rgbd_imu |
MultimodalMMA |
UTDMADRGBDIMUDataset |
RGBD + IMU | Main pipeline with Cross-Mamba + CMAR |
skel_imu |
SkelIMU_MMA |
UTDMADSkelIMUDataset |
Skeleton + IMU | Lightweight deploy variant |
tri_modal |
TriModalMMA |
UTDRGBDepthIMUDataset |
RGB / Depth / IMU | Three separate visual streams |
utdmad |
MomentumMambaHAR |
UTDMADInertialDataset |
IMU-only | Smallest model (321K params) |
mmtsa |
MMA_MMTSA |
UTD_MHAD_Dataset |
Depth + IMU | Temporal pyramid fusion |
mumu |
MuMu |
(user-provided) | Configurable | Collaborative multi-task baseline |
Custom models can be trained without modifying code via --model_module / --model_class / --dataset_module / --dataset_class.
The complete real-time deployment system is maintained in a separate repository: MMA-Deploy
ESP32-C3 x6 (200Hz IMU) ──UDP──┐
│
Depth camera ──USB──┐ ├──▶ Python server (FastAPI)
│ │ ├── IO (IMU UDP + Depth Camera)
└───────────┘ ├── Sync / Fusion / Aligner
├── ONNX / PyTorch Inference
├── Recorder (training-compatible)
└── WebSocket broadcaster
│
▼
Browser dashboard (Chart.js + Three.js)
- Hardware: 6x custom ESP32-C3 IMU nodes with ICM-42688-P sensor (200 Hz), KiCAD PCB design
- Server: FastAPI async server with ONNX Runtime / PyTorch inference at 10 Hz
- Frontend: Real-time WebSocket dashboard with 3D orientation visualization (Three.js), prediction charts (Chart.js)
- Firmware: PlatformIO / ESP-IDF based, UDP broadcast protocol
- Bridge: Training repo exports
.onnx+.meta.jsonartifacts consumed by deploy. Recordings are training-compatible (.npy+.png+meta.yaml).
| Component | Tool | Description |
|---|---|---|
| PCB | KiCAD | 6x ESP32-C3 + ICM-42688-P IMU sensor nodes |
| Wrist Mount | SolidWorks / 3MF | 3D-printed wrist attachment for sensor placement |
UTD Multimodal Human Action Dataset: 27 action classes, 8 subjects, ~861 samples.
- Train: subjects 1, 3, 5, 7 (431 samples)
- Test: subjects 2, 4, 6, 8 (430 samples)
| Modality | Sampling | Preprocessing | Shape |
|---|---|---|---|
| RGB | 30 fps | Resize 112x112, normalize [0,1] | (T=16, 3, 112, 112) |
| Depth | 30 fps | Resize 112x112, normalize [0,1] | (T=16, 1, 112, 112) |
| IMU | 200 Hz | Z-score normalization | (T=192, 6) |
27 Action Classes
| ID | Action | ID | Action | ID | Action |
|---|---|---|---|---|---|
| 1 | swipe_left | 10 | draw_circle_CCW | 19 | knock |
| 2 | swipe_right | 11 | draw_triangle | 20 | catch |
| 3 | wave | 12 | bowling | 21 | pickup_throw |
| 4 | clap | 13 | boxing | 22 | jog |
| 5 | throw | 14 | baseball_swing | 23 | walk |
| 6 | arm_cross | 15 | tennis_swing | 24 | sit2stand |
| 7 | basketball_shoot | 16 | arm_curl | 25 | stand2sit |
| 8 | draw_x | 17 | tennis_serve | 26 | lunge |
| 9 | draw_circle_CW | 18 | push | 27 | squat |
[1] A. Gu and T. Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," ICLR, 2024.
[2] C. Chen, R. Jafari, and N. Kehtarnavaz, "UTD-MHAD: A Multimodal Dataset for Human Action Recognition Utilizing a Depth Camera and a Wearable Inertial Sensor," ICIP, 2015.
[3] N. Neverova, C. Wolf, G. Taylor, and F. Nebout, "ModDrop: Adaptive Multi-modal Gesture Recognition," IEEE TPAMI, 2016.
[4] B. T. Polyak, "Some methods of speeding up the convergence of iteration methods," USSR Computational Mathematics and Mathematical Physics, 1964.
[5] A. Gu, K. Goel, and C. Re, "Efficiently Modeling Long Sequences with Structured State Spaces," ICLR, 2022.
[6] W. Wang, D. Tran, and M. Feiszli, "What Makes Training Multi-modal Classification Networks Hard?," CVPR, 2020.
[7] G. Hinton, O. Vinyals, and J. Dean, "Distilling the Knowledge in a Neural Network," NeurIPS Workshop, 2015.
This project is licensed under the MIT License.