Real-time data acquisition, fusion, inference and visualization system for 6 wireless IMU nodes (ESP32-C3) + depth camera, paired with the MMA training repository via ONNX/PyTorch artifacts.
MMA-Deploy is the production deployment component of the MMA (Momentum Mamba) multimodal Human Activity Recognition system. It provides:
- Hardware: 6 custom ESP32-C3 IMU sensor nodes (ICM-42688-P, 200 Hz) with KiCAD PCB design
- Server: Async FastAPI server with ONNX Runtime / PyTorch real-time inference (10 Hz)
- Frontend: Browser dashboard with 3D IMU visualization (Three.js), prediction charts (Chart.js), and depth camera feed
- Recording: Training-compatible data capture (
.npy+.png+meta.yaml) for seamless retraining loop
┌─────────────────────┐ ┌──────────────────┐
│ ESP32-C3 x6 │ │ Depth Camera │
│ (200Hz IMU, UDP) │ │ (30fps, USB) │
└────────┬────────────┘ └────────┬──────────┘
│ UDP │ USB
▼ ▼
┌──────────────────────────────────────────────┐
│ FastAPI Server │
│ │
│ ┌──────────┐ ┌──────────┐ ┌───────────┐ │
│ │ IMU UDP │ │ Depth │ │ Time │ │
│ │ Listener │ │ Camera │ │ Sync │ │
│ └────┬─────┘ └────┬─────┘ └───────────┘ │
│ │ │ │
│ ┌────▼─────┐ │ │
│ │ Ring │ │ │
│ │ Buffer │ │ │
│ └────┬─────┘ │ │
│ │ │ │
│ ┌────▼──────────────▼───┐ │
│ │ Aligner │ │
│ │ (temporal sync) │ │
│ └───────────┬───────────┘ │
│ │ │
│ ┌───────────▼───────────┐ │
│ │ Preprocessing │ │
│ │ (z-score, resample) │ │
│ └───────────┬───────────┘ │
│ │ │
│ ┌───────────▼───────────┐ ┌──────────────┐ │
│ │ ONNX / PyTorch │ │ Recorder │ │
│ │ Inference (10Hz) │ │ (.npy/.png) │ │
│ └───────────┬───────────┘ └──────────────┘ │
│ │ │
│ ┌───────────▼───────────┐ │
│ │ WebSocket Broadcast │ │
│ └───────────┬───────────┘ │
└──────────────┼───────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Browser Dashboard │
│ ┌────────┐ ┌──────────────┐ │
│ │Chart.js│ │ Three.js │ │
│ │ top-5 │ │ 3D orient. │ │
│ └────────┘ └──────────────┘ │
│ ┌────────┐ ┌──────────────┐ │
│ │ Depth │ │ Skeleton │ │
│ │ frame │ │ overlay │ │
│ └────────┘ └──────────────┘ │
└──────────────────────────────┘
MMA-Deploy/
├── server/ # FastAPI server
│ ├── main.py # Entry point, app lifespan
│ ├── config.py # Centralized settings (Pydantic)
│ ├── schema.py # Request/response models
│ │
│ ├── io/ # Hardware input threads
│ │ ├── imu_udp.py # UDP listener for 6 ESP32 nodes (200 Hz)
│ │ ├── depth_camera.py # Depth camera thread (RealSense / mock)
│ │ └── time_sync.py # Clock offset sync daemon
│ │
│ ├── fusion/ # Multimodal alignment
│ │ ├── aligner.py # Temporal synchronizer (resampling)
│ │ └── ring_buffer.py # Per-node circular buffer
│ │
│ ├── inference/ # Model inference
│ │ ├── model.py # ONNXModel (onnxruntime)
│ │ ├── torch_model.py # TorchStreamingModel (TorchScript)
│ │ ├── tri_modal.py # TriModalTorchModel (RGB/Depth/IMU)
│ │ ├── skel_imu.py # SkelIMUTorchModel (skeleton + IMU)
│ │ ├── pipeline.py # InferenceThread (producer loop)
│ │ ├── meta_loader.py # JSON metadata parser
│ │ ├── preprocess.py # Normalization, resampling
│ │ ├── skel_extractor.py # MediaPipe skeleton extraction
│ │ └── person_segmenter.py # Person segmentation (MediaPipe + depth)
│ │
│ ├── recorder/ # Data recording
│ │ └── writer.py # Saves IMU.npy, depth.png, meta.yaml
│ │
│ └── ws/ # WebSocket
│ └── broadcaster.py # Real-time push to web clients
│
├── web/ # Browser dashboard
│ ├── index.html
│ ├── css/style.css
│ └── js/
│ ├── ws_client.js # WebSocket consumer
│ ├── inference_panel.js # Top-5 predictions (Chart.js)
│ ├── imu_panel.js # Per-node 3D orientation (Three.js)
│ ├── depth.js # Depth frame visualization
│ ├── skel_overlay.js # Skeleton overlay on depth
│ ├── status.js # System stats (FPS, latency)
│ └── orient3d.js # Quaternion 3D visualization
│
├── firmware/ # ESP32-C3 firmware
│ ├── src/main.c # ICM-42688-P reader, UDP broadcast
│ ├── platformio.ini # Board config (esp32-c3-devkitm-1)
│ ├── CMakeLists.txt
│ ├── sdkconfig.defaults
│ ├── secrets.h.example # WiFi credential template
│ ├── components/ # ESP-IDF components (Fusion filter)
│ └── pc_host/ # Host-side utilities
│ ├── imu_receiver.py # UDP → console/CSV for debugging
│ └── requirements.txt
│
├── models/ # ONNX/PyTorch artifacts (gitignored)
│ └── README.md # Meta JSON schema documentation
│
├── scripts/ # Standalone utilities
│ ├── export_onnx.py # PyTorch checkpoint → ONNX
│ ├── calibrate_imu.py # IMU zero-offset calibration
│ ├── compute_imu_stats.py # Compute train-set z-score stats
│ ├── test_imu_latency.py # Measure E2E latency
│ └── replay_recording.py # Replay saved session (no hardware)
│
├── data/ # Live recordings (gitignored)
│ └── recordings/
│
├── tests/ # Unit tests
│ ├── test_aligner.py
│ └── test_ring_buffer.py
│
├── .env.example # Environment variable template
├── environment.yml # Conda environment
├── pyproject.toml # Python package config
├── Makefile # Build & deployment automation
├── DEPLOY_GUIDE.md # Checkpoint export & model conversion
├── DEPLOY_v4_circlefix.md # Specific model deployment guide
└── LICENSE # MIT License
- Python 3.10+
- Conda (recommended) or pip
- PlatformIO (for firmware flashing)
- 6x ESP32-C3 IMU nodes + depth camera (or use mock/replay mode)
# 1. Install Python dependencies
make install # or: pip install -e .[dev]
# 2. Configure environment
cp .env.example .env # Edit SERVER_IP, model path, etc.# Copy secrets template and fill in WiFi credentials
cp firmware/secrets.h.example firmware/src/secrets.h
# Edit firmware/src/secrets.h with your WiFi SSID and password
# Flash one node
make flash NODE=1
make monitor NODE=1make server # uvicorn on :8000
# Open http://localhost:8000 for the dashboardUse the replay script to simulate a live session from recorded data:
make replay FILE=data/recordings/sampleThis repo never imports training code. The bridge is purely artifact-based:
| From MMA | To MMA-Deploy |
|---|---|
checkpoints/model.onnx |
models/ |
checkpoints/model.meta.json |
models/ |
checkpoints/model.pt (PyTorch) |
models/ |
# In the MMA training repo
python scripts/export_onnx.py \
--checkpoint checkpoints/rgbd_imu_CMAR1.pt \
--output models/model_v1.0.0Each model artifact includes a .meta.json file specifying:
- Input/output tensor shapes and names
- Normalization statistics (mean, std)
- Class names (action labels)
- Model version and training metadata
See models/README.md for the full schema.
Recordings written by the server are directly loadable by the training-side dataset classes:
data/recordings/session_YYYYMMDD_HHMMSS/
├── meta.yaml # Session metadata (nodes, FPS, sample rate)
├── depth/ # uint16 PNG depth maps (mm)
│ ├── 000000.png
│ └── ...
└── imu/ # Per-node NumPy arrays (N, 7) [timestamp + 6-axis]
├── node_1.npy
└── ...
This enables a seamless loop: deploy → record → retrain → redeploy.
| Spec | Value |
|---|---|
| MCU | ESP32-C3 (RISC-V, WiFi) |
| IMU | ICM-42688-P (6-axis, 200 Hz) |
| Protocol | UDP broadcast |
| Nodes | 6 per system |
| PCB | Custom KiCAD design |
- Intel RealSense D455 (recommended) or any USB depth camera
- Mock mode available for development without hardware
Wrist-mounted sensor attachment designed in SolidWorks, available as .3mf / .STEP files.
This project is licensed under the MIT License.
- MMA — Training framework: Momentum Mamba models, baselines, ablation studies