Add project report and model card

README.md

@@ -80,6 +80,8 @@ Latest local run:
 
 Full output is in `bench/results/latest.md` and `bench/results/latest.json`.
 
+The full project report is in `docs/report.md`.
+
 ## Training
 
 The worker can load a small trained text model artifact from `worker/app/artifacts/tiny_text_model.json`. The current model trains on generated service-status examples, so the point is the local training and serving path, not benchmark-grade NLP accuracy.
@@ -108,6 +110,8 @@ Latest training run on an M3 Pro:
 
 The full run output is in `training/runs/latest.md` and `training/runs/latest.json`.
 
+Model evaluation and limitations are documented in `docs/model_card.md`.
+
 ## What is in here now
 
 - FastAPI gateway with `/predict`, `/health`, `/workers`, and `/metrics`

docs/model_card.md

@@ -0,0 +1,87 @@
+# Model card
+
+## Model
+
+`tiny-text-mps-v1` is a small linear text classifier used by ModelMesh workers. It classifies short service-status text as either `healthy` or `risky`.
+
+The model is intentionally small. Its purpose is to make the training and serving path testable inside an inference mesh, not to compete with production NLP models.
+
+## Training
+
+Training script:
+
+```bash
+python training/train_text_model.py
+```
+
+Latest checked-in run:
+
+| Metric | Value |
+| --- | ---: |
+| Device | mps |
+| Epochs | 320 |
+| Train examples | 3,276 |
+| Test examples | 820 |
+| Vocab size | 37 |
+| Parameters | 38 |
+| Train time | 0.4351 s |
+| Examples/sec | 2,409,512.37 |
+| Train accuracy | 1.0 |
+| Test accuracy | 1.0 |
+
+Full run output: `training/runs/latest.md`.
+
+## Evaluation
+
+Challenge evaluation script:
+
+```bash
+python training/evaluate_text_model.py
+```
+
+The challenge set is a small handwritten set of service-status examples with mixed positive and negative signals.
+
+| Metric | Value |
+| --- | ---: |
+| Examples | 16 |
+| Correct | 15 |
+| Accuracy | 0.9375 |
+
+Confusion matrix:
+
+| Expected | Predicted healthy | Predicted risky |
+| --- | ---: | ---: |
+| healthy | 6 | 1 |
+| risky | 0 | 9 |
+
+The missed example was:
+
+```text
+worker recovered after timeout and is stable
+```
+
+The model predicted `risky`, mostly because `timeout` has a strong negative learned weight and the bag-of-words model does not understand recovery order.
+
+Full evaluation output: `training/evaluation/latest.md`.
+
+## Intended use
+
+Use this model to exercise ModelMesh serving behavior:
+
+- model artifact loading
+- worker predictions
+- cache behavior
+- routing behavior
+- failure replay
+- benchmark runs
+
+## Limitations
+
+- Training data is generated service-status text, not a real production incident dataset.
+- The model is bag-of-words and does not understand word order.
+- Accuracy numbers are useful for regression checks, not for claiming broad NLP performance.
+- The model is deliberately small so the project can focus on ML systems behavior.
+
+## Next model improvement
+
+The next useful upgrade is a compact sequence model or character n-gram model trained on a larger labeled service-log dataset, still with MPS timing and artifact export.

docs/portfolio.md

@@ -0,0 +1,29 @@
+# Portfolio copy
+
+## Pinned repo title
+
+ModelMesh
+
+## GitHub repo description
+
+Trainable ML inference mesh with MPS training, load-aware routing, caching, failure replay, benchmarks, Docker, and a live dashboard.
+
+## LinkedIn Featured description
+
+Built a compact ML inference mesh with Apple Silicon training, model workers, cache-aware gateway routing, failure replay, Docker, benchmarks, and a live dashboard for p95 latency and worker health.
+
+## Resume bullets
+
+- Built a FastAPI-based ML inference mesh with a gateway, model workers, cache-aware prediction serving, retry handling, and live worker health metrics.
+- Trained and exported a small PyTorch text classifier on Apple Silicon MPS, recording training time, throughput, evaluation metrics, and a model card.
+- Added benchmark and replay tooling to compare cache behavior, round-robin routing, load-aware routing, slow-worker handling, and failure recovery.
+
+## LinkedIn post draft
+
+I built ModelMesh, a small ML inference system for testing what happens around a model after training.
+
+It has a FastAPI gateway, multiple model workers, caching, load-aware routing, worker failure controls, Docker setup, benchmark scripts, and a live dashboard. The model is intentionally small, but it trains locally on Apple Silicon MPS and exports as a JSON artifact the workers can load without PyTorch at inference time.
+
+The most useful part was measuring system behavior: cache on vs off, round-robin vs load-aware routing, slow workers, flaky workers, and recovery. It is not a production ML platform, but it is a solid lab for the pieces that usually sit around an ML model in real systems.
+
+Repo: ModelMesh

docs/report.md

@@ -0,0 +1,135 @@
+# ModelMesh report
+
+## Summary
+
+ModelMesh is a compact ML inference mesh. It trains a small model locally on Apple Silicon, serves it through multiple workers, routes requests through a gateway, caches repeated predictions, records metrics, and demonstrates recovery from slow or failing workers.
+
+The project is built to show ML systems behavior, not just model training.
+
+## What it does
+
+A client sends text to:
+
+```text
+POST /predict
+```
+
+The gateway checks cache first. On a cache miss, it selects a worker, sends the request, retries if needed, records latency and worker state, and returns the model prediction.
+
+The dashboard shows:
+
+- request count
+- cache hit rate
+- p50 and p95 latency
+- worker failures
+- in-flight requests
+- delay and fail-rate controls
+
+## Architecture
+
+```mermaid
+flowchart LR
+    client["Client"] --> gateway["FastAPI gateway"]
+    gateway --> cache["Redis or memory cache"]
+    gateway --> worker_a["Worker A"]
+    gateway --> worker_b["Worker B"]
+    worker_a --> model_a["tiny-text-mps-v1"]
+    worker_b --> model_b["tiny-text-mps-v1"]
+    gateway --> dashboard["Dashboard and metrics"]
+```
+
+The gateway supports two routing strategies:
+
+- `round_robin`: simple baseline.
+- `load_aware`: ranks workers by failures, in-flight requests, and recent latency.
+
+## Training results
+
+Latest MPS training run on an M3 Pro:
+
+| Metric | Value |
+| --- | ---: |
+| Device | mps |
+| Epochs | 320 |
+| Train examples | 3,276 |
+| Test examples | 820 |
+| Parameters | 38 |
+| Train time | 0.4351 s |
+| Examples/sec | 2,409,512.37 |
+| Train accuracy | 1.0 |
+| Test accuracy | 1.0 |
+
+The training script refuses CPU by default, so it will not silently fall back to a slow CPU run.
+
+## Model evaluation
+
+The model was also evaluated on a handwritten challenge set with mixed healthy and risky signals.
+
+| Metric | Value |
+| --- | ---: |
+| Examples | 16 |
+| Correct | 15 |
+| Accuracy | 0.9375 |
+
+Confusion matrix:
+
+| Expected | Predicted healthy | Predicted risky |
+| --- | ---: | ---: |
+| healthy | 6 | 1 |
+| risky | 0 | 9 |
+
+The one miss was a recovery sentence containing `timeout`. That is a reasonable failure for a bag-of-words model and is listed in `docs/model_card.md`.
+
+## Benchmark results
+
+Latest local scenario run:
+
+| Scenario | RPS | p50 ms | p95 ms | Cache hits | Failures |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| cache enabled | 728.95 | 5.01 | 52.04 | 72 | 0 |
+| cache disabled | 185.7 | 41.1 | 53.09 | 0 | 0 |
+| slow worker, round-robin | 187.36 | 57.79 | 111.91 | 0 | 0 |
+| slow worker, load-aware | 688.48 | 5.21 | 111.9 | 0 | 0 |
+| dead worker recovery | 707.98 | 11.37 | 28.93 | 0 | 0 |
+
+The benchmark shows two important behaviors:
+
+- Cache hits improve throughput on repeated inputs.
+- Load-aware routing avoids the slow worker more aggressively than round-robin.
+
+Full output is in `bench/results/latest.md` and `bench/results/latest.json`.
+
+## Failure replay
+
+`bench/replay_traffic.py` drives the system through four stages:
+
+1. baseline
+2. slow worker
+3. flaky worker
+4. recovery
+
+The replay uses the same worker-control path exposed by the dashboard. The latest replay completed 576 requests with 576 successful responses while showing one worker failure event and routing around it.
+
+Full output is in `bench/results/replay-latest.json`.
+
+## Deployment
+
+The repo includes:
+
+- local Docker Compose
+- production-style Docker Compose
+- Docker image builds in CI
+
+A separate Render Blueprint PR prepares the hosted demo. It still needs a provider account action and cost review before there is a public URL.
+
+## Limitations
+
+- The model is small and trained on generated service-status text.
+- The dashboard is a local/operator view, not an authenticated production control plane.
+- Benchmarks are local-machine measurements, not cloud load tests.
+- Redis is used as a cache, not as a durable feature store.
+- The project is best described as a compact ML inference mesh, not a production ML platform.
+
+## Next useful improvement
+
+The next model upgrade should use a larger labeled service-log dataset or a compact sequence model while keeping MPS timing, evaluation output, and artifact export.

tests/test_text_model_evaluation.py

@@ -0,0 +1,10 @@
+from training.evaluate_text_model import evaluate
+
+
+def test_challenge_evaluation_has_expected_shape() -> None:
+    result = evaluate()
+
+    assert result["examples"] == 16
+    assert result["correct"] >= 12
+    assert set(result["confusion_matrix"]) == {"healthy", "risky"}
+    assert set(result["per_label"]) == {"healthy", "risky"}