diff --git a/README.md b/README.md index c5f2513..8e7a118 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,8 @@ Latest local run: Full output is in `bench/results/latest.md` and `bench/results/latest.json`. +The full project report is in `docs/report.md`. + ## Training The worker can load a small trained text model artifact from `worker/app/artifacts/tiny_text_model.json`. The current model trains on generated service-status examples, so the point is the local training and serving path, not benchmark-grade NLP accuracy. @@ -108,6 +110,8 @@ Latest training run on an M3 Pro: The full run output is in `training/runs/latest.md` and `training/runs/latest.json`. +Model evaluation and limitations are documented in `docs/model_card.md`. + ## What is in here now - FastAPI gateway with `/predict`, `/health`, `/workers`, and `/metrics` diff --git a/docs/model_card.md b/docs/model_card.md new file mode 100644 index 0000000..c79fe37 --- /dev/null +++ b/docs/model_card.md @@ -0,0 +1,87 @@ +# Model card + +## Model + +`tiny-text-mps-v1` is a small linear text classifier used by ModelMesh workers. It classifies short service-status text as either `healthy` or `risky`. + +The model is intentionally small. Its purpose is to make the training and serving path testable inside an inference mesh, not to compete with production NLP models. + +## Training + +Training script: + +```bash +python training/train_text_model.py +``` + +Latest checked-in run: + +| Metric | Value | +| --- | ---: | +| Device | mps | +| Epochs | 320 | +| Train examples | 3,276 | +| Test examples | 820 | +| Vocab size | 37 | +| Parameters | 38 | +| Train time | 0.4351 s | +| Examples/sec | 2,409,512.37 | +| Train accuracy | 1.0 | +| Test accuracy | 1.0 | + +Full run output: `training/runs/latest.md`. + +## Evaluation + +Challenge evaluation script: + +```bash +python training/evaluate_text_model.py +``` + +The challenge set is a small handwritten set of service-status examples with mixed positive and negative signals. + +| Metric | Value | +| --- | ---: | +| Examples | 16 | +| Correct | 15 | +| Accuracy | 0.9375 | + +Confusion matrix: + +| Expected | Predicted healthy | Predicted risky | +| --- | ---: | ---: | +| healthy | 6 | 1 | +| risky | 0 | 9 | + +The missed example was: + +```text +worker recovered after timeout and is stable +``` + +The model predicted `risky`, mostly because `timeout` has a strong negative learned weight and the bag-of-words model does not understand recovery order. + +Full evaluation output: `training/evaluation/latest.md`. + +## Intended use + +Use this model to exercise ModelMesh serving behavior: + +- model artifact loading +- worker predictions +- cache behavior +- routing behavior +- failure replay +- benchmark runs + +## Limitations + +- Training data is generated service-status text, not a real production incident dataset. +- The model is bag-of-words and does not understand word order. +- Accuracy numbers are useful for regression checks, not for claiming broad NLP performance. +- The model is deliberately small so the project can focus on ML systems behavior. + +## Next model improvement + +The next useful upgrade is a compact sequence model or character n-gram model trained on a larger labeled service-log dataset, still with MPS timing and artifact export. diff --git a/docs/portfolio.md b/docs/portfolio.md new file mode 100644 index 0000000..497b6fe --- /dev/null +++ b/docs/portfolio.md @@ -0,0 +1,29 @@ +# Portfolio copy + +## Pinned repo title + +ModelMesh + +## GitHub repo description + +Trainable ML inference mesh with MPS training, load-aware routing, caching, failure replay, benchmarks, Docker, and a live dashboard. + +## LinkedIn Featured description + +Built a compact ML inference mesh with Apple Silicon training, model workers, cache-aware gateway routing, failure replay, Docker, benchmarks, and a live dashboard for p95 latency and worker health. + +## Resume bullets + +- Built a FastAPI-based ML inference mesh with a gateway, model workers, cache-aware prediction serving, retry handling, and live worker health metrics. +- Trained and exported a small PyTorch text classifier on Apple Silicon MPS, recording training time, throughput, evaluation metrics, and a model card. +- Added benchmark and replay tooling to compare cache behavior, round-robin routing, load-aware routing, slow-worker handling, and failure recovery. + +## LinkedIn post draft + +I built ModelMesh, a small ML inference system for testing what happens around a model after training. + +It has a FastAPI gateway, multiple model workers, caching, load-aware routing, worker failure controls, Docker setup, benchmark scripts, and a live dashboard. The model is intentionally small, but it trains locally on Apple Silicon MPS and exports as a JSON artifact the workers can load without PyTorch at inference time. + +The most useful part was measuring system behavior: cache on vs off, round-robin vs load-aware routing, slow workers, flaky workers, and recovery. It is not a production ML platform, but it is a solid lab for the pieces that usually sit around an ML model in real systems. + +Repo: ModelMesh diff --git a/docs/report.md b/docs/report.md new file mode 100644 index 0000000..c4dbcc5 --- /dev/null +++ b/docs/report.md @@ -0,0 +1,135 @@ +# ModelMesh report + +## Summary + +ModelMesh is a compact ML inference mesh. It trains a small model locally on Apple Silicon, serves it through multiple workers, routes requests through a gateway, caches repeated predictions, records metrics, and demonstrates recovery from slow or failing workers. + +The project is built to show ML systems behavior, not just model training. + +## What it does + +A client sends text to: + +```text +POST /predict +``` + +The gateway checks cache first. On a cache miss, it selects a worker, sends the request, retries if needed, records latency and worker state, and returns the model prediction. + +The dashboard shows: + +- request count +- cache hit rate +- p50 and p95 latency +- worker failures +- in-flight requests +- delay and fail-rate controls + +## Architecture + +```mermaid +flowchart LR + client["Client"] --> gateway["FastAPI gateway"] + gateway --> cache["Redis or memory cache"] + gateway --> worker_a["Worker A"] + gateway --> worker_b["Worker B"] + worker_a --> model_a["tiny-text-mps-v1"] + worker_b --> model_b["tiny-text-mps-v1"] + gateway --> dashboard["Dashboard and metrics"] +``` + +The gateway supports two routing strategies: + +- `round_robin`: simple baseline. +- `load_aware`: ranks workers by failures, in-flight requests, and recent latency. + +## Training results + +Latest MPS training run on an M3 Pro: + +| Metric | Value | +| --- | ---: | +| Device | mps | +| Epochs | 320 | +| Train examples | 3,276 | +| Test examples | 820 | +| Parameters | 38 | +| Train time | 0.4351 s | +| Examples/sec | 2,409,512.37 | +| Train accuracy | 1.0 | +| Test accuracy | 1.0 | + +The training script refuses CPU by default, so it will not silently fall back to a slow CPU run. + +## Model evaluation + +The model was also evaluated on a handwritten challenge set with mixed healthy and risky signals. + +| Metric | Value | +| --- | ---: | +| Examples | 16 | +| Correct | 15 | +| Accuracy | 0.9375 | + +Confusion matrix: + +| Expected | Predicted healthy | Predicted risky | +| --- | ---: | ---: | +| healthy | 6 | 1 | +| risky | 0 | 9 | + +The one miss was a recovery sentence containing `timeout`. That is a reasonable failure for a bag-of-words model and is listed in `docs/model_card.md`. + +## Benchmark results + +Latest local scenario run: + +| Scenario | RPS | p50 ms | p95 ms | Cache hits | Failures | +| --- | ---: | ---: | ---: | ---: | ---: | +| cache enabled | 728.95 | 5.01 | 52.04 | 72 | 0 | +| cache disabled | 185.7 | 41.1 | 53.09 | 0 | 0 | +| slow worker, round-robin | 187.36 | 57.79 | 111.91 | 0 | 0 | +| slow worker, load-aware | 688.48 | 5.21 | 111.9 | 0 | 0 | +| dead worker recovery | 707.98 | 11.37 | 28.93 | 0 | 0 | + +The benchmark shows two important behaviors: + +- Cache hits improve throughput on repeated inputs. +- Load-aware routing avoids the slow worker more aggressively than round-robin. + +Full output is in `bench/results/latest.md` and `bench/results/latest.json`. + +## Failure replay + +`bench/replay_traffic.py` drives the system through four stages: + +1. baseline +2. slow worker +3. flaky worker +4. recovery + +The replay uses the same worker-control path exposed by the dashboard. The latest replay completed 576 requests with 576 successful responses while showing one worker failure event and routing around it. + +Full output is in `bench/results/replay-latest.json`. + +## Deployment + +The repo includes: + +- local Docker Compose +- production-style Docker Compose +- Docker image builds in CI + +A separate Render Blueprint PR prepares the hosted demo. It still needs a provider account action and cost review before there is a public URL. + +## Limitations + +- The model is small and trained on generated service-status text. +- The dashboard is a local/operator view, not an authenticated production control plane. +- Benchmarks are local-machine measurements, not cloud load tests. +- Redis is used as a cache, not as a durable feature store. +- The project is best described as a compact ML inference mesh, not a production ML platform. + +## Next useful improvement + +The next model upgrade should use a larger labeled service-log dataset or a compact sequence model while keeping MPS timing, evaluation output, and artifact export. diff --git a/tests/test_text_model_evaluation.py b/tests/test_text_model_evaluation.py new file mode 100644 index 0000000..404f697 --- /dev/null +++ b/tests/test_text_model_evaluation.py @@ -0,0 +1,10 @@ +from training.evaluate_text_model import evaluate + + +def test_challenge_evaluation_has_expected_shape() -> None: + result = evaluate() + + assert result["examples"] == 16 + assert result["correct"] >= 12 + assert set(result["confusion_matrix"]) == {"healthy", "risky"} + assert set(result["per_label"]) == {"healthy", "risky"} diff --git a/training/evaluate_text_model.py b/training/evaluate_text_model.py new file mode 100644 index 0000000..af1cfb6 --- /dev/null +++ b/training/evaluate_text_model.py @@ -0,0 +1,163 @@ +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +OUTPUT_JSON = ROOT / "training" / "evaluation" / "latest.json" +OUTPUT_MD = ROOT / "training" / "evaluation" / "latest.md" + +CHALLENGE_SET = [ + ("healthy", "fast stable reliable service"), + ("risky", "slow broken timeout error"), + ("healthy", "worker recovered after timeout and is stable"), + ("risky", "cache hit but worker failed with timeout"), + ("risky", "great latency but retry caused timeout"), + ("healthy", "responsive route completed request"), + ("risky", "unhealthy worker returned fast response"), + ("healthy", "stable gateway handled burst traffic"), + ("risky", "prediction failed after worker crash"), + ("risky", "service is healthy but slow"), + ("healthy", "reliable cache and responsive worker"), + ("risky", "broken route recovered with retry"), + ("healthy", "cache recovered and request completed"), + ("risky", "gateway retry loop after timeout"), + ("healthy", "responsive worker handled traffic burst"), + ("risky", "mesh returned broken prediction error"), +] + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--model-artifact", type=Path, default=None) + parser.add_argument("--output-json", type=Path, default=OUTPUT_JSON) + parser.add_argument("--output-md", type=Path, default=OUTPUT_MD) + return parser.parse_args() + + +def evaluate(model_artifact: Path | None = None) -> dict: + from worker.app.model import load_model + + model = load_model(str(model_artifact) if model_artifact else None) + rows = [] + labels = ["healthy", "risky"] + confusion = { + "healthy": {"healthy": 0, "risky": 0}, + "risky": {"healthy": 0, "risky": 0}, + } + + for expected, text in CHALLENGE_SET: + prediction = model.predict(text) + predicted = str(prediction["label"]) + confusion[expected][predicted] += 1 + rows.append({ + "text": text, + "expected": expected, + "predicted": predicted, + "score": prediction["score"], + "correct": expected == predicted, + }) + + total = len(rows) + correct = sum(1 for row in rows if row["correct"]) + per_label = {} + for label in labels: + true_positive = confusion[label][label] + false_positive = sum( + confusion[other][label] for other in labels if other != label + ) + false_negative = sum( + confusion[label][other] for other in labels if other != label + ) + precision_base = true_positive + false_positive + recall_base = true_positive + false_negative + precision = true_positive / precision_base if precision_base else 0 + recall = true_positive / recall_base if recall_base else 0 + per_label[label] = { + "precision": round(precision, 4), + "recall": round(recall, 4), + } + + return { + "model_version": model.version, + "dataset": "handwritten service-status challenge set", + "examples": total, + "accuracy": round(correct / total, 4), + "correct": correct, + "confusion_matrix": confusion, + "per_label": per_label, + "rows": rows, + } + + +def write_outputs(result: dict, json_path: Path, md_path: Path) -> None: + json_path.parent.mkdir(parents=True, exist_ok=True) + json_path.write_text(json.dumps(result, indent=2) + "\n") + + lines = [ + "# Model evaluation", + "", + f"Model version: `{result['model_version']}`", + "", + "| Metric | Value |", + "| --- | ---: |", + f"| Examples | {result['examples']} |", + f"| Correct | {result['correct']} |", + f"| Accuracy | {result['accuracy']} |", + "", + "## Confusion matrix", + "", + "| Expected | Predicted healthy | Predicted risky |", + "| --- | ---: | ---: |", + ] + + for expected, predictions in result["confusion_matrix"].items(): + lines.append(f"| {expected} | {predictions['healthy']} | {predictions['risky']} |") + + lines.extend([ + "", + "## Label metrics", + "", + "| Label | Precision | Recall |", + "| --- | ---: | ---: |", + ]) + for label, metrics in result["per_label"].items(): + lines.append(f"| {label} | {metrics['precision']} | {metrics['recall']} |") + + lines.extend([ + "", + "## Misses", + "", + ]) + misses = [row for row in result["rows"] if not row["correct"]] + if not misses: + lines.append("No misses on this challenge set.") + else: + lines.extend([ + "| Text | Expected | Predicted | Score |", + "| --- | --- | --- | ---: |", + ]) + for row in misses: + lines.append( + f"| {row['text']} | {row['expected']} | " + f"{row['predicted']} | {row['score']} |" + ) + + lines.append("") + md_path.write_text("\n".join(lines)) + + +def main() -> None: + args = parse_args() + result = evaluate(args.model_artifact) + write_outputs(result, args.output_json, args.output_md) + print(json.dumps(result, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/training/evaluation/latest.json b/training/evaluation/latest.json new file mode 100644 index 0000000..b22997c --- /dev/null +++ b/training/evaluation/latest.json @@ -0,0 +1,141 @@ +{ + "model_version": "tiny-text-mps-v1", + "dataset": "handwritten service-status challenge set", + "examples": 16, + "accuracy": 0.9375, + "correct": 15, + "confusion_matrix": { + "healthy": { + "healthy": 6, + "risky": 1 + }, + "risky": { + "healthy": 0, + "risky": 9 + } + }, + "per_label": { + "healthy": { + "precision": 1.0, + "recall": 0.8571 + }, + "risky": { + "precision": 0.9, + "recall": 1.0 + } + }, + "rows": [ + { + "text": "fast stable reliable service", + "expected": "healthy", + "predicted": "healthy", + "score": 0.9993, + "correct": true + }, + { + "text": "slow broken timeout error", + "expected": "risky", + "predicted": "risky", + "score": 1.0, + "correct": true + }, + { + "text": "worker recovered after timeout and is stable", + "expected": "healthy", + "predicted": "risky", + "score": 0.6689, + "correct": false + }, + { + "text": "cache hit but worker failed with timeout", + "expected": "risky", + "predicted": "risky", + "score": 0.9887, + "correct": true + }, + { + "text": "great latency but retry caused timeout", + "expected": "risky", + "predicted": "risky", + "score": 0.9572, + "correct": true + }, + { + "text": "responsive route completed request", + "expected": "healthy", + "predicted": "healthy", + "score": 0.9998, + "correct": true + }, + { + "text": "unhealthy worker returned fast response", + "expected": "risky", + "predicted": "risky", + "score": 0.9949, + "correct": true + }, + { + "text": "stable gateway handled burst traffic", + "expected": "healthy", + "predicted": "healthy", + "score": 0.6106, + "correct": true + }, + { + "text": "prediction failed after worker crash", + "expected": "risky", + "predicted": "risky", + "score": 0.994, + "correct": true + }, + { + "text": "service is healthy but slow", + "expected": "risky", + "predicted": "risky", + "score": 0.6146, + "correct": true + }, + { + "text": "reliable cache and responsive worker", + "expected": "healthy", + "predicted": "healthy", + "score": 0.9994, + "correct": true + }, + { + "text": "broken route recovered with retry", + "expected": "risky", + "predicted": "risky", + "score": 0.9964, + "correct": true + }, + { + "text": "cache recovered and request completed", + "expected": "healthy", + "predicted": "healthy", + "score": 0.993, + "correct": true + }, + { + "text": "gateway retry loop after timeout", + "expected": "risky", + "predicted": "risky", + "score": 0.9937, + "correct": true + }, + { + "text": "responsive worker handled traffic burst", + "expected": "healthy", + "predicted": "healthy", + "score": 0.9185, + "correct": true + }, + { + "text": "mesh returned broken prediction error", + "expected": "risky", + "predicted": "risky", + "score": 0.9992, + "correct": true + } + ] +} diff --git a/training/evaluation/latest.md b/training/evaluation/latest.md new file mode 100644 index 0000000..ffc26e3 --- /dev/null +++ b/training/evaluation/latest.md @@ -0,0 +1,29 @@ +# Model evaluation + +Model version: `tiny-text-mps-v1` + +| Metric | Value | +| --- | ---: | +| Examples | 16 | +| Correct | 15 | +| Accuracy | 0.9375 | + +## Confusion matrix + +| Expected | Predicted healthy | Predicted risky | +| --- | ---: | ---: | +| healthy | 6 | 1 | +| risky | 0 | 9 | + +## Label metrics + +| Label | Precision | Recall | +| --- | ---: | ---: | +| healthy | 1.0 | 0.8571 | +| risky | 0.9 | 1.0 | + +## Misses + +| Text | Expected | Predicted | Score | +| --- | --- | --- | ---: | +| worker recovered after timeout and is stable | healthy | risky | 0.6689 |