Upgrade text model training and evaluation

README.md

@@ -93,7 +93,7 @@ The full project report is in `docs/report.md`.
 
 ## Training
 
-The worker can load a small trained text model artifact from `worker/app/artifacts/tiny_text_model.json`. The current model trains on generated service-status examples, so the point is the local training and serving path, not benchmark-grade NLP accuracy.
+The worker can load a small trained text model artifact from `worker/app/artifacts/tiny_text_model.json`. The current model is a compact MLP over word and character n-gram features. It trains on generated service-status examples, so the point is the local training, export, evaluation, and serving path, not benchmark-grade NLP accuracy.
 
 On Apple Silicon, the training script uses PyTorch MPS. It refuses to run on CPU by default so a slow accidental CPU run fails fast.
 
@@ -108,12 +108,16 @@ Latest training run on an M3 Pro:
 
 | Metric | Value |
 | --- | ---: |
+| Model type | mlp_text_classifier |
 | Device | mps |
-| Epochs | 320 |
+| Epochs | 260 |
 | Train examples | 3,276 |
 | Test examples | 820 |
-| Train time | 0.4351 s |
-| Examples/sec | 2,409,512.37 |
+| Features | 140 |
+| Hidden units | 16 |
+| Parameters | 2,273 |
+| Train time | 0.3991 s |
+| Examples/sec | 2,134,408.08 |
 | Train accuracy | 1.0 |
 | Test accuracy | 1.0 |
 
@@ -124,7 +128,7 @@ Model evaluation and limitations are documented in `docs/model_card.md`.
 ## What is in here now
 
 - FastAPI gateway with `/predict`, `/health`, `/workers`, and `/metrics`
-- Worker service with a tiny text classifier and optional trained artifact
+- Worker service with a compact text classifier and optional trained artifact
 - Redis-backed cache when Redis is available, with an in-memory fallback
 - Retry on failed worker calls and basic load-aware routing
 - Scenario benchmark runner with saved results

docs/architecture.md

@@ -32,7 +32,7 @@ The benchmark runner uses both modes against the same slow-worker scenario so th
 
 ## Training
 
-`training/train_text_model.py` trains the tiny text model with PyTorch. On Apple Silicon it uses `mps`, and it exits instead of silently falling back to CPU unless `--allow-cpu` is passed.
+`training/train_text_model.py` trains the compact text model with PyTorch. On Apple Silicon it uses `mps`, and it exits instead of silently falling back to CPU unless `--allow-cpu` is passed.
 
 The trained artifact is plain JSON so the worker can load it without depending on PyTorch at inference time.
 

docs/model_card.md

@@ -2,9 +2,11 @@
 
 ## Model
 
-`tiny-text-mps-v1` is a small linear text classifier used by ModelMesh workers. It classifies short service-status text as either `healthy` or `risky`.
+`tiny-text-mps-v2` is a small MLP text classifier used by ModelMesh workers. It classifies short service-status text as either `healthy` or `risky`.
 
-The model is intentionally small. Its purpose is to make the training and serving path testable inside an inference mesh, not to compete with production NLP models.
+The model uses word-count features and character n-gram features. It trains with PyTorch, then exports to JSON so workers can run inference without PyTorch.
+
+The model is intentionally small. Its purpose is to make the training, export, evaluation, and serving path testable inside an inference mesh, not to compete with production NLP models.
 
 ## Training
 
@@ -18,14 +20,16 @@ Latest checked-in run:
 
 | Metric | Value |
 | --- | ---: |
+| Model type | mlp_text_classifier |
 | Device | mps |
-| Epochs | 320 |
+| Epochs | 260 |
 | Train examples | 3,276 |
 | Test examples | 820 |
-| Vocab size | 37 |
-| Parameters | 38 |
-| Train time | 0.4351 s |
-| Examples/sec | 2,409,512.37 |
+| Features | 140 |
+| Hidden units | 16 |
+| Parameters | 2,273 |
+| Train time | 0.3991 s |
+| Examples/sec | 2,134,408.08 |
 | Train accuracy | 1.0 |
 | Test accuracy | 1.0 |
 
@@ -44,23 +48,17 @@ The challenge set is a small handwritten set of service-status examples with mix
 | Metric | Value |
 | --- | ---: |
 | Examples | 16 |
-| Correct | 15 |
-| Accuracy | 0.9375 |
+| Correct | 16 |
+| Accuracy | 1.0 |
 
 Confusion matrix:
 
 | Expected | Predicted healthy | Predicted risky |
 | --- | ---: | ---: |
-| healthy | 6 | 1 |
+| healthy | 7 | 0 |
 | risky | 0 | 9 |
 
-The missed example was:
-
-```text
-worker recovered after timeout and is stable
-```
-
-The model predicted `risky`, mostly because `timeout` has a strong negative learned weight and the bag-of-words model does not understand recovery order.
+There were no misses on this challenge set. That is useful as a regression check, but it is still a small handwritten set and should not be treated as broad NLP evidence.
 
 Full evaluation output: `training/evaluation/latest.md`.
 
@@ -78,10 +76,10 @@ Use this model to exercise ModelMesh serving behavior:
 ## Limitations
 
 - Training data is generated service-status text, not a real production incident dataset.
-- The model is bag-of-words and does not understand word order.
+- The model uses word and character n-gram features, not a transformer or sequence model.
 - Accuracy numbers are useful for regression checks, not for claiming broad NLP performance.
 - The model is deliberately small so the project can focus on ML systems behavior.
 
 ## Next model improvement
 
-The next useful upgrade is a compact sequence model or character n-gram model trained on a larger labeled service-log dataset, still with MPS timing and artifact export.
+The next useful upgrade is a larger labeled service-log dataset or a compact sequence model, still with MPS timing, evaluation output, and JSON artifact export.

docs/report.md

@@ -33,8 +33,8 @@ flowchart LR
     gateway --> cache["Redis or memory cache"]
     gateway --> worker_a["Worker A"]
     gateway --> worker_b["Worker B"]
-    worker_a --> model_a["tiny-text-mps-v1"]
-    worker_b --> model_b["tiny-text-mps-v1"]
+    worker_a --> model_a["tiny-text-mps-v2"]
+    worker_b --> model_b["tiny-text-mps-v2"]
     gateway --> dashboard["Dashboard and metrics"]
 ```
 
@@ -49,13 +49,16 @@ Latest MPS training run on an M3 Pro:
 
 | Metric | Value |
 | --- | ---: |
+| Model type | mlp_text_classifier |
 | Device | mps |
-| Epochs | 320 |
+| Epochs | 260 |
 | Train examples | 3,276 |
 | Test examples | 820 |
-| Parameters | 38 |
-| Train time | 0.4351 s |
-| Examples/sec | 2,409,512.37 |
+| Features | 140 |
+| Hidden units | 16 |
+| Parameters | 2,273 |
+| Train time | 0.3991 s |
+| Examples/sec | 2,134,408.08 |
 | Train accuracy | 1.0 |
 | Test accuracy | 1.0 |
 
@@ -68,17 +71,17 @@ The model was also evaluated on a handwritten challenge set with mixed healthy a
 | Metric | Value |
 | --- | ---: |
 | Examples | 16 |
-| Correct | 15 |
-| Accuracy | 0.9375 |
+| Correct | 16 |
+| Accuracy | 1.0 |
 
 Confusion matrix:
 
 | Expected | Predicted healthy | Predicted risky |
 | --- | ---: | ---: |
-| healthy | 6 | 1 |
+| healthy | 7 | 0 |
 | risky | 0 | 9 |
 
-The one miss was a recovery sentence containing `timeout`. That is a reasonable failure for a bag-of-words model and is listed in `docs/model_card.md`.
+There were no misses on this challenge set. The set is small and handwritten, so the result is treated as a regression check rather than a broad NLP claim.
 
 ## Benchmark results
 

requirements-train.txt

@@ -1 +1,2 @@
 torch>=2.5
+numpy>=2.0

tests/test_model.py

@@ -1,6 +1,6 @@
 import json
 
-from worker.app.model import TinyTextModel, load_model
+from worker.app.model import DEFAULT_ARTIFACT, MLPTextModel, TinyTextModel, load_model
 
 
 def test_tiny_model_scores_healthy_text_higher() -> None:
@@ -28,3 +28,41 @@ def test_model_can_load_training_artifact(tmp_path) -> None:
     assert model.predict("stable")["label"] == "healthy"
     assert model.predict("broken")["label"] == "risky"
     assert model.version == "test-model"
+
+
+def test_model_can_load_v2_mlp_artifact(tmp_path) -> None:
+    artifact = tmp_path / "model.json"
+    artifact.write_text(json.dumps({
+        "model_version": "test-mlp",
+        "model_type": "mlp_text_classifier",
+        "feature_spec": {
+            "word_vocab": ["stable", "broken"],
+            "char_ngrams": [],
+            "char_ngram_range": [3, 4],
+        },
+        "weights": {
+            "hidden": [[1.0, 0.0], [0.0, 1.0]],
+            "output": [2.0, -2.0],
+        },
+        "biases": {
+            "hidden": [0.0, 0.0],
+            "output": 0.0,
+        },
+    }))
+
+    model = load_model(str(artifact))
+
+    assert isinstance(model, MLPTextModel)
+    assert model.predict("stable")["label"] == "healthy"
+    assert model.predict("broken")["label"] == "risky"
+    assert model.version == "test-mlp"
+
+
+def test_checked_in_artifact_uses_v2_schema() -> None:
+    artifact = json.loads(DEFAULT_ARTIFACT.read_text())
+
+    assert artifact["model_version"] == "tiny-text-mps-v2"
+    assert artifact["model_type"] == "mlp_text_classifier"
+    assert set(artifact) >= {"feature_spec", "weights", "biases", "training"}
+    assert set(artifact["weights"]) == {"hidden", "output"}
+    assert set(artifact["biases"]) == {"hidden", "output"}

tests/test_text_model_evaluation.py

@@ -5,6 +5,7 @@ def test_challenge_evaluation_has_expected_shape() -> None:
     result = evaluate()
 
     assert result["examples"] == 16
-    assert result["correct"] >= 12
+    assert result["model_type"] == "mlp_text_classifier"
+    assert result["correct"] >= 14
     assert set(result["confusion_matrix"]) == {"healthy", "risky"}
     assert set(result["per_label"]) == {"healthy", "risky"}

training/evaluate_text_model.py

@@ -107,6 +107,7 @@ def evaluate(model_artifact: Path | None = None) -> dict[str, Any]:
 
     return {
         "model_version": model.version,
+        "model_type": getattr(model, "model_type", "unknown"),
         "dataset": "handwritten service-status challenge set",
         "examples": total,
         "accuracy": round(correct / total, 4),
@@ -122,6 +123,7 @@ def render_markdown(result: dict[str, Any]) -> str:
         "# Model evaluation",
         "",
         f"Model version: `{result['model_version']}`",
+        f"Model type: `{result['model_type']}`",
         "",
         "| Metric | Value |",
         "| --- | ---: |",