Harden hosted demo controls and validation

.env.example

@@ -3,5 +3,6 @@ REDIS_URL=redis://redis:6379/0
 REQUEST_TIMEOUT_SECONDS=1.5
 ROUTER_STRATEGY=load_aware
 CACHE_ENABLED=true
+GATEWAY_CONTROLS_ENABLED=true
 SIMULATE_DELAY_MS=0
 FAIL_RATE=0

README.md

@@ -62,6 +62,13 @@ Deployment notes are in `docs/deployment.md`.
 
 There is also a Render Blueprint in `render.yaml` for a hosted demo with one public gateway, two private workers, and a managed cache.
 
+## Hosted demo
+
+The hosted demo is Render-ready but does not have a checked-in public URL yet. The Blueprint
+keeps workers private and disables public worker failure controls.
+
+Deployment and validation steps are in `docs/hosted-demo.md`.
+
 ## Benchmarks
 
 The scenario runner starts local gateway and worker processes, sends load, then writes JSON and markdown results.

docs/deployment.md

@@ -22,7 +22,7 @@ For a hosted demo, deploy the services as a private internal mesh:
 
 | Service | Public | Notes |
 | --- | --- | --- |
-| gateway | yes | Exposes `/predict`, `/dashboard`, `/metrics`, and worker control proxy |
+| gateway | yes | Exposes `/predict`, `/dashboard`, `/metrics`, and `/workers` |
 | worker-a | no | Internal service on port 9000 |
 | worker-b | no | Internal service on port 9000 |
 | redis | no | Internal cache only |
@@ -35,13 +35,15 @@ REDIS_URL=redis://redis:6379/0
 REQUEST_TIMEOUT_SECONDS=1.5
 ROUTER_STRATEGY=load_aware
 CACHE_ENABLED=true
+GATEWAY_CONTROLS_ENABLED=false
 ```
 
 ## What not to overclaim
 
 This is not a production ML platform yet. It is a compact inference mesh that demonstrates training, serving, caching, routing, live worker controls, failure recovery, and benchmark evidence.
 
-The next real hosting step is choosing a provider and wiring its private service networking. The Docker setup here is intentionally provider-neutral so the app can go to Fly.io, Render, Railway, a VM, or a Kubernetes cluster without changing the Python code.
+The next real hosting step is applying the Render Blueprint and validating the gateway URL with
+`scripts/smoke_check.py`.
 
 ## Render Blueprint
 
@@ -53,4 +55,6 @@ The next real hosting step is choosing a provider and wiring its private service
 
 The gateway reads `WORKER_A_HOSTPORT` and `WORKER_B_HOSTPORT`, which Render fills from each private worker service. That avoids hardcoding internal hostnames.
 
+The Blueprint also sets `GATEWAY_CONTROLS_ENABLED=false`, so the public dashboard can show worker state without exposing delay or failure controls.
+
 Important: the private worker services use Render's `starter` plan in the Blueprint because private services do not run on the free plan. Review the cost in Render before applying the Blueprint.

docs/hosted-demo.md

@@ -0,0 +1,44 @@
+# Hosted demo
+
+ModelMesh uses the Render Blueprint in `render.yaml` for the public demo shape:
+
+- one public gateway
+- two private workers
+- one managed cache
+
+The hosted gateway exposes `/health`, `/predict`, `/metrics`, `/workers`, and `/dashboard`.
+Worker delay and failure controls are disabled in the Blueprint with
+`GATEWAY_CONTROLS_ENABLED=false`.
+
+## Deploy from Render
+
+1. Open Render and choose **New** > **Blueprint**.
+2. Connect this repository.
+3. Select `render.yaml`.
+4. Review the service list before applying:
+   - `modelmesh-gateway` uses the `starter` plan.
+   - `modelmesh-worker-a` and `modelmesh-worker-b` use private services.
+   - `modelmesh-cache` uses Render Key Value.
+5. Apply the Blueprint.
+6. Wait until the gateway and both workers finish their first deploy.
+
+Private worker services may cost money on Render. Review the pricing screen before applying
+the Blueprint.
+
+## Validate the live gateway
+
+After Render gives the gateway a public URL, run:
+
+```bash
+python scripts/smoke_check.py --base-url https://your-render-gateway-url
+```
+
+The smoke check verifies:
+
+- `/health`
+- `/predict`
+- `/metrics`
+- `/workers`
+- `/dashboard`
+
+Only add the live URL to the README or GitHub repo homepage after this smoke check passes.

docs/report.md

@@ -119,8 +119,9 @@ The repo includes:
 - local Docker Compose
 - production-style Docker Compose
 - Docker image builds in CI
+- Render Blueprint for a hosted demo
 
-A separate Render Blueprint PR prepares the hosted demo. It still needs a provider account action and cost review before there is a public URL.
+The hosted demo still needs a Render account action and cost review before there is a public URL. The Blueprint keeps workers private and disables public worker failure controls.
 
 ## Limitations
 

gateway/app/dashboard.py

@@ -113,6 +113,9 @@
     .bad {
       color: #fca5a5;
     }
+    .muted {
+      color: #64748b;
+    }
     input {
       width: 76px;
       border: 1px solid #334155;
@@ -215,6 +218,7 @@
       errors: document.getElementById("errors"),
       workers: document.getElementById("workers"),
     };
+    const controlsEnabled = __CONTROLS_ENABLED__;
 
     function pct(value) {
       return `${Math.round(value * 100)}%`;
@@ -234,6 +238,16 @@
 
     window.controlWorker = controlWorker;
 
+    function controlCell(worker) {
+      if (!controlsEnabled) {
+        return `<span class="muted">disabled</span>`;
+      }
+      return `
+        <button onclick="controlWorker(${worker.index}, document.getElementById('delay-${worker.index}').value, document.getElementById('fail-${worker.index}').value)">Apply</button>
+        <button class="secondary" onclick="controlWorker(${worker.index}, 0, 0)">Reset</button>
+      `;
+    }
+
     async function refresh() {
       try {
         const [health, metrics, workers] = await Promise.all([
@@ -247,6 +261,7 @@
         ids.cache.textContent = pct(metrics.cache_hit_rate);
         ids.p95.textContent = `${metrics.p95_latency_ms} ms`;
         ids.errors.textContent = metrics.worker_errors;
+        const inputState = controlsEnabled ? "" : "disabled";
         ids.workers.innerHTML = workers.map((worker) => `
           <tr>
             <td>${worker.url}</td>
@@ -255,12 +270,9 @@
             <td class="num">${worker.requests}</td>
             <td class="num ${worker.failures ? "bad" : ""}">${worker.failures}</td>
             <td class="num">${worker.latency_ema_ms} ms</td>
-            <td class="num"><input id="delay-${worker.index}" type="number" min="0" max="5000" value="${worker.delay_ms ?? 0}"></td>
-            <td class="num"><input id="fail-${worker.index}" type="number" min="0" max="1" step="0.05" value="${worker.fail_rate ?? 0}"></td>
-            <td>
-              <button onclick="controlWorker(${worker.index}, document.getElementById('delay-${worker.index}').value, document.getElementById('fail-${worker.index}').value)">Apply</button>
-              <button class="secondary" onclick="controlWorker(${worker.index}, 0, 0)">Reset</button>
-            </td>
+            <td class="num"><input ${inputState} id="delay-${worker.index}" type="number" min="0" max="5000" value="${worker.delay_ms ?? 0}"></td>
+            <td class="num"><input ${inputState} id="fail-${worker.index}" type="number" min="0" max="1" step="0.05" value="${worker.fail_rate ?? 0}"></td>
+            <td>${controlCell(worker)}</td>
           </tr>
         `).join("");
       } catch (error) {

gateway/app/main.py

@@ -128,7 +128,10 @@ async def health() -> dict[str, str | int]:
 
 @app.get("/dashboard", response_class=HTMLResponse)
 async def dashboard() -> str:
-    return DASHBOARD_HTML
+    return DASHBOARD_HTML.replace(
+        "__CONTROLS_ENABLED__",
+        str(settings.controls_enabled).lower(),
+    )
 
 
 @app.get("/workers")
@@ -145,6 +148,9 @@ async def control_worker(
     worker_index: int,
     update: WorkerControlRequest,
 ) -> dict[str, object]:
+    if not settings.controls_enabled:
+        raise HTTPException(status_code=403, detail="worker controls are disabled")
+
     worker = worker_for_index(worker_index)
     config = await control_remote_worker(worker, update)
     return {"worker_index": worker_index, "worker_url": worker.url, "config": config}

gateway/app/settings.py

@@ -60,6 +60,7 @@ class GatewaySettings:
     redis_url: str | None = None
     cache_enabled: bool = True
     request_timeout_seconds: float = 1.0
+    controls_enabled: bool = True
 
     @classmethod
     def from_env(cls, env: Mapping[str, str] | None = None) -> GatewaySettings:
@@ -77,4 +78,5 @@ def from_env(cls, env: Mapping[str, str] | None = None) -> GatewaySettings:
                 values.get("REQUEST_TIMEOUT_SECONDS"),
                 default=1.0,
             ),
+            controls_enabled=parse_bool(values.get("GATEWAY_CONTROLS_ENABLED"), default=True),
         )

render.yaml

@@ -53,3 +53,5 @@ services:
         value: load_aware
       - key: CACHE_ENABLED
         value: "true"
+      - key: GATEWAY_CONTROLS_ENABLED
+        value: "false"

scripts/smoke_check.py

@@ -18,6 +18,11 @@ def get_json(url: str, timeout: float = 2.0) -> dict:
         return json.loads(response.read().decode("utf-8"))
 
 
+def get_text(url: str, timeout: float = 2.0) -> str:
+    with request.urlopen(url, timeout=timeout) as response:
+        return response.read().decode("utf-8")
+
+
 def post_json(url: str, payload: dict, timeout: float = 2.0) -> dict:
     data = json.dumps(payload).encode("utf-8")
     req = request.Request(
@@ -54,19 +59,25 @@ def main() -> None:
     )
     metrics = get_json(f"{base_url}/metrics")
     workers = get_json(f"{base_url}/workers")
+    dashboard = get_text(f"{base_url}/dashboard")
 
     if prediction["prediction"]["label"] != "healthy":
         raise RuntimeError(f"unexpected prediction: {prediction}")
+    if "requests" not in metrics or "p95_latency_ms" not in metrics:
+        raise RuntimeError(f"unexpected metrics payload: {metrics}")
     if not workers:
         raise RuntimeError("gateway returned no workers")
+    if "ModelMesh" not in dashboard or "Gateway metrics" not in dashboard:
+        raise RuntimeError("dashboard did not return the expected HTML")
 
     print(
         json.dumps(
             {
                 "health": health,
                 "prediction": prediction,
                 "metrics": metrics,
-                "workers": workers,
+                "worker_count": len(workers),
+                "dashboard": "ok",
             },
             indent=2,
         )

tests/test_dashboard.py

@@ -1,5 +1,8 @@
+from dataclasses import replace
+
 from fastapi.testclient import TestClient
 
+from gateway.app import main as gateway_main
 from gateway.app.main import app
 
 
@@ -10,3 +13,17 @@ def test_dashboard_serves_html() -> None:
     assert response.status_code == 200
     assert "ModelMesh" in response.text
     assert "Gateway metrics" in response.text
+
+
+def test_dashboard_marks_controls_disabled(monkeypatch) -> None:
+    monkeypatch.setattr(
+        gateway_main,
+        "settings",
+        replace(gateway_main.settings, controls_enabled=False),
+    )
+
+    with TestClient(app) as client:
+        response = client.get("/dashboard")
+
+    assert response.status_code == 200
+    assert "const controlsEnabled = false;" in response.text

tests/test_gateway_controls.py

@@ -0,0 +1,20 @@
+from dataclasses import replace
+
+from fastapi.testclient import TestClient
+
+from gateway.app import main as gateway_main
+from gateway.app.main import app
+
+
+def test_gateway_controls_can_be_disabled(monkeypatch) -> None:
+    monkeypatch.setattr(
+        gateway_main,
+        "settings",
+        replace(gateway_main.settings, controls_enabled=False),
+    )
+
+    with TestClient(app) as client:
+        response = client.post("/workers/0/control", json={"delay_ms": 0})
+
+    assert response.status_code == 403
+    assert response.json()["detail"] == "worker controls are disabled"

tests/test_gateway_settings.py

@@ -10,13 +10,15 @@ def test_gateway_settings_reads_explicit_worker_urls() -> None:
         "CACHE_ENABLED": "false",
         "REQUEST_TIMEOUT_SECONDS": "2.5",
         "REDIS_URL": "redis://cache:6379/0",
+        "GATEWAY_CONTROLS_ENABLED": "false",
     })
 
     assert settings.worker_urls == ["http://a:9000", "http://b:9000"]
     assert settings.router_strategy == "round_robin"
     assert settings.cache_enabled is False
     assert settings.request_timeout_seconds == 2.5
     assert settings.redis_url == "redis://cache:6379/0"
+    assert settings.controls_enabled is False
 
 
 def test_gateway_settings_reads_render_hostports() -> None: