Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,27 @@
# ModelMesh

[![Tests](https://github.com/iPrakharV/ModelMesh/actions/workflows/test.yml/badge.svg)](https://github.com/iPrakharV/ModelMesh/actions/workflows/test.yml)

A small inference gateway for testing how ML services behave when traffic, caching, and worker failures get involved.

The first version runs a FastAPI gateway in front of a few model workers. The gateway accepts prediction requests, checks a cache, routes to a worker, retries on failure, and keeps basic latency and error metrics. Worker routing tracks recent latency, failures, and in-flight requests.

![ModelMesh dashboard](docs/assets/dashboard.png)

## Architecture

```mermaid
flowchart LR
client["Client or traffic replay"] --> gateway["FastAPI gateway"]
gateway --> cache["Redis cache or memory fallback"]
gateway --> router["Load-aware router"]
router --> workerA["Worker A"]
router --> workerB["Worker B"]
workerA --> model["Exported JSON model artifact"]
workerB --> model
gateway --> metrics["Metrics and dashboard"]
```

## Run it

```bash
Expand Down
Binary file added docs/assets/dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.