A fault tolerant distributed build and test execution platform that assigns work to a pool of workers, recovers jobs after worker loss, fences stale results, and reuses deterministic results.
This is a focused systems engineering prototype, not a replacement for Bazel, Buildbarn, or a hosted CI product. It exists to make the difficult coordination mechanics visible and testable in a small codebase.
- browser build console for submission, history, results, and downloads
- REST coordinator built with FastAPI
- durable job, worker, and attempt state in PostgreSQL
- ordered live stdout and stderr chunks streamed to browsers with SSE
- concurrent queue claims with
SELECT FOR UPDATE SKIP LOCKED - renewable execution leases and automatic reassignment after worker loss
- monotonically increasing lease generations that reject stale completions
- idempotent submissions and fingerprint based result reuse
- isolated temporary workspaces and a C++20 process executor with hard timeouts
- downloadable build artifact bundles with SHA 256 verification
- signal metadata, crash reports, and automatic core file collection when available
- Python, Git, Make, CMake, GCC, GDB, and common build tools in the worker image
- horizontally scalable workers packaged with Docker
- Prometheus metrics and a provisioned Grafana dashboard
- Docker Compose development stack
- Kubernetes base manifests and an AWS EKS overlay
- Python unit and API tests plus C++ compilation in CI
flowchart LR
C["Client / CI trigger"] -->|"submit and inspect jobs"| A["Coordinator API"]
B["Kaminos browser console"] -->|"REST and SSE"| A
A -->|"durable state and queue claims"| P[("PostgreSQL")]
W1["Worker 1"] -->|"lease, heartbeat, complete"| A
W2["Worker 2"] -->|"lease, heartbeat, complete"| A
W3["Worker N"] -->|"lease, heartbeat, complete"| A
W1 --> E1["C++ process executor"]
W2 --> E2["C++ process executor"]
W1 -->|"ordered logs and artifacts"| A
A --> S[("Artifact volume")]
A --> M["Prometheus"]
M --> G["Grafana"]
A worker never owns a job permanently. It receives a time limited lease and renews it while the command runs. If heartbeats stop, the coordinator expires the lease and makes the job available again. Every reassignment increments a generation number. A late completion carrying an old generation is rejected, which prevents a recovered job from being overwritten by stale work.
- Docker Engine with Docker Compose
curlfor submitting jobs from the command line
Docker Desktop, Colima, OrbStack, and Rancher Desktop are all suitable macOS runtimes. Windows users can use Docker Desktop with WSL2, while Linux users can run Docker Engine natively. Kaminos does not depend on a specific Docker runtime.
The examples below use the current docker compose syntax. On an older installation, replace it with docker-compose.
cd path/to/distributed-build-systemStart the Docker runtime installed on your system. Verify that the engine and Compose are available:
docker version
docker compose versionFor example, Colima users can start their runtime with colima start. Docker Desktop and similar graphical runtimes can be started through their applications.
Run the complete stack with three workers in the foreground:
docker compose up --build --scale worker=3Keep this terminal open to watch the live coordinator and worker logs. To run the stack in the background instead:
docker compose up --build --scale worker=3 --detachCheck the service status:
docker compose psOpen another terminal and submit a command to the coordinator:
curl -sS -X POST http://localhost:8000/api/v1/jobs \
-H 'Content-Type: application/json' \
-d '{
"command": "python -c \"print(sum(range(1000000)))\"",
"idempotency_key": "manual-demo-001",
"priority": 10,
"max_attempts": 3
}' | python3 -m json.toolThe initial response will usually show status as queued. Copy the returned job id.
Use a new idempotency_key for each logically new submission. Reusing the same key intentionally returns the original logical job.
Replace <JOB_ID> with the identifier returned during submission:
curl -sS http://localhost:8000/api/v1/jobs/<JOB_ID> \
| python3 -m json.toolOnce execution finishes, the response includes fields such as status, attempt_count, lease_owner, lease_generation, exit_code, stdout, stderr, and cache_hit.
List all submitted jobs:
curl -sS http://localhost:8000/api/v1/jobs \
| python3 -m json.toolWhen running the stack in detached mode:
docker compose logs --follow coordinator workerPress Control+C to stop following the logs. This does not stop detached containers.
Local endpoints:
| Service | URL |
|---|---|
| Kaminos build console | http://localhost:8000/ |
| API documentation | http://localhost:8000/docs |
| Coordinator metrics | http://localhost:8000/metrics |
| Prometheus | http://localhost:9090 |
| Grafana | http://localhost:3000 |
Grafana's local development credentials are admin / distbuild-local. They are intentionally nonproduction values.
If the stack is running in the foreground, first press Control+C. Then remove the containers and network:
docker compose downThe PostgreSQL volume and saved job history remain available for the next run. You may also stop your Docker runtime when it is no longer needed.
To deliberately erase the local PostgreSQL and MinIO volumes and start with empty state:
docker compose down --volumesThe --volumes operation permanently deletes locally persisted Kaminos data. Do not use it during an ordinary shutdown.
Pin a commit for reproducible caching:
curl -sS -X POST http://localhost:8000/api/v1/jobs \
-H 'Content-Type: application/json' \
-d '{
"repository":"https://github.com/example/project.git",
"revision":"0123456789abcdef",
"command":"cmake -S . -B build && cmake --build build && ctest --test-dir build",
"artifact_paths":["build/app","build/test-results/*.xml"],
"max_attempts":3,
"priority":10
}'Repository jobs using mutable HEAD deliberately bypass the cache. Command only jobs and repository jobs pinned to an immutable revision may reuse a successful result.
Declared artifact paths are relative to the cloned repository. After execution, matching regular files are collected into a gzip compressed tar archive, hashed, stored by the coordinator, and exposed through the job response and dashboard. Symlinks and paths that escape the workspace are rejected.
If a command terminates from a signal, Kaminos records the signal number. Workers enable core dumps and automatically retain core or core.* with a generated crash report when the container runtime produces a core file. Core availability remains dependent on the host and container runtime's core-dump policy.
The public Akbonline/Snake-game repository provides a small reproducible example. Because it is a Python application, its build validation runs the repository's test target rather than compiling a native binary.
In the Kaminos build console, submit:
| Field | Value |
|---|---|
| Repository URL | https://github.com/Akbonline/Snake-game.git |
| Revision | fecb47a49145d10c84149375e86aae37bfaa8a5d |
| Command | make test-nodeps |
| Artifact paths | Leave empty |
| Priority | 10 |
| Max attempts | 3 |
The same build can be submitted from a terminal:
curl -sS -X POST http://localhost:8000/api/v1/jobs \
-H 'Content-Type: application/json' \
-d '{
"repository":"https://github.com/Akbonline/Snake-game.git",
"revision":"fecb47a49145d10c84149375e86aae37bfaa8a5d",
"command":"make test-nodeps",
"idempotency_key":"snake-game-fecb47a-demo",
"priority":10,
"max_attempts":3
}' | python3 -m json.toolKaminos queues the request, leases it to an available worker, clones and checks out the pinned commit, executes the Make target, and streams the output back to the browser. The verified demonstration completed successfully with all nine tests passing.
Use a new idempotency_key to force another logical submission. Reusing the value above returns the original job, which demonstrates Kaminos's idempotent request handling. Since this test target does not generate a package, the example intentionally has no retained artifact. A repository that writes a binary, wheel, archive, report, or other output can declare those files under artifact_paths for download.
FastAPI exposes an interactive OpenAPI interface for submitting and inspecting jobs, while the provisioned Grafana dashboard visualizes queue depth, active workers, completed jobs, lease activity, and cache hits.
| Interactive API documentation | Grafana operations dashboard |
|---|---|
 |
 |
python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/ruff check src tests
.venv/bin/pytest --cov=src
cmake -S src/executor -B build/executor -DCMAKE_BUILD_TYPE=Release
cmake --build build/executor --parallel- Problem statement and architecture
- Detailed design
- Operations guide
- Demo script
- Verification record
- Blog post draft
- AWS demonstration plan
v0.2 is intentionally honest about its boundaries:
- the API and worker protocol have no authentication and must stay on a trusted local or private network
- commands are arbitrary code; workers must be treated as disposable and untrusted
- the coordinator is single replica because schema migration and leader election are not implemented
- local artifacts use a coordinator-managed Docker volume; the Kubernetes base uses pod-local storage and requires S3-compatible storage for durable production retention
- OCI or Docker image publication is not implemented; workers do not receive the host Docker socket
- cache keys still omit the full toolchain image digest and dependency closure
- Git credentials, per tenant isolation, autoscaling, cancellation, and S3 artifact upload are future work
See the design document for the next production hardening steps.