Skip to content

PSA.md

Latest commit

 

History

History
79 lines (65 loc) · 3.33 KB

PSA.md

File metadata and controls

79 lines (65 loc) · 3.33 KB
Raw

Problem Statement and Architecture

Problem

Build and test workloads are bursty, expensive, and failure prone. Running every job on one CI host creates a throughput bottleneck. Adding workers improves capacity, but introduces harder questions:

  • Who owns a job when several workers request work concurrently?
  • What happens when a worker disappears halfway through a build?
  • How is a late result prevented from overwriting a newer retry?
  • Which completed work is safe to reuse?
  • How can an operator see backlog and recovery behavior?

The project addresses those coordination problems with a deliberately compact control plane.

Goals

  1. Distribute independent shell based build and test jobs across multiple workers.
  2. Recover automatically when a worker process or machine disappears.
  3. Preserve job state and attempt history across coordinator restarts.
  4. Prevent duplicate submissions and safely reuse deterministic results.
  5. Expose enough telemetry to explain the system while it runs.
  6. Stream live build output and retain declared outputs and crash evidence.
  7. Package the same services for Docker Compose, Kubernetes, and a short lived EKS demonstration.

Non goals for v0.2

  • compatibility with the Bazel Remote Execution API
  • hermetic build graph scheduling or remote action caching
  • hostile multi tenant execution
  • production authentication, authorization, or secrets brokering
  • globally distributed coordination
  • OCI image building or registry publication
  • guaranteed core generation across every host runtime
  • durable Kubernetes artifact retention before an S3-compatible backend is configured

Proposed architecture

The coordinator is the durable control plane. PostgreSQL is both the system of record and the queue serialization point. Stateless workers pull work rather than receiving inbound commands, making worker scaling and networking straightforward.

sequenceDiagram
    participant Client
    participant API as Coordinator
    participant DB as PostgreSQL
    participant W1 as Worker A
    participant W2 as Worker B

    Client->>API: POST job
    API->>DB: insert queued job
    W1->>API: request lease
    API->>DB: lock and claim oldest highest priority job
    API-->>W1: job plus generation 1 and expiry
    loop while command runs
        W1->>API: heartbeat generation 1
        API->>DB: extend lease
        W1->>API: fenced ordered log chunk
        API-->>Client: SSE log event
    end
    Note over W1: worker disappears
    API->>DB: expire lease and requeue
    W2->>API: request lease
    API-->>W2: same job, generation 2
    W2->>API: complete generation 2
    API->>DB: commit successful result
    W1-->>API: late completion generation 1
    API-->>W1: 409 stale lease
Loading

Success criteria

  • three local workers execute submitted jobs
  • stopping the leased worker causes the job to run on another worker after lease expiry
  • the recovered execution completes with attempt count 2 and lease generation 2
  • an old generation cannot complete that job
  • a repeated safe fingerprint returns as a cache hit without worker execution
  • stdout and stderr are visible before job completion
  • declared outputs can be downloaded with a verified checksum
  • signal crashes record explicit metadata and retain a core file when the runtime emits one
  • tests, lint, container health checks, and Kubernetes manifest rendering pass