installation-reference.md

shuttle

🧵 The workflow engine of the quipu stack — every transition signed, every window frozen

A shuttle carries the weft through the warp. Runs move through their workflow's declared steps; each pass leaves a signed knot in the record.

Workflow engines usually mean a daemon, a queue, a server, and a dashboard that disagrees with reality. Shuttle is none of those. A local JSONL log is the hot path; quipu is the governed record. Runs move through their workflow's declared steps, every transition is signed by the agent that performed it (per-agent ed25519), and the append-only history is exported into quipu's time-windowed operational graphs — where completed windows are deep-frozen whole into read-only archives that stay queryable forever.

Consumers — shantytown crews, creel tabs, NeuralAmplifier — read the graph, never shuttle.

Stack overview

Role in the stack

  agents / crews                          (advance runs; each step signed)
        │
        ▼
    shuttle                               (state machine + JSONL outbox)
        │  export: signed Turtle, idempotent batches
        ▼
     quipu                                (windowed graphs · deep-frozen archives)
        │
        ▼
  shantytown · creel · NeuralAmplifier    (read the graph, never shuttle)

• camayoc owns the vocabulary: the workflow slice (WorkflowRun, TransitionEvent, …), each term owed to a competency question in competency/workflow-and-archive.md.
• caboodle installs and verifies shuttle as part of the box.

Quick start

pip install -e .                       # Python 3.11+, one dep (cryptography)

shuttle keys init agent-a --workflow triage
#   -> prints the public key + registration Turtle; a HUMAN loads it into
#      the dataKind=identity graph. Shuttle never registers its own keys.

shuttle define examples/triage.json
shuttle start triage run-1 --agent agent-a
shuttle advance run-1 --step claim  --agent agent-a    # signed
shuttle advance run-1 --step work   --agent agent-a
shuttle advance run-1 --step finish --agent agent-a    # terminal
shuttle status

shuttle export                          # drain the log into quipu windows
shuttle verify run-1                    # re-verify signatures FROM the graph
shuttle freeze-window 2026-07           # archive a completed window

The shape

file	owns
shuttle/model.py	the pure state machine; events are the truth, state is a fold
shuttle/state.py	the append-only JSONL outbox + export watermark
shuttle/signing.py	per-agent ed25519 (0600 host files), shuttle-transition-v1 canonical message
shuttle/quipu_client.py	windowed writes ride /knot's STRICT graph lane; a SHACL refusal is a refusal, never a silent 200
shuttle/windows.py	{ns}/window/shuttle/runs/{YYYY-MM} graphs + the urn:shuttle:dataset:open dataset
shuttle/export.py	log → signed Turtle → idempotent /episode batches
shuttle/seeds.py	seed steps: a seeds work item performs the step, reconcile advances the run when it closes
shuttle/cli.py	the whole v1 surface

Design: docs/design/shuttle.md.

Seed steps

A transition can be performed by a seeds work item instead of by hand:

{"step": "review", "from": "open", "to": "reviewed",
 "seed": {"title": "Review the change", "labels": ["review"]},
 "on_abandon": "reject"}

shuttle reconcile --agent <you> (on a timer, or by hand) does two things for every open run:

• For each seed step legal from the run's state, it runs sd create --workflow-run <run> --step <step> --visit <n>. seeds derives the id from that key, so repeating the create returns the same seed. A run that re-enters the state gets a new visit and a new seed.
• It reads the seed with sd show --json. Outcome done advances the run through the step, signed, with the seed id, outcome and close time recorded as evidence. Any other outcome takes on_abandon. With no on_abandon the run is FLAGGED (exit 1) and never advanced silently.

seeds knows nothing about workflows beyond the link field. A missed tick costs one tick, not a stuck run. SHUTTLE_SD names the sd binary; where seeds stores its work is sd's own configuration.

What runs today

v1 is landed and in production use: signed runs, windowed export, freezable history, seed steps, and the cross-repo acceptance passing against a live quipu. On the fleet host it carries the ansible apply lane, repo watch and stack watch workflows, and an unattended actuator installs each release.

just check    # parse + file-size ratchet
just test     # check + the unit suite
just e2e      # the cross-repo acceptance against a live quipu

Documentation: the shuttle book. Changes: CHANGELOG.md.

License

Apache-2.0. Releases up to and including v0.5.0 were published under the MIT License and remain available under it.

For pre-registered job identities, put --require-existing-keys before the subcommand (for example, shuttle --require-existing-keys advance ...). It requires an existing Ed25519 key owned by the current Unix user, in a regular 0600 file; missing keys, final-component symlinks and unsafe files are refused. The policy covers the whole command, including export verification, and never creates a replacement key. Older installations reject this flag. Default first-use key initialization remains available for the normal registration ceremony. This option checks local custody, not remote registration or job scope admission. SDK callers can use signing.existing_keys_only() around an entire operation, or sign_transition(..., require_existing_key=True) for one signature. Parent directory custody remains the operator's responsibility.