Skip to content

docs: adopt the shared CueLABS Kafka bus in the system design #263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
siyanbolagiggs1 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

docs: adopt the shared CueLABS Kafka bus in the system design #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cea95c9
docs(decisions): X-11 shared message bus
siyanbolagiggs1 Oct 13, 2026
54ec8d7
chore(gitignore): ignore local copies of the CueLABS skills and perso…
siyanbolagiggs1 Oct 13, 2026
b2e11d7
docs(system-design): proposed system design, on the shared message bus
siyanbolagiggs1 Oct 13, 2026
a0fab6b
Merge remote-tracking branch 'upstream/main' into khalid-dev
siyanbolagiggs1 Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,9 @@ tmp/

# NOTE: do NOT ignore .dockerignore; commit lockfiles (package-lock.json,
# go.sum, pubspec.lock, .terraform.lock.hcl).

# ── Personal notes ──
/systemDesign.md

# ── Local copies of the CueLABS skills (installed from oss-engineering-standards) ──
.claude/skills/
20 changes: 20 additions & 0 deletions docs/decisions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,3 +216,23 @@ blocks.
the template); tier-2 fields are high-sensitivity in every data-model §4
classification; verification is delegated to the money/filing provider —
no in-house document review. ☑
- **X-11 Shared message bus (RATIFIED, boss 2026-10-13)**: expendit, apparule
and upstat share **one Aiven Kafka service** under the CueLABS delivery
standard's `references/shared-messaging.md` (the contract; read it before
designing any Kafka use). Three topics ("lanes") serve every product:
`cuelabs.uploads` (gateway → control plane), `cuelabs.requests` (control
plane → processing), `cuelabs.results` (processing → control plane); two
more are held in reserve and need a decision. Event types are
`<product>.<domain>.<event>`; consumers read the type first and skip what
isn't theirs. **Payloads go by pointer** (claim-check in the product's own
storage prefix) because every product can read the lanes. No compacted
topics: config travels in requests or as a snapshot object. Retention is
the service's fixed setting, so nothing may depend on the bus keeping a
message for more than a day; each control plane re-sends overdue work
from its own store. One Kafka user per product (`expendit`, `apparule`,
`upstat`); consumer groups `<product>-<service>`. The service powers off
after 24 h with no traffic and drops its messages: **keep-alive owner is
expendit** (its `api/common` reaper sends a heartbeat every 6 h and powers
the service back on through the Aiven API); apparule and upstat run no
heartbeat or watchdog. Bulk data never rides the bus (upstat telemetry
goes through the OpenTelemetry Collector instead). ☑
Loading