feat(pathways): resilient cluster start that never stalls the data pump

.claude-plugin/marketplace.json

@@ -7,7 +7,7 @@
       "name": "flowcore",
       "source": "./",
       "description": "Operate Flowcore data cores, IAM and Data Pathways, and build apps with @flowcore/pathways.",
-      "version": "0.1.0",
+      "version": "0.2.0",
       "author": { "name": "Flowcore" }
     }
   ]

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "flowcore",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Flowcore MCP server plus skills for the platform model, IAM, Data Pathways and the @flowcore/pathways SDK.",
   "author": { "name": "Flowcore", "email": "support@flowcore.io", "url": "https://flowcore.io" },
   "homepage": "https://flowcore.io",

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## [0.2.0] - 2026-09-30
+
+### Added
+- `flowcore-pathways`: `references/resilient-startup.md`, a production start runtime for virtual pathways. It retries the whole start with `stopPump()` + `stopCluster()` between attempts, exits when the retries run out, exits when a leader holds the lease without a running pump, detects `Failed to bootstrap leader runtime after becoming leader` in the logger adapter, and separates liveness from readiness. It also adds handler timeouts sized to `deliveryTimeoutMs`.
+- Evaluation scenario for a resilient cluster start.
+
+### Changed
+- `flowcore-pathways`: new rule 11. A start error must never leave a silent stall. The minimal example now makes one start attempt (`startOnce`) and uses the resilient runtime.
+- `flowcore-pathways` troubleshooting: rows for a leader without a pump, `Cluster already started` on retry, `{}` error logs, hanging handlers, dropped events after retries, and Bun `NODE_ENV` inlining.
+- `flowcore-data-pathways` troubleshooting: row for a virtual pathway without pulses while pods are healthy.
+
+### Fixed
+- `flowcore-pathways` cluster failure modes: a PostgreSQL outage does not make the leader step down. Lease renewal throws, the error is logged as `Lease loop error`, and the leader keeps its role (verified against `@flowcore/pathways` 2.10.4).
+
 ## [0.1.0] - 2026-09-30
 
 ### Added

plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
   "name": "flowcore",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Connects coding agents to the Flowcore event platform and teaches them to operate it: the resource hierarchy, IAM policies that actually match, virtual and managed Data Pathways, and building apps with the @flowcore/pathways SDK.",
   "author": {
     "name": "Flowcore",

skills/flowcore-data-pathways/references/troubleshooting.md

@@ -10,6 +10,7 @@ Start every diagnosis read-only: `get_data_pathway`, then `show_pathway_dashboar
 | Right event types | Compare `virtualConfig.flowTypes` (virtual) or `config.sources[].eventTypes` (managed) with the event types that actually receive events (`get_time_buckets`) |
 | Events exist | `get_time_buckets` + `get_events` for the event type |
 | Virtual: service running | The consumer's pump runs only on the cluster leader. Ask the user for the service logs and leader state. |
+| Virtual: no pulses, pods healthy | The leader holds the lease without a running pump (a caught start error, or `Failed to bootstrap leader runtime after becoming leader`). Delete the leader pod to recover. Then fix the service start with the `flowcore-pathways` skill (`references/resilient-startup.md`). |
 | Managed: endpoint reachable | The endpoint must be public HTTPS and accept the configured auth headers |
 | Permissions | The pathway or service API key needs `fetch` on the data core (see `flowcore-iam`) |
 

skills/flowcore-pathways/SKILL.md

@@ -62,7 +62,14 @@ Each rule has a reason. Keep the reason when you explain a decision to the user.
     (no `fireAndForget`) returns after the handler has processed the event, so you can return the real
     resource. Use `fireAndForget: true` only when the user asks, or for write-only and remote-consumer
     events. A write timeout means the event is stored but not yet processed: do not retry the write.
-11. **Never invent APIs.** If an option or method is not in the installed library, say
+11. **A start error must never leave a silent stall.** Retry the whole start with teardown
+    (`stopPump()` then `stopCluster()`) between attempts, exit the process when the retries run out,
+    and exit when a leader holds the lease without a running pump. Serve liveness before the start.
+    Reason: the lease and the pump are separate. A pod that caught a `startPump()` error, or a new
+    leader whose bootstrap failed (`Failed to bootstrap leader runtime after becoming leader`), keeps
+    the lease with no pump, and no other pod can take over. Use the runtime in
+    [resilient-startup](references/resilient-startup.md) for every production virtual service.
+12. **Never invent APIs.** If an option or method is not in the installed library, say
     "I cannot verify that API" and offer a verified alternative. Known non-existent examples:
     `enableEncryption()`, field-level encryption, automatic key rotation, `getStatus()`.
 
@@ -88,14 +95,15 @@ virtual pathway registration. Any other value gives a single local pump and no r
 ```typescript
 import { z } from "zod" // zod 3 (peer dependency ^3.25.63)
 import {
+  type ClusterManager,
   createPostgresPathwayChunkStore,
   createPostgresPathwayCoordinator,
   createPostgresPathwayDeliveryStore,
   createPostgresPathwayState,
   createPostgresPumpStateManagerFactory,
-  ConsoleLogger,
   PathwaysBuilder,
 } from "@flowcore/pathways"
+import { pathwaysLogger } from "./pathways-logger" // see references/resilient-startup.md
 
 const connectionString = process.env.DATABASE_URL!
 const statePrefix = "orders_service" // one prefix per deployable sharing a database
@@ -114,7 +122,7 @@ export const pathways = new PathwaysBuilder({
   pathwayName: process.env.FLOWCORE_PATHWAY_NAME!, // distinct per environment
   pathwayLabels: { name: "Orders service", description: "Consumes order events" },
   autoProvision: { pathway: true }, // register the virtual pathway (skipped in development)
-  logger: new ConsoleLogger(), // default is NoopLogger, which hides every log line below
+  logger: pathwaysLogger, // from references/resilient-startup.md; the default NoopLogger hides everything
 })
   .withPathwayState(createPostgresPathwayState({ connectionString, statePrefix }))
   .withPathwayChunkStore(createPostgresPathwayChunkStore({ connectionString, statePrefix }))
@@ -138,10 +146,14 @@ export const pathways = new PathwaysBuilder({
     await ordersReadModel.upsert(event.payload)
   })
 
-export async function startPathways() {
+// ONE start attempt. Never call it directly with a bare `.catch()`: run it through
+// startPathways() in references/resilient-startup.md, which retries with teardown, exits when the
+// retries run out, and watches the leader. Serve HTTP liveness first, then `void startPathways()`.
+export async function startOnce(): Promise<ClusterManager | null> {
+  let cluster: ClusterManager | null = null
   if (isProduction) {
     const coordinator = await createPostgresPathwayCoordinator({ connectionString }, { statePrefix })
-    await pathways.startCluster({
+    cluster = await pathways.startCluster({
       coordinator,
       advertisedAddress: process.env.POD_IP!, // host only; the library builds ws://host:port
       port: 9090,
@@ -151,11 +163,7 @@ export async function startPathways() {
     stateManagerFactory: await createPostgresPumpStateManagerFactory({ connectionString, statePrefix }),
     notifier: { type: "websocket" },
   })
-}
-
-export async function stopPathways() {
-  await pathways.stopPump()
-  await pathways.stopCluster()
+  return cluster
 }
 
 // In a request handler (same app handles the event):
@@ -173,8 +181,10 @@ export async function recordExport(orderId: string, target: string) {
 }
 ```
 
-`ordersReadModel` stands for your own persistence code. In real code, pass an adapter to your own
-logger that implements the library `Logger` interface (`debug`, `info`, `warn`, `error`).
+`ordersReadModel` stands for your own persistence code. Give the handler a timeout (see
+[resilient-startup](references/resilient-startup.md#handler-timeout)); a handler that never settles
+blocks its pump group. The logger adapter must log `message`, `name` and `stack` of errors and must
+keep the leader-bootstrap fatal detection.
 
 ## Verification checklist
 
@@ -193,7 +203,12 @@ Run these before you call the work done.
 - [ ] Events flow: an awaited `pathways.write()` returns without timeout and the read model changes.
       To inspect stored events use MCP `get_time_buckets`, then `get_events` with the event type id.
 - [ ] Generated ORM migrations contain no `pathway_` tables (grep the SQL).
-- [ ] Graceful shutdown calls `stopPump()` then `stopCluster()`.
+- [ ] Graceful shutdown calls `stopPump()` then `stopCluster()`, once.
+- [ ] Production start goes through the resilient runtime: no bare `startPathways().catch(...)`, retry
+      with teardown, exit after the last attempt, leader watchdog, fatal detection in the logger.
+- [ ] Liveness does not depend on pathways; readiness reports the pathways status.
+- [ ] Failure test: stop the database during boot, and delete the leader pod. The start either
+      recovers or the pod exits. No pod stays Ready as leader without a running pump.
 
 ## API keys and IAM
 
@@ -211,6 +226,9 @@ every other id is a full UUID. Use the `flowcore-iam` skill for policies, roles
   pump, `statePrefix`, notifiers, concurrency and pump groups, pause and resume, failure modes.
 - [references/schema-and-migrations.md](references/schema-and-migrations.md): library-owned
   tables, Drizzle `tablesFilter`, read-only mirrors, deploy-time migrations.
+- [references/resilient-startup.md](references/resilient-startup.md): the production start runtime
+  (retry with teardown, exit, leader watchdog, health endpoints, handler timeouts) and why the naive
+  start stalls.
 - [references/troubleshooting.md](references/troubleshooting.md): symptoms, causes and fixes.
 - [references/evaluation-scenarios.md](references/evaluation-scenarios.md): scenarios to test
   answers produced with this skill.

skills/flowcore-pathways/references/cluster-and-pump.md

@@ -29,14 +29,18 @@ await pathways.startPump({ stateManagerFactory, notifier: { type: "websocket" }
 In production + virtual, `startPump()` without an active cluster throws
 `Cluster mode must be started before production virtual pump startup`.
 
-Shutdown:
+This is one attempt. In production run it through the runtime in
+[resilient-startup](resilient-startup.md): retry with `stopPump()` + `stopCluster()` between attempts,
+exit when the retries run out, and exit when a leader has no running pump. Two library facts make
+this necessary:
 
-```typescript
-process.on("SIGTERM", async () => {
-  await pathways.stopPump()
-  await pathways.stopCluster()
-})
-```
+- A failed `startCluster()` leaves the cluster manager in place. A second call throws
+  `Cluster already started` until `stopCluster()` runs.
+- The lease is not tied to the pump. A leader whose pump start or bootstrap failed keeps renewing the
+  lease, and no other instance can take over.
+
+Shutdown, once: `await pathways.stopPump()` then `await pathways.stopCluster()`. `stopCluster()`
+releases the lease when this instance leads.
 
 ## `startCluster(options)` (`PathwayClusterOptions`)
 
@@ -147,5 +151,9 @@ ignored (all pumps are reset, with a warning).
 | Leader crashes | Lease expires after `leaseTtlMs`; another instance takes over from the saved cursor. |
 | Follower crashes | Dropped after `staleThresholdMs`; events go to the remaining instances. |
 | No followers | Leader processes events itself. |
-| PostgreSQL unavailable | Lease renewal fails, the leader steps down and stops its pump. No instance leads until the database is back. |
+| PostgreSQL unavailable | Lease renewal throws. The error is logged as `Lease loop error` and the leader keeps its role. No other instance can acquire the lease until the database is back. |
+| `startPump()` throws on the leader, and the app continues | The leader keeps the lease with no pump. Delivery stops for the whole cluster. Prevent it with [resilient-startup](resilient-startup.md). |
+| A new leader fails its bootstrap | Logged as `Failed to bootstrap leader runtime after becoming leader`. No retry and no lease release. Same stall. The runtime in [resilient-startup](resilient-startup.md) exits on it. |
+| A data pump group fails after a successful start | The library restarts that group with backoff (1 s doubling to 30 s, without limit). The other groups continue. |
+| Handler never settles | Its pump group stops delivery. Give handlers a timeout. |
 | Network partition | Two leaders are possible for up to `leaseTtlMs`. Keep handlers idempotent. |

skills/flowcore-pathways/references/evaluation-scenarios.md

@@ -36,6 +36,28 @@ Must not include:
 - managed delivery as an automatic fallback
 - cluster mode inside Next.js or another serverless runtime
 
+## Resilient cluster start
+
+Input: "Our virtual pathway stopped delivering after a deploy, but all pods are healthy." or
+"Make the production start of this pathways service robust."
+
+Must include:
+- the cause: a leader holds the lease without a running pump, after a caught `startPump()` error or
+  a failed leader bootstrap (`Failed to bootstrap leader runtime after becoming leader`)
+- immediate recovery: delete the leader pod, then check pulses with `get_data_pathway` /
+  `show_pathway_dashboard`
+- a start that retries with `stopPump()` + `stopCluster()` between attempts and exits after the last
+  attempt
+- a leader watchdog and fatal detection in the logger adapter
+- liveness independent of pathways, readiness that reports the pathways status
+- error logs with `message`, `name` and `stack`, and a handler timeout
+
+Must not include:
+- `startPathways().catch(console.error)` or any start error that is only logged
+- a retry that calls `startCluster()` again without `stopCluster()`
+- `pathways.pump?.isRunning` as the only health signal
+- disabling or switching the pathway to managed to make delivery work
+
 ## Serverless production target
 
 Input: "Deploy the consumer to Vercel."