diff --git a/.changeset/wss-reconfigure-resubscribe.md b/.changeset/wss-reconfigure-resubscribe.md new file mode 100644 index 000000000..2af1c2914 --- /dev/null +++ b/.changeset/wss-reconfigure-resubscribe.md @@ -0,0 +1,5 @@ +--- +'@viamrobotics/motion-tools': patch +--- + +Fix world state entities going stale after a machine reconfigure diff --git a/docs/superpowers/plans/2026-05-08-pipeline-view.md b/docs/superpowers/plans/2026-05-08-pipeline-view.md new file mode 100644 index 000000000..925655a27 --- /dev/null +++ b/docs/superpowers/plans/2026-05-08-pipeline-view.md @@ -0,0 +1,2158 @@ +# Pipeline View Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the MVP slice of the pipeline view (spec phases 1–3): a `/pipeline` route that auto-derives perception stages from the live machine config, renders them as a DAG with directed edges, and "spotlights" selected stages in the existing 3D scene with per-stage color tinting. + +**Architecture:** New SvelteKit route at `/pipeline` with a trimmed providers shell. A new `src/lib/pipeline/` library owns derivation (pure functions), context (`usePipeline`), and UI (`PipelineGraph`, `StageNode`, `EdgeLayer`). Existing `usePointclouds` / `usePointcloudObjects` hooks gain an optional `filter` parameter and stamp every entity they spawn with a new `PipelineSource(stageId)` trait so the route can match scene entities to graph nodes. Visibility (`Invisible` trait) and tint (`PipelineTint` + `OriginalColors` traits) are reactive effects in the route page. + +**Tech Stack:** Svelte 5 runes, Threlte, Koota ECS, TanStack Query, `@viamrobotics/sdk`, `@viamrobotics/svelte-sdk`, Vitest, `@testing-library/svelte`, Playwright. + +**Spec:** `docs/superpowers/specs/2026-05-08-pipeline-view-design.md` + +**Out of scope (deferred to follow-up plans):** 2D preview pane (spec §6 / phase 4), override panel + persistence (spec §7 / phase 5). + +--- + +## File Structure + +**Created** + +- `src/lib/pipeline/types.ts` — shared types (`Stage`, `Edge`, `StageOutput`, `PipelineGraph`, `RGB`, `Warning`). +- `src/lib/pipeline/tint.ts` — deterministic stage-id → RGB mapping. +- `src/lib/pipeline/derive.ts` — pure functions building `PipelineGraph` from `(config, propertiesMap)`. +- `src/lib/pipeline/visibility.ts` — pure helper: given active set + entity list, compute which to mark `Invisible`. +- `src/lib/pipeline/usePipeline.svelte.ts` — Svelte context: graph, active set, status, tint, solo/toggle/clear API. +- `src/lib/pipeline/StageNode.svelte` — single graph node (icon, label, status dot, tint chip). +- `src/lib/pipeline/EdgeLayer.svelte` — SVG arrow layer between nodes. +- `src/lib/pipeline/layout.ts` — DAG layout (column-major topo levels) producing `{x,y}` per stage id. +- `src/lib/pipeline/PipelineGraph.svelte` — left-pane panel composing nodes + edges. +- `src/lib/pipeline/__tests__/derive.spec.ts` +- `src/lib/pipeline/__tests__/tint.spec.ts` +- `src/lib/pipeline/__tests__/layout.spec.ts` +- `src/lib/pipeline/__tests__/visibility.spec.ts` +- `src/lib/pipeline/__tests__/StageNode.spec.ts` +- `src/lib/pipeline/__tests__/PipelineGraph.spec.ts` +- `src/routes/pipeline/+page.svelte` — composition only; mounts providers, canvas, graph panel, effects. +- `src/routes/pipeline/+page.ts` — extracts `partID` from URL params. +- `e2e/pipeline.test.ts` — Playwright smoke test (matches local `*.test.ts` convention). + +**Modified** + +- `src/lib/ecs/traits.ts` — add `PipelineSource(stageId: string)`, `PipelineTint({r,g,b})`. +- `src/lib/hooks/usePointclouds.svelte.ts` — accept optional `filter`, stamp `PipelineSource` on spawn, export `applyFilter`. +- `src/lib/hooks/usePointcloudObjects.svelte.ts` — same shape (re-uses `applyFilter`). +- `src/lib/hooks/usePartConfig.svelte.ts` — widen `PartConfig` to expose `attributes` on components and a typed `services[]`. +- `src/lib/components/SceneProviders.svelte` — accept optional `pointcloudsFilter` / `pointcloudObjectsFilter` props, threaded into the existing `providePointclouds` / `providePointcloudObjects` calls. +- `src/lib/components/Entities/Points.svelte` — read `PipelineTint` and prefer it over geometry's `color` attribute when present. +- `.gitignore` — add `.superpowers/`. + +--- + +## Conventions for every task + +- **TDD discipline.** Write the failing test first, run it to see the failure, write the smallest passing implementation, run again to see green, commit. +- **Run scope.** Use `pnpm test -- ` (Vitest) for the file under test, not the full suite, until the task closes. End each phase with `pnpm check && pnpm lint && pnpm test`. +- **Commit format.** Conventional Commits: `feat(pipeline): …`, `test(pipeline): …`, `refactor(hooks): …`. One commit per task unless a task explicitly says "split." +- **Imports.** Use `$lib/...` aliases (matches existing code). +- **Svelte 5 only.** Runes (`$state`, `$derived`, `$effect`, `$props`); no Svelte 4 syntax. No `` — snippets only. +- **No emojis** in code or commit messages. + +--- + +## Phase 1 — Skeleton + +### Task 0: Housekeeping + +**Files:** +- Modify: `.gitignore` + +- [ ] **Step 1:** Append `.superpowers/` on a new line in `.gitignore`. +- [ ] **Step 2:** Verify with `git status` that `.superpowers/` is now ignored. +- [ ] **Step 3:** Commit. + +```bash +git add .gitignore +git commit -m "chore: ignore .superpowers brainstorming workspace" +``` + +--- + +### Task 1: Pipeline types + +**Files:** +- Create: `src/lib/pipeline/types.ts` + +- [ ] **Step 1:** Write the file. Pure type module — no tests required. + +```ts +// src/lib/pipeline/types.ts + +export type StageId = string + +export type StageOutput = + | 'pointcloud' + | 'image' + | 'objects' + | 'detections' + | 'classifications' + +export type StageApi = 'camera' | 'vision' + +export interface Stage { + id: StageId + label: string + api: StageApi + model: string + outputs: StageOutput[] + derivedFrom: 'config' | 'override' +} + +export interface Edge { + from: StageId + to: StageId + derivedFrom: 'config' | 'override' +} + +export interface Warning { + stageId: StageId | null + message: string +} + +export interface PipelineGraph { + stages: Stage[] + edges: Edge[] + warnings: Warning[] +} + +export interface RGB { + r: number + g: number + b: number +} + +export type StageStatus = 'idle' | 'loading' | 'ok' | 'error' +``` + +- [ ] **Step 2:** `pnpm check` — must pass with no errors. +- [ ] **Step 3:** Commit. + +```bash +git add src/lib/pipeline/types.ts +git commit -m "feat(pipeline): add shared types" +``` + +--- + +### Task 2: Deterministic stage-id tint + +**Files:** +- Create: `src/lib/pipeline/tint.ts` +- Test: `src/lib/pipeline/__tests__/tint.spec.ts` + +- [ ] **Step 1: Write the failing test.** + +```ts +// src/lib/pipeline/__tests__/tint.spec.ts +import { describe, expect, it } from 'vitest' + +import { tintForStageId } from '../tint' + +describe('tintForStageId', () => { + it('returns r,g,b values in [0,1]', () => { + const rgb = tintForStageId('wrist-cam-left') + expect(rgb.r).toBeGreaterThanOrEqual(0) + expect(rgb.r).toBeLessThanOrEqual(1) + expect(rgb.g).toBeGreaterThanOrEqual(0) + expect(rgb.g).toBeLessThanOrEqual(1) + expect(rgb.b).toBeGreaterThanOrEqual(0) + expect(rgb.b).toBeLessThanOrEqual(1) + }) + + it('is deterministic for the same id', () => { + expect(tintForStageId('merged-cloud')).toEqual(tintForStageId('merged-cloud')) + }) + + it('produces different colors for different ids', () => { + const a = tintForStageId('wrist-cam-left') + const b = tintForStageId('wrist-cam-right') + expect(a).not.toEqual(b) + }) + + it('distributes hues across many ids', () => { + const ids = Array.from({ length: 20 }, (_, i) => `stage-${i}`) + const tints = new Set(ids.map((id) => JSON.stringify(tintForStageId(id)))) + // Expect close to N distinct colors (hash collisions tolerated up to ~10%) + expect(tints.size).toBeGreaterThanOrEqual(18) + }) +}) +``` + +- [ ] **Step 2:** Run `pnpm test -- src/lib/pipeline/__tests__/tint.spec.ts`. Expected: FAIL ("Cannot find module '../tint'"). + +- [ ] **Step 3: Write implementation.** + +```ts +// src/lib/pipeline/tint.ts +import type { RGB } from './types' + +const hashString = (s: string): number => { + let h = 2166136261 + for (let i = 0; i < s.length; i += 1) { + h ^= s.charCodeAt(i) + h = Math.imul(h, 16777619) + } + return h >>> 0 +} + +const hslToRgb = (h: number, s: number, l: number): RGB => { + const c = (1 - Math.abs(2 * l - 1)) * s + const hp = h * 6 + const x = c * (1 - Math.abs((hp % 2) - 1)) + let r = 0 + let g = 0 + let b = 0 + if (hp < 1) [r, g, b] = [c, x, 0] + else if (hp < 2) [r, g, b] = [x, c, 0] + else if (hp < 3) [r, g, b] = [0, c, x] + else if (hp < 4) [r, g, b] = [0, x, c] + else if (hp < 5) [r, g, b] = [x, 0, c] + else [r, g, b] = [c, 0, x] + const m = l - c / 2 + return { r: r + m, g: g + m, b: b + m } +} + +export const tintForStageId = (id: string): RGB => { + const hue = (hashString(id) % 360) / 360 + return hslToRgb(hue, 0.65, 0.55) +} +``` + +- [ ] **Step 4:** Run the test again. Expected: PASS. + +- [ ] **Step 5:** Commit. + +```bash +git add src/lib/pipeline/tint.ts src/lib/pipeline/__tests__/tint.spec.ts +git commit -m "feat(pipeline): deterministic stage-id tint helper" +``` + +--- + +### Task 3: Derive — stage list (no edges yet) + +`derive.ts` is built incrementally. Task 3 produces stages from config + properties map. Edges land in Task 15 (Phase 3). + +**Files:** +- Create: `src/lib/pipeline/derive.ts` +- Test: `src/lib/pipeline/__tests__/derive.spec.ts` + +- [ ] **Step 1: Write the failing test.** + +```ts +// src/lib/pipeline/__tests__/derive.spec.ts +import { describe, expect, it } from 'vitest' + +import { buildGraph } from '../derive' + +const config = { + components: [ + { name: 'wrist-cam-left', api: 'rdk:component:camera', model: 'webcam', attributes: {} }, + { + name: 'merged-cloud', + api: 'rdk:component:camera', + model: 'transform', + attributes: { source: 'wrist-cam-left' }, + }, + { name: 'arm-1', api: 'rdk:component:arm', model: 'ur5', attributes: {} }, + ], + services: [ + { + name: 'object-segmenter', + api: 'rdk:service:vision', + model: 'segmenter', + attributes: { camera_name: 'merged-cloud' }, + }, + ], +} + +const properties = new Map([ + ['wrist-cam-left', { supportsPcd: true, mimeTypes: ['image/jpeg'] }], + ['merged-cloud', { supportsPcd: true, mimeTypes: [] }], + [ + 'object-segmenter', + { classificationsSupported: false, detectionsSupported: false, objectPointCloudsSupported: true }, + ], +]) + +describe('buildGraph: stages', () => { + it('emits one stage per camera with relevant outputs', () => { + const graph = buildGraph(config, properties) + const ids = graph.stages.map((s) => s.id).sort() + expect(ids).toEqual(['merged-cloud', 'object-segmenter', 'wrist-cam-left']) + }) + + it('skips arms and other irrelevant components', () => { + const graph = buildGraph(config, properties) + expect(graph.stages.find((s) => s.id === 'arm-1')).toBeUndefined() + }) + + it('infers camera outputs from properties', () => { + const graph = buildGraph(config, properties) + const wrist = graph.stages.find((s) => s.id === 'wrist-cam-left')! + expect(wrist.outputs).toEqual(expect.arrayContaining(['pointcloud', 'image'])) + const merged = graph.stages.find((s) => s.id === 'merged-cloud')! + expect(merged.outputs).toEqual(['pointcloud']) + }) + + it('infers vision outputs from properties', () => { + const graph = buildGraph(config, properties) + const seg = graph.stages.find((s) => s.id === 'object-segmenter')! + expect(seg.outputs).toEqual(['objects']) + expect(seg.api).toBe('vision') + }) + + it('drops cameras with no relevant outputs', () => { + const graphProps = new Map([ + ['cam', { supportsPcd: false, mimeTypes: [] }], + ]) + const graph = buildGraph( + { + components: [ + { name: 'cam', api: 'rdk:component:camera', model: 'webcam', attributes: {} }, + ], + services: [], + }, + graphProps + ) + expect(graph.stages).toHaveLength(0) + }) + + it('returns empty edges in this milestone', () => { + const graph = buildGraph(config, properties) + expect(graph.edges).toEqual([]) + }) +}) +``` + +- [ ] **Step 2:** Run `pnpm test -- src/lib/pipeline/__tests__/derive.spec.ts`. Expected: FAIL. + +- [ ] **Step 3: Write implementation.** + +```ts +// src/lib/pipeline/derive.ts +import type { + Edge, + PipelineGraph, + Stage, + StageApi, + StageOutput, + Warning, +} from './types' + +export interface CameraProperties { + supportsPcd?: boolean + mimeTypes?: string[] +} + +export interface VisionProperties { + classificationsSupported?: boolean + detectionsSupported?: boolean + objectPointCloudsSupported?: boolean +} + +export type StageProperties = CameraProperties | VisionProperties + +export type PropertiesMap = ReadonlyMap + +interface ResourceConfig { + name: string + api?: string + model?: string + attributes?: Record +} + +interface MachineConfig { + components?: ResourceConfig[] + services?: ResourceConfig[] +} + +const apiOf = (res: ResourceConfig): StageApi | undefined => { + if (res.api?.includes(':camera')) return 'camera' + if (res.api?.includes(':vision')) return 'vision' + return undefined +} + +const cameraOutputs = (props: CameraProperties | undefined): StageOutput[] => { + const outputs: StageOutput[] = [] + if (props?.supportsPcd) outputs.push('pointcloud') + if (props?.mimeTypes && props.mimeTypes.length > 0) outputs.push('image') + return outputs +} + +const visionOutputs = (props: VisionProperties | undefined): StageOutput[] => { + const outputs: StageOutput[] = [] + if (props?.objectPointCloudsSupported) outputs.push('objects') + if (props?.detectionsSupported) outputs.push('detections') + if (props?.classificationsSupported) outputs.push('classifications') + return outputs +} + +const resourceToStage = ( + res: ResourceConfig, + properties: PropertiesMap +): Stage | undefined => { + const api = apiOf(res) + if (!api) return undefined + const props = properties.get(res.name) + const outputs = + api === 'camera' + ? cameraOutputs(props as CameraProperties | undefined) + : visionOutputs(props as VisionProperties | undefined) + if (outputs.length === 0) return undefined + return { + id: res.name, + label: res.name, + api, + model: res.model ?? '', + outputs, + derivedFrom: 'config', + } +} + +export const buildGraph = ( + config: MachineConfig, + properties: PropertiesMap +): PipelineGraph => { + const stages: Stage[] = [] + const warnings: Warning[] = [] + const edges: Edge[] = [] // edges land in phase 3 + + for (const res of [...(config.components ?? []), ...(config.services ?? [])]) { + const stage = resourceToStage(res, properties) + if (stage) stages.push(stage) + } + + return { stages, edges, warnings } +} +``` + +- [ ] **Step 4:** Run the test. Expected: PASS. + +- [ ] **Step 5:** Commit. + +```bash +git add src/lib/pipeline/derive.ts src/lib/pipeline/__tests__/derive.spec.ts +git commit -m "feat(pipeline): derive stages from machine config + properties" +``` + +--- + +### Task 4: `usePipeline` Svelte context + +**Files:** +- Create: `src/lib/pipeline/usePipeline.svelte.ts` + +`usePipeline` is mostly orchestration. Tested implicitly through component tests in Tasks 6 and 7. Pure-function bits (active-set transitions) should be testable but the API itself is small enough that direct unit testing isn't load-bearing. + +- [ ] **Step 1:** Write the file. + +```ts +// src/lib/pipeline/usePipeline.svelte.ts +import { SvelteSet } from 'svelte/reactivity' +import { getContext, setContext } from 'svelte' + +import type { PipelineGraph, RGB, StageId, StageStatus } from './types' + +import { tintForStageId } from './tint' + +const KEY = Symbol('pipeline-context') + +interface PipelineContext { + graph: () => PipelineGraph + active: () => ReadonlySet + tint: (id: StageId) => RGB + status: (id: StageId) => StageStatus + useNativeColors: () => boolean + setUseNativeColors: (value: boolean) => void + solo: (id: StageId) => void + toggle: (id: StageId) => void + clear: () => void +} + +export interface ProvidePipelineParams { + graph: () => PipelineGraph + status?: (id: StageId) => StageStatus +} + +export const providePipeline = (params: ProvidePipelineParams): PipelineContext => { + const active = new SvelteSet() + let useNativeColors = $state(false) + + const ctx: PipelineContext = { + graph: params.graph, + active: () => active, + tint: tintForStageId, + status: params.status ?? (() => 'idle'), + useNativeColors: () => useNativeColors, + setUseNativeColors: (value) => { + useNativeColors = value + }, + solo: (id) => { + active.clear() + active.add(id) + }, + toggle: (id) => { + if (active.has(id)) active.delete(id) + else active.add(id) + }, + clear: () => active.clear(), + } + + setContext(KEY, ctx) + return ctx +} + +export const usePipeline = (): PipelineContext => { + const ctx = getContext(KEY) + if (!ctx) throw new Error('usePipeline called outside providePipeline') + return ctx +} +``` + +- [ ] **Step 2:** `pnpm check`. Expected: PASS. + +- [ ] **Step 3:** Commit. + +```bash +git add src/lib/pipeline/usePipeline.svelte.ts +git commit -m "feat(pipeline): usePipeline context with active-set + tint API" +``` + +--- + +### Task 5: `StageNode` component + +**Files:** +- Create: `src/lib/pipeline/StageNode.svelte` +- Test: `src/lib/pipeline/__tests__/StageNode.spec.ts` + +- [ ] **Step 1: Write the failing test.** + +```ts +// src/lib/pipeline/__tests__/StageNode.spec.ts +import { render, screen } from '@testing-library/svelte' +import userEvent from '@testing-library/user-event' +import { describe, expect, it, vi } from 'vitest' + +import StageNode from '../StageNode.svelte' + +const baseStage = { + id: 'wrist-cam-left', + label: 'Wrist L', + api: 'camera' as const, + model: 'webcam', + outputs: ['pointcloud' as const, 'image' as const], + derivedFrom: 'config' as const, +} + +describe('StageNode', () => { + it('renders the label', () => { + render(StageNode, { + stage: baseStage, + active: false, + status: 'ok', + tint: { r: 1, g: 0, b: 0 }, + onclick: vi.fn(), + }) + expect(screen.getByText('Wrist L')).toBeInTheDocument() + }) + + it('marks itself as active via aria-pressed', () => { + render(StageNode, { + stage: baseStage, + active: true, + status: 'ok', + tint: { r: 1, g: 0, b: 0 }, + onclick: vi.fn(), + }) + expect(screen.getByRole('button', { name: /Wrist L/ })).toHaveAttribute( + 'aria-pressed', + 'true' + ) + }) + + it('fires onclick with shiftKey when shift-clicked', async () => { + const onclick = vi.fn() + const user = userEvent.setup() + render(StageNode, { + stage: baseStage, + active: false, + status: 'ok', + tint: { r: 1, g: 0, b: 0 }, + onclick, + }) + await user.keyboard('{Shift>}') + await user.click(screen.getByRole('button', { name: /Wrist L/ })) + await user.keyboard('{/Shift}') + expect(onclick).toHaveBeenCalledWith(expect.objectContaining({ shiftKey: true })) + }) +}) +``` + +- [ ] **Step 2:** Run `pnpm test -- src/lib/pipeline/__tests__/StageNode.spec.ts`. Expected: FAIL (component not found). + +- [ ] **Step 3: Write component.** + +```svelte + + + + +``` + +- [ ] **Step 4:** Run the test. Expected: PASS. + +- [ ] **Step 5:** Commit. + +```bash +git add src/lib/pipeline/StageNode.svelte src/lib/pipeline/__tests__/StageNode.spec.ts +git commit -m "feat(pipeline): StageNode component" +``` + +--- + +### Task 6: `PipelineGraph` panel (no edges) + +**Files:** +- Create: `src/lib/pipeline/PipelineGraph.svelte` +- Test: `src/lib/pipeline/__tests__/PipelineGraph.spec.ts` + +The panel renders a stack of `StageNode` instances in deterministic order, exposes a "Use native colors" toggle, and dispatches click events to `usePipeline`. Edges land in Task 14. + +- [ ] **Step 1: Write the failing test.** + +```ts +// src/lib/pipeline/__tests__/PipelineGraph.spec.ts +import { render, screen } from '@testing-library/svelte' +import userEvent from '@testing-library/user-event' +import { describe, expect, it } from 'vitest' + +import GraphHarness from './__fixtures__/GraphHarness.svelte' + +const stages = [ + { + id: 'wrist-cam-left', + label: 'Wrist L', + api: 'camera' as const, + model: 'webcam', + outputs: ['pointcloud' as const], + derivedFrom: 'config' as const, + }, + { + id: 'merged-cloud', + label: 'Merged', + api: 'camera' as const, + model: 'transform', + outputs: ['pointcloud' as const], + derivedFrom: 'config' as const, + }, +] + +describe('PipelineGraph', () => { + it('renders one node per stage', () => { + render(GraphHarness, { stages, edges: [], warnings: [] }) + expect(screen.getByRole('button', { name: /Wrist L/ })).toBeInTheDocument() + expect(screen.getByRole('button', { name: /Merged/ })).toBeInTheDocument() + }) + + it('plain click solos the stage', async () => { + const user = userEvent.setup() + const harness = render(GraphHarness, { stages, edges: [], warnings: [] }) + await user.click(screen.getByRole('button', { name: /Wrist L/ })) + expect(harness.component.activeIds()).toEqual(['wrist-cam-left']) + }) + + it('shift-click adds without clearing', async () => { + const user = userEvent.setup() + const harness = render(GraphHarness, { stages, edges: [], warnings: [] }) + await user.click(screen.getByRole('button', { name: /Wrist L/ })) + await user.keyboard('{Shift>}') + await user.click(screen.getByRole('button', { name: /Merged/ })) + await user.keyboard('{/Shift}') + expect(harness.component.activeIds().sort()).toEqual(['merged-cloud', 'wrist-cam-left']) + }) + + it('toggle native colors button is wired', async () => { + const user = userEvent.setup() + const harness = render(GraphHarness, { stages, edges: [], warnings: [] }) + expect(harness.component.useNativeColors()).toBe(false) + await user.click(screen.getByRole('button', { name: /native colors/i })) + expect(harness.component.useNativeColors()).toBe(true) + }) +}) +``` + +- [ ] **Step 2:** Create the test fixture component. + +```svelte + + + + + + +``` + +- [ ] **Step 3:** Run `pnpm test -- src/lib/pipeline/__tests__/PipelineGraph.spec.ts`. Expected: FAIL. + +- [ ] **Step 4: Write component.** + +```svelte + + + + +``` + +- [ ] **Step 5:** Run the test. Expected: PASS. + +- [ ] **Step 6:** Commit. + +```bash +git add src/lib/pipeline/PipelineGraph.svelte src/lib/pipeline/__tests__/PipelineGraph.spec.ts src/lib/pipeline/__tests__/__fixtures__/GraphHarness.svelte +git commit -m "feat(pipeline): PipelineGraph panel rendering nodes" +``` + +--- + +### Task 6.5: Widen `PartConfig` + thread filter props through `SceneProviders` + +Two prerequisite shared-code edits the route depends on. Bundled here so the route in Task 7 mounts cleanly. **No tests** in this task — it's wiring; behavior is validated by Task 9 / Task 12 / Task 18 against real config. + +**Files:** +- Modify: `src/lib/hooks/usePartConfig.svelte.ts` +- Modify: `src/lib/components/SceneProviders.svelte` + +- [ ] **Step 1: Widen `PartConfig`.** In `src/lib/hooks/usePartConfig.svelte.ts`, replace the `PartConfig` interface with: + +```ts +export interface PartConfigComponent { + name: string + api?: string + model?: string + attributes?: Record + frame?: Frame +} + +export interface PartConfigService { + name: string + api?: string + model?: string + attributes?: Record +} + +export interface PartConfig { + components: PartConfigComponent[] + services?: PartConfigService[] + fragment_mods?: { + fragment_id: string + // eslint-disable-next-line @typescript-eslint/no-explicit-any + mods: any[] + }[] +} +``` + +`Struct.toJson()` already returns these fields when the underlying config has them; we're just exposing them in the typed surface. + +Also export `usePartConfig` as a named export if it isn't already (the existing file may only export `providePartConfig`): + +```ts +export const usePartConfig = (): PartConfigContext => { + const ctx = getContext(key) + if (!ctx) throw new Error('usePartConfig called outside providePartConfig') + return ctx +} +``` + +(Check first: if `usePartConfig` already exists, leave it alone.) + +- [ ] **Step 2:** `pnpm check`. Expected: PASS — no consumers should regress because the new fields are optional. + +- [ ] **Step 3: Thread filter props through `SceneProviders`.** In `src/lib/components/SceneProviders.svelte`: + +```svelte + +``` + +(The `{ filter }` option arrives in Tasks 9 + 10. Check this task in **after** those tasks — see "Task ordering" below.) + +> **Task ordering note.** Tasks 9 and 10 add the `filter` option to the hook signatures. This task (6.5) is logically a *prerequisite* for Task 7's route shell, but its `SceneProviders` edit *uses* the option. Implement the prerequisite split this way: do Step 1 (widen `PartConfig`) now, then jump to Tasks 9 + 10 to add the `filter` plumbing, then return to do Step 3 (thread filter props through `SceneProviders`), then proceed to Task 7. The phase-2 ordering in the plan reflects narrative order; this is the build order. + +- [ ] **Step 4:** `pnpm check`. Expected: PASS. + +- [ ] **Step 5:** Commit (split into two commits — one per file — for clean review): + +```bash +git add src/lib/hooks/usePartConfig.svelte.ts +git commit -m "feat(hooks): widen PartConfig with attributes + services" + +git add src/lib/components/SceneProviders.svelte +git commit -m "feat(scene): thread pointcloud filter props through SceneProviders" +``` + +--- + +### Task 7: `/pipeline` route shell + +**Files:** +- Create: `src/routes/pipeline/+page.ts` +- Create: `src/routes/pipeline/+page.svelte` + +The page reuses `` for the full provider tree (frames, geometries, draw API, hierarchy, **and** the pointcloud hooks with our filter), and mounts `Settings` / `RefreshRate` inside the Canvas snippet via the existing `domPortal` pattern (matching `App.svelte`). Property fetching for graph derivation is wired via `useResourceNames` + `createResourceQuery` (mirroring `usePointclouds`). No scene filter behavior yet — that lands in Task 12. + +- [ ] **Step 1:** Add `+page.ts`. + +```ts +// src/routes/pipeline/+page.ts +import type { PageLoad } from './$types' + +export const ssr = false +export const prerender = false + +export const load: PageLoad = ({ url }) => { + const partID = url.searchParams.get('partID') ?? '' + return { partID } +} +``` + +- [ ] **Step 2:** Add `+page.svelte`. + +```svelte + + + +
+ + +
+ + + {#snippet children({ focus: _focus })} + + + + {@const pointclouds = usePointclouds()} + {@const pointcloudObjects = usePointcloudObjects()} +
+
+ pointclouds.refetch()} + /> + pointcloudObjects.refetch()} + /> +
+ +
+ {/snippet} + + +
+
+``` + +- [ ] **Step 3:** `pnpm check`. Expected: PASS. (TypeScript will grumble about the `pointcloudsFilter` / `pointcloudObjectsFilter` props on `SceneProviders` until Task 6.5 Step 3 is done — see the **Task ordering note** in Task 6.5. If you implemented in plan order, jump back and finish Task 6.5 Step 3 now.) + +- [ ] **Step 4:** `pnpm dev` and visit `http://localhost:5173/pipeline?partID=`. Expected: graph populated with derived stages (no edges yet), Threlte canvas renders, RefreshRate controls visible top-right. Until Task 12 lands, the scene shows every pipeline stage's points; clicking a node has no scene effect yet (only graph state changes). + +> **Settings UX caveat.** `Settings.svelte` opens its `FloatingPanel` via a gear button portalled into ``, which lives in the (deliberately omitted) `` overlay. That means the gear button doesn't render on this route. The Settings panel itself functions correctly once opened — but you may need to add a Pipeline-route-specific opener (e.g. a small button in the `PipelineGraph` header) in a follow-up if Settings access becomes important. Not a v1 blocker. + +- [ ] **Step 5:** Commit. + +```bash +git add src/routes/pipeline/+page.ts src/routes/pipeline/+page.svelte +git commit -m "feat(pipeline): /pipeline route with providers + graph panel" +``` + +--- + +## Phase 2 — Spotlight scene + +### Task 8: ECS traits for pipeline rendering + +**Files:** +- Modify: `src/lib/ecs/traits.ts` + +- [ ] **Step 1:** Append to `src/lib/ecs/traits.ts`: + +```ts +import { trait } from 'koota' +// (existing imports unchanged) + +/** The stage id (resource name) that produced this entity. Set by the pipeline-aware hooks. */ +export const PipelineSource = trait(() => '') + +/** Per-stage tint applied in pipeline view; rendered in place of geometry color. */ +export const PipelineTint = trait({ r: 0, g: 0, b: 0 }) +``` + +(`OriginalColors` from spec §5 is not added — Task 13 takes a renderer-side override path that doesn't need it. If a future renderer adopts `PipelineTint` and *does* need to mutate geometry, add the trait then.) + +- [ ] **Step 2:** `pnpm check`. Expected: PASS. + +- [ ] **Step 3:** Commit. + +```bash +git add src/lib/ecs/traits.ts +git commit -m "feat(ecs): pipeline-rendering traits" +``` + +--- + +### Task 9: `usePointclouds` filter + `PipelineSource` stamp + +**Files:** +- Modify: `src/lib/hooks/usePointclouds.svelte.ts` +- Test: `src/lib/hooks/__tests__/usePointclouds.filter.spec.ts` + +The hook is large; this task only adds (a) an optional `filter` to `providePointclouds`, applied in `enabledClients`, and (b) a `traits.PipelineSource(name)` trait on every entity it spawns. No other behavior changes. The new test exercises the filter shape only — full hook integration is implicitly tested by the existing app. + +- [ ] **Step 1: Write the failing test.** + +```ts +// src/lib/hooks/__tests__/usePointclouds.filter.spec.ts +import { describe, expect, it } from 'vitest' + +import { applyFilter } from '$lib/hooks/usePointclouds.svelte' + +describe('usePointclouds applyFilter', () => { + const allow = new Set(['cam-a', 'cam-c']) + + it('returns true when no filter is given', () => { + expect(applyFilter('cam-a', undefined)).toBe(true) + }) + + it('returns true when name is in the filter', () => { + expect(applyFilter('cam-a', allow)).toBe(true) + }) + + it('returns false when name is missing from the filter', () => { + expect(applyFilter('cam-b', allow)).toBe(false) + }) +}) +``` + +- [ ] **Step 2:** Run `pnpm test -- src/lib/hooks/__tests__/usePointclouds.filter.spec.ts`. Expected: FAIL. + +- [ ] **Step 3: Modify the hook.** Add at the top of `usePointclouds.svelte.ts` (after imports): + +```ts +export const applyFilter = (name: string, filter: ReadonlySet | undefined): boolean => + filter === undefined || filter.has(name) + +export interface ProvidePointcloudsOptions { + filter?: () => ReadonlySet | undefined +} +``` + +Update the `providePointclouds` signature: + +```ts +export const providePointclouds = ( + partID: () => string, + options: ProvidePointcloudsOptions = {} +) => { + // ... existing code ... +} +``` + +In the `enabledClients` `$derived.by`, add the filter check alongside the existing checks. Replace this stretch: + +```ts +if ( + fetchedPropQueries && + client.current?.name && + interval !== RefetchRates.OFF && + disabledCameras[client.current?.name] !== true +) { + results.push(client as { current: CameraClient }) +} +``` + +with: + +```ts +const allow = options.filter?.() +if ( + fetchedPropQueries && + client.current?.name && + interval !== RefetchRates.OFF && + disabledCameras[client.current?.name] !== true && + applyFilter(client.current.name, allow) +) { + results.push(client as { current: CameraClient }) +} +``` + +In the spawn block, add `traits.PipelineSource(name)` to the entity traits: + +```ts +const entity = world.spawn( + ...hierarchy.parentTraits(name), + traits.Name(`${name} pointcloud`), + traits.BufferGeometry(geometry), + traits.Points, + traits.PipelineSource(name) +) +``` + +- [ ] **Step 4:** Run the test. Expected: PASS. + +- [ ] **Step 5:** `pnpm check && pnpm test -- usePointclouds`. Expected: existing tests stay green. + +- [ ] **Step 6:** Commit. + +```bash +git add src/lib/hooks/usePointclouds.svelte.ts src/lib/hooks/__tests__/usePointclouds.filter.spec.ts +git commit -m "feat(hooks): usePointclouds optional filter + PipelineSource trait" +``` + +--- + +### Task 10: `usePointcloudObjects` filter + `PipelineSource` stamp + +**Files:** +- Modify: `src/lib/hooks/usePointcloudObjects.svelte.ts` + +Same shape as Task 9. The test from Task 9 exports `applyFilter` from `usePointclouds`; this hook should import-and-reuse the same helper. + +- [ ] **Step 1:** At the top of `usePointcloudObjects.svelte.ts`: + +```ts +import { applyFilter } from './usePointclouds.svelte' + +export interface ProvidePointcloudObjectsOptions { + filter?: () => ReadonlySet | undefined +} +``` + +- [ ] **Step 2:** Update the signature: + +```ts +export const providePointcloudObjects = ( + partID: () => string, + options: ProvidePointcloudObjectsOptions = {} +) => { + // ... +} +``` + +- [ ] **Step 3:** In its `enabledClients` `$derived.by`, add the filter check (mirror Task 9). Then in **both** spawn blocks (the pointcloud entity around line ~204 and the geometry entity around line ~238), add `traits.PipelineSource(name)` to the entity traits — `name` here is the vision-service name from the enclosing loop. + +- [ ] **Step 4:** `pnpm check && pnpm test -- usePointcloudObjects`. Expected: PASS. + +- [ ] **Step 5:** Commit. + +```bash +git add src/lib/hooks/usePointcloudObjects.svelte.ts +git commit -m "feat(hooks): usePointcloudObjects optional filter + PipelineSource trait" +``` + +--- + +### Task 11: Pure visibility helper + +The visibility effect is the only place that talks to ECS, but the policy it implements (which entities should be `Invisible` given an active set) is pure and deserves a unit test before we wire it. + +**Files:** +- Create: `src/lib/pipeline/visibility.ts` +- Test: `src/lib/pipeline/__tests__/visibility.spec.ts` + +- [ ] **Step 1: Write the failing test.** + +```ts +// src/lib/pipeline/__tests__/visibility.spec.ts +import { describe, expect, it } from 'vitest' + +import { computeVisibility } from '../visibility' + +describe('computeVisibility', () => { + const entities = [ + { id: 1, source: 'wrist-cam-left' }, + { id: 2, source: 'merged-cloud' }, + { id: 3, source: 'object-segmenter' }, + { id: 4, source: 'arm-1' }, // not a stage; not a pipeline source + ] + + it('hides every pipeline-sourced entity not in active', () => { + const result = computeVisibility(entities, new Set(['merged-cloud']), new Set([ + 'wrist-cam-left', + 'merged-cloud', + 'object-segmenter', + ])) + expect(result.hide).toEqual([1, 3]) + expect(result.show).toEqual([2]) + }) + + it('leaves entities whose source is not a stage alone', () => { + const result = computeVisibility(entities, new Set(['merged-cloud']), new Set([ + 'wrist-cam-left', + 'merged-cloud', + 'object-segmenter', + ])) + expect(result.hide).not.toContain(4) + expect(result.show).not.toContain(4) + }) + + it('shows everything when active is empty', () => { + const result = computeVisibility(entities, new Set(), new Set([ + 'wrist-cam-left', + 'merged-cloud', + 'object-segmenter', + ])) + expect(result.hide).toEqual([]) + expect(result.show).toEqual([1, 2, 3]) + }) +}) +``` + +- [ ] **Step 2:** Run `pnpm test -- src/lib/pipeline/__tests__/visibility.spec.ts`. Expected: FAIL. + +- [ ] **Step 3: Write implementation.** + +```ts +// src/lib/pipeline/visibility.ts +import type { StageId } from './types' + +export interface PipelineEntity { + id: E + source: string +} + +export interface VisibilityResult { + show: E[] + hide: E[] +} + +export const computeVisibility = ( + entities: ReadonlyArray>, + active: ReadonlySet, + stageIds: ReadonlySet +): VisibilityResult => { + const show: E[] = [] + const hide: E[] = [] + for (const entity of entities) { + if (!stageIds.has(entity.source)) continue + if (active.size === 0 || active.has(entity.source)) show.push(entity.id) + else hide.push(entity.id) + } + return { show, hide } +} +``` + +Note the empty-active behavior: when no stage is selected, **everything** in the pipeline shows. This matches the user expectation that the page reads as "live monitor for the pipeline" when nothing is solo'd. + +- [ ] **Step 4:** Run the test. Expected: PASS. + +- [ ] **Step 5:** Commit. + +```bash +git add src/lib/pipeline/visibility.ts src/lib/pipeline/__tests__/visibility.spec.ts +git commit -m "feat(pipeline): pure visibility policy helper" +``` + +--- + +### Task 12: Wire visibility effect into the route + +**Files:** +- Modify: `src/routes/pipeline/+page.svelte` + +- [ ] **Step 1:** Add inside the ` + + +``` + +- [ ] **Step 4: Update `PipelineGraph.svelte`** to compute positions and render `EdgeLayer`. Replace the `
    ` block with: + +```svelte + {@const positions = layoutGraph(pipeline.graph())} + {@const orderedStages = [...stages].sort((a, b) => { + const pa = positions.get(a.id)! + const pb = positions.get(b.id)! + return pa.level - pb.level || pa.ordinal - pb.ordinal + })} + +
    + +
      + {#each orderedStages as stage (stage.id)} +
    1. + +
      2. + {/each} +
    +
    +``` + +(Use the same `StageNode` props as before.) + +Add at the top of the script: + +```ts + import { layoutGraph } from './layout' + import EdgeLayer from './EdgeLayer.svelte' +``` + +- [ ] **Step 5:** Run `pnpm test -- src/lib/pipeline/__tests__/PipelineGraph.spec.ts`. Expected: PASS (the previously-passing tests stay green; the new edge test now passes). + +- [ ] **Step 6:** Commit. + +```bash +git add src/lib/pipeline/EdgeLayer.svelte src/lib/pipeline/PipelineGraph.svelte src/lib/pipeline/__tests__/PipelineGraph.spec.ts +git commit -m "feat(pipeline): SVG edge layer with topo-level layout" +``` + +--- + +## Wrap-up + +### Task 18: End-to-end smoke against a live machine + +By this point, Task 7 already consumes the real `partConfig`, Task 14 wires the attribute walker, and Task 17 renders edges. This task is a manual verification gate, not new code. + +- [ ] **Step 1:** `pnpm dev`. Open `http://localhost:5173/pipeline?partID=` against a fixture machine that has at least one transform-camera and one vision service. +- [ ] **Step 2:** Confirm: + - Graph nodes match the machine's cameras + vision services (no arms, motors, sensors). + - Edges from upstream cameras to transform cameras are drawn. + - Edges from cameras to vision services are drawn (when configured via `camera_name`-shaped attributes). + - Click a stage → only that stage's points render in scene with its tint color. + - Shift-click a second stage → both render with distinct tints. + - Toggle "Native colors" → original RGB returns. + - Edit machine config (in another tab) to add a stage; the graph updates without reload. +- [ ] **Step 3:** No commit; this is a manual check. If anything fails, stop and triage; do not advance. + +--- + +### Task 19: Playwright smoke test + +**Files:** +- Create: `e2e/pipeline.test.ts` + +- [ ] **Step 1:** Inspect `e2e/` for an existing test (e.g. `cat e2e/arm.test.ts`) and `cat playwright.config.ts`. Pattern-match the `goto` URL form, fixture setup, and partID handling. + +- [ ] **Step 2: Add the smoke test.** + +```ts +// e2e/pipeline.test.ts +import { expect, test } from '@playwright/test' + +test.describe('/pipeline', () => { + test('renders the pipeline panel and at least one stage node', async ({ page }) => { + // Adjust partID + any auth bootstrap to match e2e/arm.test.ts + await page.goto('/pipeline?partID=test') + await expect(page.getByRole('complementary')).toBeVisible() + await expect(page.getByRole('button', { name: /Pipeline|stage|camera/i }).first()).toBeVisible({ + timeout: 10_000, + }) + }) +}) +``` + +(If the fixture machine has no cameras configured, mark this test as `test.fixme` with a TODO and revisit when a fixture exists.) + +- [ ] **Step 3:** `pnpm test:e2e`. Expected: PASS or `fixme` if no fixture machine. + +- [ ] **Step 4:** Commit. + +```bash +git add e2e/pipeline.test.ts +git commit -m "test(pipeline): playwright smoke for /pipeline route" +``` + +--- + +### Task 20: Final verification + changeset + +**Files:** +- Create: `.changeset/pipeline-view-mvp.md` + +- [ ] **Step 1:** `pnpm check`. Expected: 0 errors. +- [ ] **Step 2:** `pnpm lint`. Expected: clean (or only auto-fixable warnings — apply `pnpm lint --fix` if so). +- [ ] **Step 3:** `pnpm test`. Expected: full suite green. +- [ ] **Step 4:** Add a changeset: + +```markdown +--- +'@viamrobotics/visualization': minor +--- + +Add `/pipeline` route — auto-derives the perception pipeline DAG from the machine +config and spotlights selected stages in the existing 3D scene with per-stage +color tinting. +``` + +(Match the package name and bump kind to repo convention; check `.changeset/` for an existing template.) + +- [ ] **Step 5:** Commit. + +```bash +git add .changeset/pipeline-view-mvp.md +git commit -m "chore(changeset): pipeline view MVP" +``` + +- [ ] **Step 6:** Run `git log --oneline main..HEAD` to confirm the commit graph reads cleanly. + +--- + +## Summary of deferred work + +These remain out-of-scope for this plan and will be addressed in follow-up plans: + +- **2D preview pane** (`StagePreview2D.svelte`, `getImages` integration, detection / classification overlays) — spec phase 4. +- **Override panel + persistence** (`PipelineOverrides.svelte`, `override.ts`, localStorage per `partID`) — spec phase 5. +- `RefreshRates.images` decoupling from `RefreshRates.pointclouds`. +- `getPipelineMetadata` DoCommand convention for custom modules. +- Per-stage `transformPCD` to a common frame for compare-mode alignment (revisit only if alignment problems surface in practice). diff --git a/docs/superpowers/specs/2026-05-08-pipeline-view-design.md b/docs/superpowers/specs/2026-05-08-pipeline-view-design.md new file mode 100644 index 000000000..138655d18 --- /dev/null +++ b/docs/superpowers/specs/2026-05-08-pipeline-view-design.md @@ -0,0 +1,367 @@ +# Pipeline View — Design + +**Status:** Draft +**Date:** 2026-05-08 +**Author:** nick.hehr@viam.com +**Branch:** `feat/pipeline-view` + +## Purpose + +The visualization app currently renders every camera point cloud, every vision-service object cloud, and every framed component in one shared scene with no notion of how they relate. There is no way to trace data from a source camera through transforms and joins to a final segmenter output, and no way to compare the output of two stages of the same pipeline against each other. + +This feature adds a **Pipeline view** at `/pipeline` that: + +- Auto-derives the perception pipeline from the machine config (with a manual override escape hatch). +- Renders the pipeline as a DAG in a left-pane graph panel. +- "Spotlights" the selected stage (or stages) in the existing 3D scene, hiding non-active stages. +- Shows 2D outputs (images, detections, classifications) in a right-pane preview when the active stage is non-3D. +- Tints each active stage in compare mode so multi-stage overlays are visually distinct. + +The user value: identify which stage of a perception pipeline is misbehaving, with the source-to-output relationship made visible, so config tuning has a fast feedback loop against a live machine. + +## Goals + +- **Live monitor flavored.** Stages re-render against fresh data on the existing refresh cadence; a config edit on the live machine flows through. +- **No new "pipeline file" to author.** Auto-derivation from machine config is the default; users only touch override settings when the auto-derivation can't see something. +- **Reuse the existing canvas, ECS world, hooks, and PCD worker.** Pipeline view is a new route, not a fork of the app. +- **Minimal change surface in shared code.** The only modification to existing hooks is an optional resource filter — no breaking changes for the main app. +- **All four output kinds in v1:** point clouds, segmenter object clouds, 2D images (multi-named), detections / classifications. + +## Non-Goals + +- **Editing machine config from the pipeline view.** Config remains a Viam-platform concern; pipeline view is read-and-visualize. +- **Server-side pipeline graph.** Derivation is purely client-side; no Connect-RPC additions for v1. +- **`getPipelineMetadata` DoCommand convention.** Worthwhile follow-up for custom modules but not v1. +- **Multi-part pipelines spanning machines.** A pipeline lives within one part. +- **Per-stage refresh rates.** Existing `RefreshRates` settings apply to all stages of a kind. Per-stage cadence is a follow-up. +- **Snapshot-mode pipeline replay.** The existing `/snapshot` route handles frozen scenes; pipeline view is live. +- **Frame-system `transformPCD` alignment.** Existing entities are placed via parent frames already; revisit only if compare-mode alignment surfaces problems in practice. +- **Replacing or modifying the main app's UX.** Pipeline view is opt-in via a separate route. + +## Decisions + +| # | Decision | Rationale | +|---|---|---| +| P1 | New route at `/pipeline`; trimmed shell (`Settings` + `RefreshRate` overlays kept; tree, dashboard, XR, etc. dropped) | Focused UX for a focused use case; main app unchanged | +| P2 | Auto-derive pipeline graph from machine config + manual override (localStorage per `partID`) | Common case is free; escape hatch for custom modules | +| P3 | Edges discovered by attribute-**value** matching against known resource names (not by attribute-key heuristics) | Robust to varied attribute naming (`src`, `camera`, `cameras`, `service`, `detector`, `vision`, etc.); custom modules covered automatically | +| P4 | Output kinds discovered via runtime `getProperties()` (cached via existing TanStack pattern), not from model-name guessing | Authoritative; aligns with how `usePointclouds` / `usePointcloudObjects` already gate fetching | +| P5 | Spotlight scene paradigm (one shared ``) — graph picks active stages; non-active hidden via `Invisible` trait | Reuses existing entity renderer; no multi-canvas complexity | +| P6 | Per-stage color tint via new `PipelineTint` trait + header toggle to restore native RGB | Compare mode legibility; user-controllable when colors themselves matter | +| P7 | 2D preview in a right-pane `StagePreview2D`, mounted only when the active set contains a 2D-only output | Doesn't take space when not needed | +| P8 | 2D image stages use `getImages()` (multi-named-image) with stream fallback when supported | Aligns with current camera API; reuses existing `Camera.svelte` stream logic where applicable | +| P9 | Optional `filter` parameter added to `providePointclouds` / `providePointcloudObjects`; defaults to undefined | Pipeline route only fetches stages it shows; main app behavior unchanged | +| P10 | `usePipeline` Svelte context owns graph, active set, status, tint, preview-2D-stage; both graph panel and scene read from it | Single source of truth; predictable reactivity | +| P11 | Cycle detection drops the back-edge with a warning on the offending node | Pipeline view stays usable even with broken configs | +| P12 | All persistence (overrides, preview pane width, native-RGB toggle) keyed by `partID` in `localStorage` | Per-machine state without backend changes | + +## Architecture + +### Route shape + +``` +/pipeline +├─ providers shell: provideWorld, provideSettings, provideEnvironment, +│ providePartConfig, providePipeline +├─ +│ ├─ (grid, camera controls, environment lights — existing) +│ └─ (ECS-driven entity rendering — existing) +└─ overlays + ├─ (left, ~280px, resizable; new) + ├─ (right; mounts when preview2D !== undefined; new) + ├─ (existing component) + └─ (existing component, with new "Pipeline overrides" tab) +``` + +The page lives at `src/routes/pipeline/+page.svelte` and is a composition-only file. All meaningful code is library code under `src/lib/pipeline/`. + +### Module map + +``` +src/lib/pipeline/ + types.ts # Stage, Edge, StageKind, StageOutput, PipelineGraph, RGB + derive.ts # buildGraph(config: Struct, properties: PropertiesMap): PipelineGraph + override.ts # mergeOverrides(graph, overrides): PipelineGraph + tint.ts # tintForStageId(id: string): RGB + usePipeline.svelte.ts # context provider: graph, active, preview2D, tint, status, solo/toggle/clear + PipelineGraph.svelte # left-pane DAG renderer (FloatingPanel-style; resizable) + StageNode.svelte # one node: icon, label, status dot, tint chip, eye toggle + EdgeLayer.svelte # SVG edges between StageNodes (computed from layout) + StagePreview2D.svelte # right-pane image / detection / classification renderer + PipelineOverrides.svelte # modal/tab for overrides (visibility, outputs, manual edges) + +src/lib/pipeline/__tests__/ + derive.spec.ts + override.spec.ts + tint.spec.ts + +src/routes/pipeline/ + +page.svelte # composition only + +page.ts # part selection from URL params + +src/lib/hooks/ + usePointclouds.svelte.ts # MODIFIED — accept optional filter + usePointcloudObjects.svelte.ts # MODIFIED — accept optional filter +``` + +### Component responsibilities + +**`derive.ts`** — Pure function from `(config: Struct, properties: PropertiesMap) → PipelineGraph`. + +1. Collect `resourceNames` from `config.components[].name ∪ config.services[].name`. +2. For each component / service whose API is `camera` or `vision`, build a candidate `Stage`. Set `outputs` from the cached `properties` map: cameras → `pointcloud` iff `supportsPcd`, `image` iff `mimeTypes.length > 0`; vision services → union of `{detections, classifications, objects}` flagged by `getProperties`. +3. Stages with no relevant outputs are dropped. +4. For each candidate, recursively walk its `attributes` Struct. Every string-valued leaf and every string array element is checked against `resourceNames`. Each match (other than self) yields a candidate edge `match → candidate`. Dedupe. +5. Topologically sort; if cyclic, drop the latest-discovered back-edge and append a warning. +6. Return `{ stages: Stage[], edges: Edge[], warnings: Warning[] }`. + +**`override.ts`** — Merges user overrides into the auto-derived graph: + +```ts +interface Overrides { + hidden: Set // exclude from graph entirely + outputsByStage: Map // pin outputs (replace discovered) + extraEdges: Edge[] // user-added edges +} +``` + +Validates that referenced stage ids still exist in the derived graph and silently drops stale entries (the override-panel UI surfaces them as "unknown — remove"). + +**`tint.ts`** — `tintForStageId(id) → {r,g,b}`. Hash the id, map to evenly-distributed hue in HSL, fixed S/L. Deterministic so the same stage always gets the same color across reloads. + +**`usePipeline.svelte.ts`** — Context provider, called by `+page.svelte`: + +```ts +interface PipelineContext { + graph(): PipelineGraph + active(): ReadonlySet + preview2D(): StageId | undefined + tint(id: StageId): RGB + status(id: StageId): 'idle' | 'loading' | 'ok' | 'error' + solo(id: StageId): void + toggle(id: StageId): void // shift-click; flips membership in active + clear(): void +} +``` + +Internally, `graph` is `$derived` from `usePartConfig` + per-stage `getProperties` queries + overrides. `active` is `$state` (a `SvelteSet`). `preview2D` is `$derived` from `active`: the most-recently-toggled active stage whose outputs include any of `image | detections | classifications` (and whose outputs do **not** include `pointcloud | objects`, OR whose `pointcloud`/`objects` is hidden via override). `status` aggregates per-stage TanStack queries. + +**`preview2D` worked examples** (also serve as test fixtures): + +| Stage outputs (effective, post-override) | Active set (chronological) | `preview2D()` | +|---|---|---| +| `[pointcloud]` | `{ pcd-stage }` | `undefined` (3D-only; no preview pane) | +| `[image]` | `{ image-stage }` | `image-stage` | +| `[pointcloud, image]` | `{ stage }` | `undefined` (has a 3D output; renders in scene only — does **not** open preview) | +| `[pointcloud, image]` with `pointcloud` overridden off | `{ stage }` | `stage` (now effectively 2D-only) | +| `[detections]` then `[pointcloud]` shift-toggled | `{ det-stage, pcd-stage }` (det first) | `det-stage` (most-recent 2D-only candidate) | +| `[detections]` then another `[image]` | `{ det-stage, image-stage }` | `image-stage` (most-recently toggled wins) | + +**`PipelineGraph.svelte`** — A `FloatingPanel`-shaped left-pane component (matches the existing tree's docking style). Layout computed via simple Sugiyama-style top-down DAG layout (small N — under 30 stages typically — so a hand-rolled layout is fine; no new dep). Renders: + +- One `` per stage with: icon (camera | vision | unknown), label (override or resource name), status dot (color-coded), tint chip (filled when active), eye toggle (independent show/hide for graph-only filtering). +- One SVG `` over the node grid drawing arrows between connected nodes. +- Header with: "⚙ Overrides" button, "Use native colors" toggle, "Clear selection" button. + +Click semantics: +- Plain click on a node → `solo(id)`. +- Shift-click → `toggle(id)`. +- Click on the graph panel's empty area (not on a node, not on the SVG edge layer) → `clear()`. The Threlte canvas is not a clear target — orbiting/clicking the scene leaves the active set untouched. + +**`StagePreview2D.svelte`** — Mounts iff `preview2D() !== undefined`. Reads the stage's outputs and renders accordingly: + +- For `image` stages: `getImages()` on `RefreshRates.pointclouds` cadence (a small misnomer, but reuses the existing rate; introducing `RefreshRates.images` is a follow-up). Returns `NamedImage[]`; rendered as labeled tiles in a vertical stack. If the camera's `mimeTypes` indicate a streamable format and the user has a stream-capable connection, fall back to the existing `StreamClient` approach (extracted from `Camera.svelte` into a shared utility). **UX caveat:** because images and pointclouds share a refresh rate in v1, users with a slow `pointclouds` setting (e.g. 5s) will see equally slow image updates; users with a fast setting (e.g. 100ms) will spend bandwidth they may not want on `getImages` polls. Calling this out so it isn't a planning surprise — the stream fallback mitigates the bandwidth side when available. +- For `detections` stages: call `getDetectionsFromCamera(cameraName)` against the upstream camera (resolved from the DAG: the most-recently-active upstream camera in the active set; if none active, the configured `camera_name`). Render bounding boxes overlaid on that camera's current image (also via `getImages` for that camera). +- For `classifications` stages: `getClassificationsFromCamera(cameraName)`; render a ranked label list above the source image. +- For `objects`: not 2D — never the preview-2D candidate; rendered as 3D in the scene like today. + +Pane width is resizable via a drag handle; persisted to settings. + +**`PipelineOverrides.svelte`** — A new tab inside the existing `Settings` component. Three sections: + +1. **Visibility** — list of all stages with checkboxes. +2. **Outputs** — per-stage chip selector for `pointcloud | image | objects | detections | classifications`. Pre-checked from discovered outputs; unchecking forces an output off in this pipeline. +3. **Manual edges** — list of user-added `from → to` pairs with delete buttons; "Add edge" form with two pickers populated from the resource name set. + +A "Reset to derived" button clears all overrides for the current `partID`. + +### Hook modifications + +```ts +// existing +export const providePointclouds = (partID: () => string) => { ... } + +// modified +export interface PointcloudsOptions { + filter?: () => Set | undefined +} +export const providePointclouds = (partID: () => string, options: PointcloudsOptions = {}) => { + // inside enabledClients: + // if (options.filter && !options.filter().has(client.current.name)) continue + // rest unchanged +} +``` + +`providePointcloudObjects` gets the same shape. Existing call sites pass no options (no behavior change). + +The pipeline route's provider call is: + +```ts +providePointclouds(() => partID, { filter: () => pipelineResourceNames() }) +providePointcloudObjects(() => partID, { filter: () => pipelineResourceNames() }) +``` + +`pipelineResourceNames()` is a derived getter: the set of stage ids in the merged graph minus those hidden via override. + +### Scene rendering — filter and tint + +Two reactive effects in `+page.svelte`: + +1. **Visibility effect.** Watches `active()` and the world's entities. For each entity with a `Name` trait that matches a stage id: if the stage is in `active`, ensure the entity has no `Invisible` trait; otherwise add `Invisible`. Entities whose `Name` is not a stage id (e.g. arms, geometries) are left alone — the pipeline view doesn't hide non-pipeline entities by default. (Alternative considered: hide everything not in `active`. Rejected: drops the framed components that contextualize the pointclouds. The user can use the existing tree in the main app for that view.) +2. **Tint effect.** Watches `active()` + the "use native colors" setting. For each active entity, when native-colors is OFF: ensure a `PipelineTint(tint(id))` trait is present and the geometry's `color` attribute (if any) is removed; the existing `Color` trait reading then drives material color. When the entity becomes inactive or the toggle flips ON: remove the trait and let the original color render again. (The `color` attribute removal is one-way in Three.js — once deleted, we'd need to re-parse to restore. Plan: store the original color attribute on the entity in a `OriginalColors` trait at first apply, restore on tint removal. The existing PCD worker output already gives us the bytes to re-create the attribute without a re-fetch.) + + **Scope of the tint effect.** Applies only to entities that carry the `Points` trait (pointcloud + segmenter object clouds). Mesh/geometry entities (frames, arms, custom geometries) are not pipeline stages and are not tinted. This keeps the `Color`/`Colors`/`OriginalColors` interplay confined to the one consumer that actually needs it (the `Points` renderer); the existing `Color`-trait usage on meshes is unchanged. + +### Data flow + +``` +machine config (Struct) + │ + └─► usePartConfig ─┐ + │ +per-stage getProperties (TanStack, staleTime ∞) ─┤ + │ +overrides (localStorage, partID) ─┤ + ▼ + buildGraph + mergeOverrides → PipelineGraph + │ + ▼ + usePipeline context + ├─► PipelineGraph.svelte (renders) + ├─► usePointclouds.filter + ├─► usePointcloudObjects.filter + ├─► visibility effect (Invisible trait) + ├─► tint effect (PipelineTint trait) + └─► StagePreview2D (when 2D active) +``` + +## Pipeline schema (in-memory; not persisted as JSON) + +```ts +type StageId = string // resource name + +type StageOutput = + | 'pointcloud' + | 'image' + | 'objects' + | 'detections' + | 'classifications' + +interface Stage { + id: StageId + label: string + api: 'camera' | 'vision' + model: string + outputs: StageOutput[] + derivedFrom: 'config' | 'override' +} + +interface Edge { + from: StageId + to: StageId + derivedFrom: 'config' | 'override' +} + +interface Warning { + stageId: StageId | null + message: string +} + +interface PipelineGraph { + stages: Stage[] + edges: Edge[] + warnings: Warning[] +} +``` + +## Layout + +``` +┌─ header: part selector · refresh · "Pipeline overrides" ──────────────┐ +├─ graph (left, ~280px) ─┬─ scene (center, flex-1) ──┬─ preview-2d ────┤ +│ ▼ wrist-cam-left ● │ │ named-img: color│ +│ │ │ [3D point clouds] │ [tile] │ +│ ▼ wrist-l-cropped ● │ │ │ +│ │ │ [optional 2nd cloud │ named-img: depth│ +│ ▼ merged-cloud ◑ │ in compare mode, │ [tile] │ +│ │ │ different tint] │ │ +│ ▼ object-segmenter ◐ │ │ (only mounted │ +│ │ │ when preview2D │ +│ ⚙ Overrides │ │ is active) │ +└─────────────────────────┴────────────────────────────┴─────────────────┘ +``` + +## Error handling + +| Failure | Surface | Recovery | +|---|---|---| +| `getProperties` failure for a stage | Red status dot + tooltip; outputs default to "unknown" until override pins them | User opens overrides → set outputs manually | +| Pointcloud / image fetch failure | Stage status = error; scene/preview retains last-good or shows "no data yet" | Re-fetch on next refresh tick | +| Cyclic edges | Back-edge dropped; warning annotation on the receiving node | User uses overrides → "Hide edge" / "Add edge" to fix | +| Resource referenced in attributes but absent from resources list | Edge silently ignored; reported in `warnings[]` shown in overrides panel | User edits machine config or accepts | +| Detection stage's input camera not active | Preview pane warning: "input camera not active — graph lineage broken" | User shift-clicks the input camera into the active set | +| Stale override (references resource that no longer exists) | Override panel shows it as "unknown — remove" with a one-click remove | User clicks remove | + +## Testing + +**Unit (Vitest)** + +- `derive.ts`: + - Linear chain (camera → transform → segmenter) + - Fan-in (multiple cameras → join) + - Fan-out (one camera → multiple services) + - Unknown model with attribute references + - Cyclic config — verifies back-edge dropped + warning emitted + - Attribute-key variations (`src`, `source`, `camera`, `cameras`, `source_cameras`, `service`, `detector`, `vision`) + - Self-reference filtered + - Reference to non-existent resource → warning, no edge + - **Noise rejection:** an attributes Struct contains a string field whose value coincidentally matches a resource name but is not a dependency (e.g. a `description` or `label` field) — verifies a single edge is recorded if the value appears, and serves as a documented limitation; the override panel's "delete edge" handles real-world false positives +- `override.ts`: + - Override wins over derived + - Stale override (referenced stage gone) emits a warning, doesn't crash + - Reset-to-derived clears all +- `tint.ts`: + - Deterministic per-id + - Reasonable hue distribution for N=20 + +**Component (Vitest + @testing-library/svelte)** + +- `PipelineGraph.svelte`: click → solo, shift-click → toggle, eye toggle → independent visibility. +- `StagePreview2D.svelte`: image stage renders tiles; detection stage renders bboxes given a fixture image + detection list. + +**E2E (Playwright)** + +- Open `/pipeline` against the existing fixture machine; assert the graph renders with expected nodes; solo a stage; assert the scene contains only that stage's points (count entities in the world that don't have `Invisible`). + +## Open Questions + +None blocking — implementation can begin. + +## Out-of-scope follow-ups + +1. `getPipelineMetadata` DoCommand convention so custom modules can declare lineage explicitly. +2. Per-stage `RefreshRates` (e.g. images at 100ms, pointclouds at 5s). +3. `transformPCD` to a common frame for compare mode if alignment problems surface. +4. Snapshot integration: capturing a pipeline frame and replaying it via `/snapshot`. +5. Server-side pipeline graph if a future backend wants to drive pipeline UX from a service. +6. Multi-part pipelines spanning machines. + +## Phased rollout + +1. **Phase 1 — Skeleton.** Route, providers shell, `derive.ts` + tests, `usePipeline`, `PipelineGraph` rendering nodes only. Unit tests green. +2. **Phase 2 — Spotlight scene.** Hook `filter` plumbing, `Invisible` + `PipelineTint` reactive effects, solo / toggle, header tint switch. Pointcloud + objects end-to-end. +3. **Phase 3 — Edges + cycles.** Attribute-value walker, edge layer in graph, cycle detection. +4. **Phase 4 — 2D preview.** `getImages` + stream fallback for image stages; detections / classifications overlays. +5. **Phase 5 — Override panel.** Visibility, outputs pinning, manual edges, persistence. + +Each phase is independently testable and demoable. diff --git a/src/lib/hooks/__tests__/reconcileWorldState.spec.ts b/src/lib/hooks/__tests__/reconcileWorldState.spec.ts new file mode 100644 index 000000000..d5cda6f11 --- /dev/null +++ b/src/lib/hooks/__tests__/reconcileWorldState.spec.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from 'vitest' + +import { reconcileWorldState } from '$lib/hooks/reconcileWorldState' + +describe('reconcileWorldState', () => { + it.each<{ + name: string + snapshot: Iterable + rendered: Iterable + toAdd: string[] + toRemove: string[] + }>([ + { + name: 'adds snapshot-only UUIDs and removes rendered-only UUIDs, keeping the intersection', + snapshot: ['B', 'C'], + rendered: ['A', 'B'], + toAdd: ['C'], + toRemove: ['A'], + }, + { + name: 'adds everything on initial mount (nothing rendered yet)', + snapshot: ['A', 'B'], + rendered: [], + toAdd: ['A', 'B'], + toRemove: [], + }, + { + name: 'is a no-op when snapshot and rendered match', + snapshot: ['A'], + rendered: ['A'], + toAdd: [], + toRemove: [], + }, + { + name: 'removes everything when the snapshot is empty', + snapshot: [], + rendered: ['A', 'B'], + toAdd: [], + toRemove: ['A', 'B'], + }, + { + name: 'accepts any iterable and de-duplicates the snapshot in toAdd', + snapshot: new Set(['A', 'B']), + rendered: new Set(['A']), + toAdd: ['B'], + toRemove: [], + }, + { + name: 'preserves snapshot iteration order in toAdd', + snapshot: ['C', 'A', 'B'], + rendered: [], + toAdd: ['C', 'A', 'B'], + toRemove: [], + }, + ])('$name', ({ snapshot, rendered, toAdd, toRemove }) => { + const result = reconcileWorldState(snapshot, rendered) + expect(result.toAdd).toEqual(toAdd) + expect(result.toRemove).toEqual(toRemove) + }) +}) diff --git a/src/lib/hooks/reconcileWorldState.ts b/src/lib/hooks/reconcileWorldState.ts new file mode 100644 index 000000000..ebd76d6c5 --- /dev/null +++ b/src/lib/hooks/reconcileWorldState.ts @@ -0,0 +1,33 @@ +export interface WorldStateReconciliation { + /** UUIDs present in the fresh snapshot but not yet rendered. */ + toAdd: string[] + /** UUIDs currently rendered but absent from the fresh snapshot. */ + toRemove: string[] +} + +/** + * Diffs a fresh world-state snapshot against the currently rendered entity set. + * Pure: callers own spawning/destroying and tombstone bookkeeping. Preserves + * snapshot iteration order in `toAdd` and rendered iteration order in `toRemove`. + * Duplicate UUIDs within either input collapse to a single entry, so a + * duplicated snapshot UUID appears at most once in `toAdd`. + */ +export const reconcileWorldState = ( + snapshotUUIDs: Iterable, + renderedUUIDs: Iterable +): WorldStateReconciliation => { + const snapshot = new Set(snapshotUUIDs) + const rendered = new Set(renderedUUIDs) + + const toAdd: string[] = [] + for (const uuid of snapshot) { + if (!rendered.has(uuid)) toAdd.push(uuid) + } + + const toRemove: string[] = [] + for (const uuid of rendered) { + if (!snapshot.has(uuid)) toRemove.push(uuid) + } + + return { toAdd, toRemove } +} diff --git a/src/lib/hooks/useWorldState.svelte.ts b/src/lib/hooks/useWorldState.svelte.ts index 8caff5b97..d40cdeb5f 100644 --- a/src/lib/hooks/useWorldState.svelte.ts +++ b/src/lib/hooks/useWorldState.svelte.ts @@ -11,8 +11,10 @@ import { import { createResourceClient, createResourceQuery, + useMachineStatus, useResourceNames, } from '@viamrobotics/svelte-sdk' +import { untrack } from 'svelte' import { Matrix4 } from 'three' import { asFloat32Array, inMeters } from '$lib/buffer' @@ -20,6 +22,7 @@ import { createChunkLoader, type EntityChunk } from '$lib/chunking' import { drawTransform, updateMetadata } from '$lib/draw' import { hierarchy, traits, useWorld } from '$lib/ecs' import { isPointCloud } from '$lib/geometry' +import { reconcileWorldState } from '$lib/hooks/reconcileWorldState' import { metadataFromStruct } from '$lib/metadata' import { createPose, poseToMatrix } from '$lib/transform' @@ -30,8 +33,12 @@ type TransformEvent = TransformChangeEvent & { transform: TransformWithUUID } +const UNRECONCILED = Symbol('unreconciled') + export const provideWorldStates = () => { const partID = usePartID() + const machineStatus = useMachineStatus(() => partID.current) + const revision = $derived(machineStatus.current?.config?.revision) const resourceNames = useResourceNames(() => partID.current, 'world_state_store') const clients = $derived( resourceNames.current.map(({ name }) => @@ -47,7 +54,7 @@ export const provideWorldStates = () => { const cleanups: (() => void)[] = [] for (const client of clients) { - cleanups.push(createWorldState(client)) + cleanups.push(createWorldState(client, () => revision)) } return () => { @@ -127,7 +134,10 @@ const decodeWorldStateChunk = (response: unknown, fallbackStart: number): Entity return { start, positions, colors, opacities, done } } -const createWorldState = (client: { current: WorldStateStoreClient | undefined }) => { +const createWorldState = ( + client: { current: WorldStateStoreClient | undefined }, + revision: () => string | undefined +) => { const { invalidate } = useThrelte() const world = useWorld() const relationships = useRelationships() @@ -253,7 +263,10 @@ const createWorldState = (client: { current: WorldStateStoreClient | undefined } } } - let initialized = false + // Tracks which config revision the snapshot has been reconciled against, so the + // snapshot effect runs once per revision (initial mount + each reconfigure) rather + // than on every incidental query settle. + let reconciledRevision: string | undefined | typeof UNRECONCILED = UNRECONCILED let flushScheduled = false let rafId = 0 let pendingEvents: TransformEvent[] = [] @@ -270,6 +283,24 @@ const createWorldState = (client: { current: WorldStateStoreClient | undefined } }) ) + // Force a fresh snapshot on reconfigure. The createResourceQuery key omits the + // config revision, so a same-name rebuild would otherwise be served stale cache. + // This effect owns the revision edge (reads it tracked); the reconcile effect + // below deliberately reads revision untracked so it reacts to the snapshot + // settling instead of racing this refetch. + // + // A rare mount ordering (snapshot settles before machineStatus loads) can cause + // one redundant, idempotent refetch on the first real revision; accepted over a + // guard that would risk skipping a genuine reconfigure. + $effect(() => { + const rev = revision() + if (rev === undefined) return + if (reconciledRevision === UNRECONCILED) return // initial mount handled below + untrack(() => { + void listUUIDs.refetch() + }) + }) + const applyEvents = (events: TransformEvent[]) => { for (const event of events) { if (event.changeType === TransformChangeType.ADDED) { @@ -302,19 +333,40 @@ const createWorldState = (client: { current: WorldStateStoreClient | undefined } $effect(() => { if (!getTransformQueries) return - if (initialized) return if (getTransformQueries.some((query) => query?.isLoading)) return + // Read revision untracked: the refetch effect owns the revision edge and kicks + // off listUUIDs.refetch(); this effect must fire on the FRESH snapshot settling + // (getTransformQueries recomputing), not on the revision flip — otherwise it + // reconciles against the pre-refetch snapshot and latches the revision, blocking + // the real snapshot when it lands. + const rev = untrack(() => revision()) + if (reconciledRevision === rev) return + const transforms = getTransformQueries .flatMap((query) => query?.data) .filter((transform) => transform !== undefined) - for (const transform of transforms) { + const byUUID = new Map(transforms.map((t) => [t.uuidString, t])) + const { toAdd, toRemove } = reconcileWorldState(byUUID.keys(), entities.keys()) + + for (const uuid of toRemove) { + destroyEntity(uuid) + } + + for (const uuid of toAdd) { + const transform = byUUID.get(uuid) + if (!transform) continue + // The fresh snapshot is authoritative for presence: a UUID it reports as + // present must not stay tombstoned, or spawnEntity would refuse it. Note this + // can override a newer stream REMOVED for the same UUID (snapshot wins) — an + // accepted MVP boundary; see the design doc's snapshot-vs-stream race note. + removedUUIDs.delete(uuid) spawnEntity(transform) } invalidate() - initialized = true + reconciledRevision = rev }) /** @@ -343,6 +395,12 @@ const createWorldState = (client: { current: WorldStateStoreClient | undefined } } $effect(() => { + // Re-subscribe on a config-revision change: an AlwaysRebuild reconfigure + // swaps the backing resource instance under the same name, ending the old + // gRPC stream cleanly. `client.current` is a stable reference so this effect + // would not otherwise re-fire; reading `revision()` gives it that dependency. + revision() + if (!client.current) return const controller = new AbortController()