analyst: shape-collapse retry with a conformance record

[drewstone]

Ask

Give Analyst runs a shape-collapse retry: when an analyst emits rows but the schema accepts none of them (emitted > 0, accepted === 0), re-sample up to N times and report a conformance record, instead of returning an empty finding list that reads the same as "nothing to find".

Where this comes from

tangle-network/blueprint-agent upgraded agent-eval 0.145.14 -> 0.182.0 (ticket tangle-network/blueprint-agent#2463) with the rule "replace a local duplicate with the upstream export only after a parity test passes". The local wrapper withShapeCollapseRetry in scripts/experiments/lib/analyst-kinds/index.ts was checked against 0.182.0 and no upstream equivalent was found, so it stays local.

What was checked on 0.182.0

• @tangle-network/agent-eval/analyst and the root barrel: no export re-samples an analyst on a whole-array schema collapse, and no conformance record type exists (grep -iE "collapse|conformance|resample" over dist/analyst/index.d.ts and dist/index.d.ts finds only the campaign resamples bootstrap option).
• Nearest mechanisms, none of which cover this: withJudgeRetry (transport retry with model fallback for a judge call; it does not see emitted-versus-accepted counts), createChatClient({ maximumAttempts }) (transport attempts), createLlmCorrectnessChecker(chat, { maxAttempts }) (re-asks on malformed structured output for the correctness checker only), TraceAnalystDefinition.requireStructuredFindings and minimumEvidenceCitations (declarative, no attempt count), and the 0.161.0 bridge rule that a findings submission cannot become a silent empty result (validation, not a retry).

The local behaviour to match

withShapeCollapseRetry(base: Analyst<TInput>, options?: { maxRetries?: number; onRetry?; onConformance? }): Analyst<TInput>

• Tees ctx.log to read the factory's { emitted, accepted } fields: the only signal that separates "the model picked a bad shape" from "there was nothing to find".
• Re-samples only while findings.length === 0 && emitted > 0, up to maxRetries (default 3, NaN and negatives fall back to the default so a bad value cannot disable retries). A genuine empty (emitted === 0) passes through untouched.
• Emits one AnalystConformanceRecord per analyze() after retries settle: { analystId, attempts, emittedPeak, emittedFinal, accepted, collapsed, recovered, conformanceRate } with conformanceRate = accepted / emittedFinal (1 when nothing was emitted).

Parity test that cannot run today

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'
// Does not exist on 0.182.0:
import { withShapeCollapseRetry, type AnalystConformanceRecord } from '@tangle-network/agent-eval/analyst'

describe('analyst shape-collapse retry', () => {
  it('re-samples a collapsed response and reports recovery', async () => {
    let calls = 0
    const base = {
      id: 'probe',
      async analyze(_input: unknown, ctx: { log?: (m: string, f?: Record<string, unknown>) => void }) {
        calls++
        ctx.log?.('rows', { emitted: 3, accepted: calls === 1 ? 0 : 3 })
        return calls === 1 ? [] : [{ subject: 'a' }, { subject: 'b' }, { subject: 'c' }]
      },
    }
    const records: AnalystConformanceRecord[] = []
    const wrapped = withShapeCollapseRetry(base as never, { maxRetries: 3, onConformance: (r) => records.push(r) })
    const findings = await wrapped.analyze({}, { log: () => {} } as never)
    assert.equal(findings.length, 3)
    assert.equal(calls, 2)
    assert.deepEqual(records, [{ analystId: 'probe', attempts: 2, emittedPeak: 3, emittedFinal: 3, accepted: 3, collapsed: true, recovered: true, conformanceRate: 1 }])
  })
  it('passes a genuine empty through without re-sampling', async () => {
    let calls = 0
    const base = { id: 'probe', async analyze(_i: unknown, ctx: { log?: (m: string, f?: Record<string, unknown>) => void }) { calls++; ctx.log?.('rows', { emitted: 0, accepted: 0 }); return [] } }
    const wrapped = withShapeCollapseRetry(base as never, { maxRetries: 3 })
    assert.deepEqual(await wrapped.analyze({}, { log: () => {} } as never), [])
    assert.equal(calls, 1)
  })
})

The full local implementation is at scripts/experiments/lib/analyst-kinds/index.ts (withShapeCollapseRetry, AnalystConformanceRecord, ShapeRetryOptions) in blueprint-agent; happy to send it as a PR if the shape is accepted.