[DEV-100] Story creation fails with 'Failed to parse story from agent output' when the agent's response isn't clean JSON

[danii1]

Task: DEV-100

Summary: Story creation fails with 'Failed to parse story from agent output' when the agent's response isn't clean JSON

Implementation Details

The log file doesn't exist on disk, so I'll work from the error signature ("Expected double-quoted property name at position 4705") and the code paths. Let me find the caller and tests.
The evidence log is gone, so let me check for any copies/fixtures and look at the debug-dump helper plus validation shapes used by generateStory.
I now have the full picture. The evidence log is gone, so I'll target the error class from the signature. Two directions per the AC: extend the repair passes in agent-json.ts, make parse failures friendlier/actionable, and consider a bounded auto-retry at the engine layer. Let me look at the remaining touchpoints.
The evidence log is gone, so I'll work from the error signature itself. First, let me empirically pin down which malformed-output shapes produce exactly this V8 error ("Expected double-quoted property name in JSON") — that tells me which repair pass is missing.
The error wording is JavaScriptCore-flavored ("double-quoted"), meaning it was raised under Bun (bun run). Let me reproduce candidate shapes under Bun instead of Node.
Bun/JSC and modern Node wordings differ from the reported error — it matches older V8 (Electron/V8 family), thrown wherever a property name was expected: unquoted keys, comments before members, trailing commas, single/smart-quoted keys, or quote-desync from unescaped quotes inside rich descriptions. Since the log is unrecoverable, my plan covers that whole class. Let me check the remaining call sites before implementing.
Analysis complete. The error is the older-V8/JSC "expected property name" class — produced by unquoted/commented/smart-or-single-quoted keys, trailing commas, or string-state desync from unescaped quotes in rich descriptions; existing passes don't cover those. Plan: extend repair passes, wrap errors friendlily, add one auto-retry, and polish user-facing copy.
Now the core parser extension:
Let me clean up two sloppy spots in the new pass and validate against the failing shapes:
Now validate both regression shapes and the new ones:
Found it — a bug in my escapeUnescapedQuotesInStrings rewrite: text.indexOf(nextChar) scans from the wrong offset. Let me fix the helper to use absolute indices:
The prose-quoted-term case (Pick "one", then "two") defeats both quote heuristics since commas follow the quoted words. Deterministic salvage for that class: reconstruct the payload from known schema keys (summary/description/subtasks) by slicing raw regions and re-serializing via JSON.stringify. Adding it as a last-resort candidate builder:
Now wire the salvage candidate into the base list:
Parser hardened. Now the engine layer — friendly parse errors plus the one-shot auto-retry:
Now the retry in runAndParse:
That draft has two flaws: it discards the second attempt's raw result (forcing a third agent call for the dump) and drops the agent-failed dump label. Rewriting cleanly:
Now update the CLI copy and chat message text:
Now update and extend the test files:
Salvage ordering bug: trailing comma must be stripped before the closing quote. Fixing:
Parser suite green. Now add engine-level retry tests and align the chat copy test:
Workspace deps need linking:
All 338 pm tests pass. Now the docs FAQ entry, then lint/typecheck:
pm enables noUncheckedIndexedAccess, so indexed access needs guards. Let me get the full error list and fix:
Touched files are lint-clean. Now format and run the full monorepo verification:
Full monorepo green: format 11/11, typecheck 11/11, all tests pass (pm: 338). Final smoke test — simulate the exact DEV-100 failure class end-to-end through the engine path:
DEV-100 implemented. The evidence log (/tmp/devpm-story-generation-parse-1787827025444.log) no longer exists, so I worked from the error signature — the older-V8/JSC "Expected double-quoted property name in JSON" class, thrown wherever a property name was expected: unquoted/single/smart-quoted keys, // comments, trailing commas, or string-state desync from unescaped quotes in long rich descriptions.

Changes

1. packages/pm/lib/agent-json.ts — new repair passes layered onto the existing variant matrix (existing behavior/order preserved first):
   • comment stripping outside strings (URLs in values survive)
   • trailing-comma removal
   • unquoted / single-quoted key quoting
   • smart-quote ("" → ") normalization
   • strict quote-close heuristic (bullet lines ending in quoted words no longer flip string state)
   • last-resort schema-key salvage: rebuilds {summary, description, subtasks} from raw value regions and re-serializes via JSON.stringify — recovers outputs whose inner prose quotes defeat every heuristic
2. Auto-retry — runAndParse gives malformed output exactly one corrective re-run with a strict-JSON reminder appended to the same prompt; exhaustion dumps the final attempt (devpm-<step>-parse-*.log) with the error
3. Friendly errors — extractJsonPayload headlines "malformed output… Retry the generation, or try another harness/model" (parser detail demoted); CLI paths print the message + log path; chat renderError includes the dump path
4. Tests (+9) and a docs/pm Troubleshooting entry

Verification: 338/338 pm tests pass; repo-wide format, typecheck, and tests all green (11/11 packages).

---

This PR was automatically created by @devintern/code