v3.10.0 — recipe append blocks, /api/status sidecar health - #53
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e cap Heading, fence markers and the truncation note were assembled after the byte budget was spent on the row body alone, so a long title could push the emitted block (and the document) past BLOCK_BYTE_CAP/DOC_BYTE_CAP. Reserve the worst-case chrome size before serializing rows and hand only the remainder to serializeRows; skip the block if that leaves less than MIN_USEFUL_BUDGET for the body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The old finish() wrote the append block into result.markdown before
extractWeb() ran renderDecision(), which measures result.markdown
length. A large enough append block could push a thin static
extraction past the 500-char floor and silently skip the Playwright
re-render on exactly the SPA pages this feature targets.
finish() now stashes { markdown, notes } on result.append instead of
attaching. A new attachAppendBlock() helper does the attaching
(markdown, contentLength, extractorReason - same as before) and is
called only once the render decision is settled: on the no-render
path, on the rendered path (after the existing extractorReason
assignment, so the render note survives instead of being overwritten),
on the sidecar-failure path, and in extractHtml() (which never goes
through renderDecision at all).
Fixes seven findings from the whole-branch review of the recipe `append` field (structured-data extraction from embedded JSON state blobs into a fenced JSON block at the end of the document): 1. lib/web.js: rewrote the stale comment above the append-block render call, which claimed finish() attaches the block on every return path. It only stashes it on result.append; the callers attach it via attachAppendBlock() once the render decision is settled. 2. lib/recipes.js: append-block `title` now rejects line breaks via a regex, so a newline in the title can no longer open the code fence early and leave the JSON body unfenced. 3. lib/append-data.js: resolveSegments() now guards the string-segment branch with Object.prototype.hasOwnProperty and rejects string segments against arrays, so inherited properties (constructor, __proto__) and exotic array properties (length) no longer resolve. 4. lib/append-data.js: the kept === 0 skip note (a single row exceeds the block's byte cap) now says so explicitly, instead of reusing the "Budget erschöpft" wording that belongs to the genuine document- budget-exhaustion case a few lines above. 5. lib/append-data.js: the row-count note and truncation text now use the German singular "Zeile" when exactly one row is kept, via a small rowNoun() helper, instead of always saying "1 Zeilen". 6. test/recipes-append.test.js: added a test for the comments=true return path (extractWeb + injected fetch + render: 'skip'), which extractHtml can never reach since it hardcodes comments = false. 7. CHANGELOG.md: extended the HTML comment under [Unreleased] to note that the reference-link list at the end of the file also needs a matching entry when the release is cut. Test growth: 1223 -> 1229 passing (+6), 288 -> 289 suites (+1). All green, no other output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Playwright sidecar on the production instance crash-looped for eight weeks without anyone noticing. Nothing was broken from the outside: the service kept answering 200, extraction just silently fell back to empty results for every page that needs a browser render. Add an endpoint that makes that state observable. It probes the /health route of each configured sidecar in parallel, answers 200 when they all respond and 503 as soon as one is down, so a plain HTTP monitor catches it without keyword matching. An unconfigured sidecar is not a failure - running without Playwright or markitdown is a supported setup. Error reasons stay a closed vocabulary (unreachable / timeout / unhealthy / http <code> / misconfigured) because the endpoint is public and a raw error message would carry the internal sidecar hostname. Results are cached for 5s and concurrent callers share one round of probes, so polling cannot amplify into internal traffic. Also pin playwright-stealth, the one unpinned sidecar dependency: a rebuild could otherwise have pulled an incompatible release unnoticed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018W84fQHmxwu4S519Zd4Vde
Two features in this batch: recipe append blocks, which lift numeric data out of SSR state blobs where no CSS selector can reach it, and GET /api/status, which makes a dead extraction sidecar visible instead of letting it degrade the output in silence. The 3.x list in README and help page names only the status endpoint. The append field stays deliberately undocumented outside the changelog: it is an internal recipe escape hatch, not a contributor-facing feature, so SITE-RECIPES.md does not mention it either. node --test: 1251 pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018W84fQHmxwu4S519Zd4Vde
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v3.10.0 — recipe append blocks, sidecar health endpoint
Two independent features that accumulated on top of 3.9.0.
append: pull numeric data out of SSR state blobsSingle-page frameworks serialize their initial state into a
<script>block (ng-state,__NEXT_DATA__, …) and then render it as a chart. The numbers are in the HTML, but no CSS selector reaches them, so extraction returns prose and drops the data the page is actually about.The new optional recipe field
appendnames a script selector and a path into the JSON, optionally projects a subset of fields and caps the row count, and appends the result as a fenced JSON code block at the end of the document. Output size is capped, and the block is attached on every extraction path — but only after the render decision, so it cannot influence which extractor wins.Deliberately not documented in
SITE-RECIPES.md: this is an internal escape hatch for instance-local recipes, not a contributor-facing feature.GET /api/status: make a dead sidecar visibleThe Playwright sidecar on the production instance crash-looped for eight weeks without anyone noticing. Nothing looked broken from outside — the service kept answering
200, extraction just silently fell back to near-empty results for every page that needs a browser render.{"ok":false,"version":"3.10.0","services":{ "trafilatura":{"status":"ok","latencyMs":3}, "playwright":{"status":"down","error":"unreachable"}, "markitdown":{"status":"not-configured"}}}/healthin parallel;200when they all answer,503as soon as one is down, so a plain HTTP monitor catches it without keyword matching.not-configured, not a failure — running without Playwright or markitdown stays a supported setup.unreachable,timeout,unhealthy,http <code>,misconfigured). The endpoint is public and a raw error message would carry the internal sidecar hostname; a test asserts that.Also pins
playwright-stealth, the one unpinned sidecar dependency, to the version already running in production.Verification
node --test: 1251 pass, 0 fail.markitdown-sidecar/test_limits.py+test_youtube.py: 13 pass.test/status.test.jsandtest/status-endpoint.test.js./api/statusreturns200with all three sidecars healthy; the503path verified against a real server instance with unreachable sidecar URLs.🤖 Generated with Claude Code
https://claude.ai/code/session_018W84fQHmxwu4S519Zd4Vde