fix(storage): bound Blazegraph queries server-side, and refuse truncated CONSTRUCT results

[branarakic]

Summary

Aborting a query client-side does not stop it. The adapter puts a 30s AbortController on every store request
but sends no server-side deadline, so when the client gives up Blazegraph keeps executing and allocating.

This is the root cause behind the two error classes that dominate Base mainnet publish failures
(store scheduler queue wait timeout (normal: blazegraph.query) ×151 and
blazegraph operation exceeded its 30000ms deadline ×123, together ~86% of the counted publish errors).

Measured on the fleet

/bigdata/status?showQueries=details on a Base core, container up with no restart in the window:

observation	value
abandoned queries in flight	steady 2–5
longest observed elapsed	32.5 min
same query re-sampled 576s later	elapsed +617s — i.e. still executing
queryErrorCount over 20,953 accepted queries	0 — the server never cancelled anything
Blazegraph CPU	307–670% of 400% available

Thread dumps showed 9 RUNNABLE com.bigdata.journal.Journal.executorService* threads in query operator frames while
all 10 Tomcat http-nio-8080-exec threads were idle — nobody was waiting on those queries. They are pure waste,
and a handful of them saturate a 4-CPU core.

The fix

Send X-BIGDATA-MAX-QUERY-MILLIS on both read paths (SELECT/ASK and CONSTRUCT). The deployed 2.1.x engine honours
it (BigdataRDFContext$AbstractQueryTask reads it via getQueryTimeout).

Why the bound is wider than the client deadline, not tighter

This is the design decision worth reviewing. SERVER_QUERY_DEADLINE_FACTOR = 4, so the 30s default yields a 120s
server bound.

1. Bounding abandoned work doesn't need a tight bound. Any finite value converts "unbounded" into "at most 2
   minutes". The orphan pool is what saturates the core, and the pool size is a function of arrival rate × lifetime
   — cutting lifetime from 32 min to 2 min is a ~16× reduction.

2. A tight bound introduces silent data loss. Blazegraph commits 200 OK and begins streaming before it knows
   the query will finish, then appends its error into the already-committed body. A CONSTRUCT killed mid-stream
   while the client is still reading yields a short but structurally valid n-quads document — and
   parseNQuadsText does if (!match) continue;, silently discarding the truncated tail and the appended error.
   The caller gets a sync page that looks complete and is not.

   Keeping the server bound comfortably above the client deadline means the client always aborts first, so no
   caller is ever parsing a truncated body. The failure stays loud.

This is the one substantive difference from the equivalent slice in #1817, which computes
max(5000, remaining - 2000). That inverts the invariant exactly where it matters most: under the queue pressure
that produces the queue wait timeout errors, a query is often admitted with near-zero remaining budget, gets
granted the 5000ms floor, and the server then outlives the client — reaching the truncation path.

Belt-and-braces: assertCompleteNQuads

For the residual case (an operator-set global web.xml queryTimeout, or an engine error mid-result), CONSTRUCT
bodies are checked for truncation before parsing.

Deliberately narrow — it detects truncation, not any unparseable line — so it cannot start rejecting the
odd-but-harmless serialisations the tolerant parser has always accepted. It fires on exactly two signals:

• the body carries an appended Java exception marker
• the final statement does not end in .

Testing

Six new cases in packages/storage/test/blazegraph.unit.test.ts:

• header present on SELECT and CONSTRUCT, and asserted > the client deadline
• scales with a custom client timeout
• rejects a body truncated by an appended engine error
• rejects a body cut mid-statement
• still accepts complete bodies, including empty results and comment lines (the anti-regression case)

Mutation-verified: SERVER_QUERY_DEADLINE_FACTOR = 1 fails the width assertion; removing assertCompleteNQuads
fails both truncation tests.

Full storage suite: 477 passed, 0 failures (32 files). tsc --noEmit clean.

Scope / risk

• Reads only. getQueryTimeout is referenced only from AbstractQueryTask — not from UpdateTask — so the
  header is inert on sparqlUpdate. Long INSERT/replaceGraph/deleteByPattern work remains unbounded server-side;
  that is a separate change and is not claimed here.
• No behaviour change on the success path: a query that completes within the client budget is unaffected.
• Reversible: the factor is a single exported constant.

Related

• fix(cli): mount the Blazegraph journal on a named volume, and never guess its name #2093 — journal volume / recreate safety (independent, no overlap)
• fix(store): blazegraph store survivability — bounded heap, journal volume, server-side query timeout, runtime health monitor #1817 — the larger survivability PR; this is the narrow adapter slice, with the deadline-direction correction above
• Investigate recurring SWM store scheduler queue wait timeouts on mainnet #1990 — the store-scheduler timeout symptom this addresses upstream of

[Bojan131]

Independently verified this on a real Blazegraph rig while working on a duplicate attempt (#2095, now closed in favour of this one). Posting the measurements as supporting evidence, plus one correction.

Rig: pinned islandora/blazegraph:6.4.3@sha256:015e308a…, namespace properties copied from scripts/devnet.sh start_blazegraph, 3,000 quads + pad graph, expensive per-solution SHA512 chain. Every measurement drives the real built BlazegraphStore; residual server work read from /bigdata/status?showQueries. Two query shapes: streaming, and buffering (ORDER BY) — the buffering shape is the real orphan case, since the server never touches the socket and so never learns the client hung up.

Residual server work after the client gave up (8 s budget, 4 runs/state):

path	canary	this PR
SELECT buffering	76,406–78,933 ms	24,154–24,290 ms (clamped to 32 s − 8 s, ±0.29 s)
CONSTRUCT buffering	74,038–93,142 ms	24,082–24,283 ms
CONSTRUCT streaming	24,081–27,543 ms	24,091–24,237 ms

The wider-bound decision is correct, and I have the failure it avoids. I independently built the tight-bound variant (remaining − 10%) before reading your rationale, and it reproduces exactly what you predicted: on the same server, same budget, a CONSTRUCT whose true answer is 3,000 quads resolved as a success with 64–93 quads, 4/4 runs, no error thrown. On the small dataset it instead 500s, because <32 KB had been emitted so the response was never committed — the loss only appears once the body commits early. This PR: 32/32 CONSTRUCT runs threw cleanly, zero partial resolves.

assertCompleteNQuads — one gap, not currently reachable. Driving the real store against crafted bodies:

body	this PR	canary
complete (control)	accepted 5	accepted 5
mid-statement cut	rejected	accepted 3
clean line-boundary cut, no marker	accepted 3 of 5	accepted 3
partial + appended Java trace	rejected	accepted 3
literal legitimately containing a Java trace	accepted (no false positive — the line-anchoring in b95b391a4 works)	accepted
real Blazegraph-truncated body captured off the server (200 OK, 103,498 bytes, Java marker, final line mid-statement)	rejected	silently accepted 74 quads

So a clean-boundary cut with no marker would slip through in principle, but every real truncation this image produces trips both signals. Worth a comment rather than a change.

One correction — the UPDATE rationale is wrong on this image, though the code is safe. You write that the header "is read only by AbstractQueryTask, not UpdateTask". Measured on islandora/blazegraph:6.4.3 (build 2.1.5-SNAPSHOT), same INSERT…WHERE:

• without the header → HTTP 200 after 88,809 ms, COMMIT: mutationCount=3000, 3,000 quads committed
• with X-BIGDATA-MAX-QUERY-MILLIS: 5000 → HTTP 500 at 5,006 ms, 0 quads committed — killed and rolled back mid-flight (reproduced twice)

This PR is unaffected because it doesn't send the header on sparqlUpdate, and I confirmed that in the built dist — ordinary UPDATE traffic is byte-identical and clean (6× DROP + 200-quad INSERT + COUNT, all OK, zero 500s). But the stated reason would mislead whoever adds it next: on this engine the header does reach the update path, and it aborts the transaction rather than merely the wait.

Suite on this branch: 478 passed / 26 skipped.

[otReviewAgent]

💡 Suggestion: Keep the new CONSTRUCT tests from turning the adapter suite into a near-1k-line catch-all

Why it matters
The file does not cross 1k lines yet, but this PR consumes most of the remaining space with feature-specific cases and repeated setup. A small decomposition now would keep the tests easier to scan and leave room for future adapter coverage without normalizing file sprawl.

Suggestion
Extract a small helper or table-drive the CONSTRUCT truncation cases, and consider moving this focused response-integrity suite into its own test file if more Blazegraph read-path cases are expected. Preserve the exact scenarios; the goal is to reduce scaffolding and keep the main adapter suite below the 1k-line pressure point.