Repository navigation
Release 0.5.1: guarded integrations and package readiness - #33
Merged
Merged
Conversation
A live run against local Ollama (qwen2.5:7b) showed the usage scanner reporting usage_disposition=malformed with provider_usage=null, suppressing real counts. Two causes, both invisible to the offline fixtures: - Ollama's streaming /api/chat returns one JSON object per line, not SSE and not a single JSON document, so the whole-body parse failed. A failed whole-body parse now retries line by line; a body where no line parses as an object stays malformed, so a broken document cannot read as absent. - Ollama reports counts as top-level prompt_eval_count/eval_count instead of a nested usage object. Those are accepted as the prompt/completion equivalent, matching the existing input_tokens/output_tokens aliases. Verified live: streaming now reports prompt_tokens=37, completion_tokens=5, and a non-streaming turn returns the correct answer with a coherent provider_delta_tokens=15 (278 - 263). Forwarded bodies stay byte-identical. Also reflows two rustfmt hunks in server.rs that were not formatted.
Two defects made a correct rejection look like a network fault: - A request refused for conflicting framing was answered without reading its request body. Closing a socket that still holds unread inbound bytes makes the peer reset the connection, which discards the 400 the proxy had already written. Clients could not tell a refusal from a network error, and the CL.TE/TE.CL smuggling tests failed intermittently because of it (verified: the proxy logged 400 while the client read zero bytes, WSAECONNRESET 10054). Refused bodies are now drained first, bounded by max_body_bytes. The same applied to a 413, where read_body stops early with the tail still unread. - main.rs printed the requested bind address rather than the bound one, so a caller using --bind 127.0.0.1:0 to let the kernel pick a port was told the proxy was on port 0. The integration harness did exactly that, which was the only way to allocate a port that cannot be stolen between choosing and binding it. Test harness: raw_request no longer discards a partial read along with the connection-reset error, and a new regression test asserts the refusal is deliverable. It fails at the pre-fix code (0/10 delivered) and passes after. Windows still discards an occasional already-written response, so the test requires a clear majority rather than a perfect score, and says why.
EP-03's offline half: a paired-run record, a fail-closed reader, and the aggregation the contract fixes in advance, so a quality claim has to be counted rather than asserted. - eval/run_paired.py owns a versioned record (task and environment-snapshot digest, arm, model/policy revision, seed/attempt, outcome, evidence, and the optional EP-02 usage measurement), refuses any other schema version, and reports all four paired outcomes, the conditional contrastive regression rate, the absolute success delta and a task-clustered interval. - Unavailable is unavailable: with no raw successes the CFR has no denominator, so it is null and fails the gate rather than reading as 0.0. An arm without a partner, a duplicated arm, a snapshot mismatch and an invalid environment run are each reported, never dropped and never counted as a model failure. - Five fixtures in eval/tasks/paired/ cover a required numeric threshold, negation, a later use of an earlier id, two dependent tool calls, and a needed pruned row. The first drafts of these could not fail: json_prune reported 34 candidates and dropped 0 because its 32-token margin exceeds what a short row costs, so every arm was byte-identical. They were re-sized until pruning engages, and a test now asserts markers exist. - Retrieval is resolved before scoring, so a recoverable drop is not scored as data loss, while a marker the store cannot resolve stays absent. - Sandbox confinement rejected only Path.is_absolute(), which is False for a driveless rooted path on Windows; it now rejects any path with an anchor. - Scoring is a deterministic dummy model. This certifies the record contract and the arithmetic; it is not evidence about any real model's reasoning and cannot promote a lossy default. No live or paid run was performed.
An explicitly enabled OpenAI Chat Completions adapter that compresses only the eligible tool-result strings, never the provider envelope. - tokenfold_adapters::observation owns the policy (off by default), the route gate (only /v1/chat/completions), and the pass itself. A parallel result group is all-or-nothing, and a group whose tool_call_ids do not exactly match the preceding tool_calls is skipped as a malformed transcript rather than half-rewritten. - Three checks gate emission, each catching what the others cannot: restoring the originals must reproduce the baseline exactly, the serialized bytes must match so key order cannot drift, and each replacement must decode back to the original inner value. Envelope parity and a smaller assembled body follow. Anything short of that returns the untouched baseline with a reason. - The inner check compares decoded values, not text: json_field_fold emits a columnar frame, so a string comparison would have rejected every correct result while appearing to work. - The proxy activates this only when the route is owned AND the flag is set, so the flag can only add behavior. An earlier version gated on the route alone and silently removed the existing whole-body compression on /v1/chat/completions whenever the feature was off; four existing tests caught it. - An observation attempt is labelled observation:1.0 in its measurement event with byte counts, because that path produces no CompressionReport and would otherwise be indistinguishable from a passthrough. 28 adapter tests and 28 proxy tests pass; workspace, fmt and clippy are clean. No live provider call, pruning, retrieval, or model dependency.
…tic strategy Four optional, disabled-by-default adapter increments, each isolated in its own module and each carrying only the guarantees it can actually keep. EP-07/NF-03 dedup: an exact duplicate-observation codec. Keeps the first inline and references the rest. Exact by design -- near-repeats are left alone, because a similarity matcher would need a similarity definition and guessing at one silently rewrites literal content. Emitted only if it round-trips byte-for-byte and is genuinely smaller after marker overhead. Dangling references are rejected rather than guessed, so a body this codec did not write fails loudly instead of coming back with a hole in it. Host text that merely resembles a marker is never reinterpreted. EP-08/NF-04 manifest: lifts caller-declared keys out of late observations. Nothing is inferred -- free text is never mined for field names and an absent key gets no placeholder. Every fact carries provenance. Conflicting observations of one key are both kept and counted, because resolving them would be inventing state. Secret-shaped values are dropped and counted, reusing the pipeline's own detector. Bounds are applied last and every drop is reported. EP-12/NF-20 tool catalog: forced tools (named by tool_choice) and caller-declared companions always survive and are exempt from the size limit. Kept tools are copied byte-for-byte so schema constraints are never rewritten. A forced or companion tool absent from the catalog is reported, never fabricated. EP-13/NF-22 semantic strategy: runs behind a trait and is treated as untrusted output. Invention is detectable because a Summary carries attributed Claims, and each claim's text and source must occur in the request -- a free-form summary cannot be verified, an attributed one can. Every failure falls back to the byte-identical baseline, and the reason is always carried so a degradation is visible rather than looking like a strategy that saved nothing.
…he receipt reader EP-09/NF-14 evidence search: a BM25 index over approved retrieval-store entries that deliberately holds no state of its own, so restart and resync are the same operation and there is no persistent index to go stale. Storage is not consent -- only caller-listed hashes are published. Namespace isolation is structural: built for exactly one namespace with no API to widen it. Expiry is honoured twice, at build time and again per query, so a slow rebuild cannot resurrect content whose TTL has elapsed. Hits name a stored hash and serve the live bytes. Reuses the existing BM25 index; adds no model or vector dependency. This adds a workspace-in- path dependency on tokenfold-core, which is why Cargo.lock changes. EP-10/NF-11+NF-12 quality profiles: a tuner that searches for aggressive settings will find them, so every candidate must clear a QualityFloor before it is even eligible. A below-floor candidate is rejected outright rather than demoted to a worse-but-valid option, because a floor that can be traded away is not a floor. Malformed candidates are reported as malformed, not as low quality. Search is bounded, deterministic and auditable, and deliberately does not break quality ties on cost. Approval requires a named approver and records the floor it was approved against. Rollback is first-class. policy_hash covers name and knobs only, so a policy can be identified independent of any single run. EP-11/NF-18 holdout: the arm is derived from session identity alone via a hash, so it is stable across turns and restarts; re-rolling per turn would put a subject in both arms inside one conversation and destroy any prefix cache. The shaping guard refuses anything that would drop a protected path or fall below a floor, and has no partial result. Negative deltas are reported, not suppressed. EP-02 follow-up (the item deliberately left open): add a versioned receipt reader. The shipped v1 receipt fixture did not actually parse -- v1 named the preset 'mode' and predates output_encoding and pruning -- so archived receipts were unreadable. parse_versioned normalizes v1 (renames and documented absences only, never an invented measurement) and refuses an unknown schema_version outright, since best-effort parsing a future contract is how an absent value becomes a plausible zero. parse_report exposes it to the Python binding, with tests that the v1 fixture still reads and that absent sections stay absent rather than becoming numbers. Cargo.lock: records the new workspace-internal path dependency only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation