Skip to content

Fix escaped origin identities, endpoint probes and relay error propagation - #80

Merged
Coldwings merged 12 commits into
data-accelerator:mainfrom
Coldwings:fix/origin-identity-and-relay-errors
Oct 12, 2026
Merged

Coldwings merged 12 commits into
data-accelerator:mainfrom
Coldwings:fix/origin-identity-and-relay-errors

Conversation

@Coldwings

@Coldwings Coldwings commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #77
Fixes #78
Fixes #79

Distinct escaped origin paths could return another object's cached bytes, endpoint-specific Range metadata could leak across origins serving the same digest, and a failed chunked peer relay could complete as HTTP 200 without a read error.

Changes

  • Preserve escaped origin paths and split digest segments before unescaping. Encode field separators in malformed fallback identities instead of deleting them.
  • Scope size/Range metadata and probe singleflight to the origin endpoint. Renewed signatures still reuse metadata, and immutable digest blocks still deduplicate across endpoints. Custom fetchers can implement fetch.Prober; nested wrappers preserve that interface on shared probes and credential-refusal retries.
  • Abort committed peer responses with http.ErrAbortHandler when sources fail; propagate known relay-length mismatches and never cache partial or invalid blocks. Quarantine the actual faulty parent so later requests can fall back to origin.
  • Decline inconsistent block key/Origin URL pairs before cache lookup, routing or origin access, preserving honest parent circuits.
  • Bind peer transport and validation outcomes to the breaker entry/generation at admission. Validate relay geometry before recording success; stale completions cannot undo quarantine or consume newer probe slots. Half-open recovery waits for all admitted siblings, and a failure vetoes an earlier success. Roster discovery retains admission bypass without owning data-probe slots.
  • Update package contracts, regression-test descriptions, measured coverage and CHANGELOG; add ADR-0004 after searching all existing ADRs, with acceptance taking effect when this PR merges.

Validation

With Go 1.27.0, the main module passes:

go vet ./...
go test ./... -race -count=1
go test ./... -cover -count=1
go build ./...
DART_GOLDEN_REF=1 go test ./internal/chunk ./internal/hashring ./internal/cluster -count=1
bash scripts/check-docs.sh

Combined statement coverage is 92.4%; chunk 99.1%, fetch 92.4%, engine 88.4%, peer 90.5%, registry 90.5%. Formatting and diff checks pass. All eight documentation checks pass, with no advisory notes.

The separate providers/k8s module passes vet, race tests, coverage tests and build (provider package 81.8% coverage). The main module remains dependency-free.

Regression tests copied to current upstream main reproduce all three defects: distinct equal-length escaped objects collide, both probe orders mix capabilities/unknown totals, a stalled endpoint blocks an independent probe, and a chunked source failure returns no read error. They pass on this branch. Independent review also reproduced a nested-wrapper probe composition defect; regressions fail before delegation is fixed and pass after it, including ten consecutive race runs. Real HTTP tests cover incomplete Content-Length failure framing, multi-hop truncation, quarantine/fallback, and requester identity consistency; deterministic concurrent tests cover stale completions and half-open sibling outcomes. A source error after the entire advertised Content-Length has already been flushed cannot be signaled by truncation; fallible streaming sources must omit the sizer and use chunked framing.

Compatibility and rollout

Digest identities, ChunkKey construction, HRW and independent golden vectors are unchanged. Affected non-digest URL identities now retain escaped paths; upgrade all nodes together when serving encoded separators, reserved characters or non-ASCII paths. Cache stores restart cold; do not carry old URL-keyed entries through an external warm-cache layer. Each new endpoint performs its own one-byte capability probe, even when its digest blocks are cached elsewhere.

Review history

The earlier PR was incorrectly opened against the fork and is closed: Coldwings/dart#1. Its four Copilot findings were fixed with regression tests and resolved; those older reviews do not establish approval of this PR's current head. This PR targets data-accelerator/dart:main from Coldwings:fix/origin-identity-and-relay-errors and uses the upstream issues above.

Checklist

  • go vet ./... / go test ./... -race -count=1 / go test ./... -cover -count=1 all pass (AGENTS.md)
  • docs/<pkg>.md updated for behavior/contract changes; no new package
  • bash scripts/check-docs.sh passes
  • ADR: 0004 — escaped origin identity correction and endpoint/content separation (T3/T4/T5)

MingzhouDR and others added 10 commits October 12, 2026 04:20
Keep encoded path boundaries and separators distinct so different immutable origin objects cannot reuse write-once blocks. Split digest segments before unescaping and document coordinated upgrades for affected URL identities. Add literal identity vectors and real cache-isolation regressions without changing ChunkKey or HRW.
Keep endpoint size and Range support separate from digest-addressed content. Add a dedicated coalescing probe group so concurrent origins cannot share metadata flights, while refreshed signatures still coalesce and cached blocks remain shared across registries. Cover both probe orders, unknown totals, independent stalled probes and cross-origin cache reuse.
Prevent net/http from ending a failed chunked stream as a successful response. Abort after committed source errors, propagate known relay geometry mismatches, and retain clean early failures. Cover both response framing modes and interruption across a real intermediate engine; update protocol behavior and upgrade notes.
Charge definitive block protocol failures to the parent circuit rather than leaving it healthy after a clean HTTP EOF. A soft failure would be reset by every later transport success, so quarantine the invalid parent immediately and retain normal cooldown recovery. Add a real repeated-request regression proving origin fallback and cache reuse. Address Copilot feedback on PR #1; refs #4.
Prevent pre-quarantine transport completions from clearing newer circuits or consuming half-open probe slots. Capture entry identity and generation for block and roster requests, then check and record their outcome under one lock. Validate relay geometry before recording success through StreamValidated, using the bounded request context. Add deterministic concurrent regressions for delayed Get/Stream/roster success, stale failure after recovery and validator context bounds. Address the second Copilot review on PR #1; refs #4.
Derive the expected chunk key from the supplied Origin URL, namespace and block index before cache lookup or upstream I/O. A malformed requester must not blame an honest parent for an unrelated object geometry. Preserve cache-only reads without Origin and decline caller addressing errors cleanly. Cover buffered/streaming and warm/cold relays, wrong URL/chunk/namespace pairs, healthy circuits and valid control reads. Address the third Copilot review on PR #1; refs #4.
…fix/origin-identity-and-relay-errors

# Conflicts:
#	docs/fetch.md
#	docs/peer.md
Record upstream issue references and Go 1.27 coverage. Clarify that malformed URLs are rejected before metadata caching and roster success cannot consume outstanding data probes.

@Coldwings Coldwings left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review: no remaining blocking findings in the current PR diff.

Reviewed escaped path/digest segmentation, field-separator exclusion, endpoint metadata and probe/content singleflight separation, requester key/Origin consistency before cache access, response commitment/error handling, and breaker entry/generation lifetime across Get, Stream and roster requests. Half-open recovery keeps admitted siblings live until all finish, with failure veto and separate roster slot ownership. Cut-through forwarding and digest block reuse remain intact; trusted immutable origins and uniform cluster configuration remain explicit preconditions.

Real HTTP regressions reproduce all three defects on current upstream main and pass on this branch. Reviewed the four historical Copilot concerns against their fixes and deterministic regressions; older fork reviews do not substitute for review of this head. Original hash constructions and independent golden vectors are unchanged, and the Python reference checks pass.

Main-module vet, race tests, coverage tests and build pass with Go 1.27.0 (92.5% total coverage). Separate K8s-module vet, race tests, coverage tests and build pass. Formatting, diff checks and all eight documentation checks pass with no advisory notes. Package contracts, CHANGELOG and proposed ADR-0004 describe the coordinated upgrade and cold-cache requirement for affected non-digest URLs.

Residual validation scope: local real-HTTP tests establish transport/caching behavior, not Kubernetes end-to-end throughput at 100 Gbps. Remote CI and Copilot review are being checked separately.

@Coldwings
Coldwings requested a balanced review from Copilot October 12, 2026 06:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Coldwings

Copy link
Copy Markdown
Collaborator Author

Independent review of the current PR revision found one blocking API composition defect and one documentation correction.

  • Correctness/concurrency review: no blocking findings in generation-scoped outcomes, half-open sibling handling, roster slot ownership, key/Origin consistency or cut-through failure propagation. Focused regressions passed normally and with the race detector.
  • API/contracts/tests review: Coalescing.Probe currently calls an inner Fetcher.Fetch even when the inner fetcher implements Prober. Nested content-keyed wrappers can therefore mix endpoint metadata again. A deterministic regression reproduced that failure. Both the shared request and credential-refusal retry must preserve inner Probe delegation. docs/chunk.md also still mentions the removed stripUnitSep helper.
  • Residual scope: failure-first cohort ordering, permit eviction/reuse and cancellation during cohort drain have less direct coverage. Once an entire advertised Content-Length has been flushed, a later source error cannot be signaled by truncation; production cut-through relays use chunked framing and preserve the partial-response error signal.

I will fix the composition defect and documentation, rerun the required checks, and obtain both independent reviews on the updated head before considering merge.

Keep underlying Prober semantics for shared probes and renewed-credential retries so nested content-keyed wrappers cannot merge endpoint capabilities. Add regressions and clarify escaped separator and HTTP stream framing contracts.
@Coldwings

Copy link
Copy Markdown
Collaborator Author

Addressed both findings from the first independent review in fix(fetch): preserve nested endpoint probe delegation.

The shared request and credential-refusal retry now use the same operation callback. For probes, that callback delegates to an inner Prober when present, retaining endpoint isolation through nested content-keyed wrappers. The new nested-wrapper and renewed-credential regressions fail before this fix and pass after it; the probe regressions passed ten consecutive race runs.

Corrected the obsolete separator documentation and clarified the completed Content-Length boundary: a later source error cannot be expressed by truncation once all advertised bytes are already sent, so fallible streaming sources must omit the sizer.

Main-module and separate K8s-module vet/race/coverage/build checks pass, along with the independent Python goldens, formatting/diff checks and all eight documentation checks. Both independent reviewers are rechecking the updated head; remote CI is rerunning.

@Coldwings Coldwings left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the updated revision: no remaining blocking findings in the implementation review.

The nested-wrapper defect identified by independent review is fixed in both the shared probe and the renewed-credential retry; the same operation callback preserves inner Prober semantics. Both new regressions fail before the fix, and the probe suite passes ten race runs after it. Reviewed cancellation, bounded flight lifetime, refusal retry, and unchanged content-fetch semantics. Corrected stale separator documentation and narrowed the HTTP error-signal contract to incomplete Content-Length or unfinished chunked framing.

Full main-module vet/race/coverage/build checks pass (Go 1.27.0, 92.4% combined coverage); separate K8s-module vet/race/coverage/build, independent Python hash/golden checks, formatting/diff and all eight docs checks pass. Independent reviews and CI are being checked against this updated head before merge.

@Coldwings

Copy link
Copy Markdown
Collaborator Author

Independent re-review of the updated PR revision: no blocking findings from either reviewer.

  • Correctness/concurrency: inner Prober dispatch is preserved for both shared requests and renewed-credential retries; each caller retains its own URL and retry context. Escaped identities, write-once cache admission, relay aborts, generation-scoped quarantine, half-open sibling failure veto and roster slot ownership remain correct.
  • API/contracts/docs/tests: the nested-wrapper defect and stale separator explanation are resolved. The completed Content-Length limitation is explicit; production cut-through relays use chunked framing. The complete diff remains consistent with trusted immutable origins, endpoint metadata isolation and shared digest blocks. ADR-0004 and the coordinated upgrade guidance cover the identity correction.
  • Independent focused race tests across chunk, fetch, engine and peer passed, including composition/retry regressions; documentation governance passed. The full main-module and K8s-module verification remains recorded separately.

Residual limits: custom fetchers without Prober must preserve endpoint isolation themselves; unresolved geometry deliberately prevents caching rather than promising validation. Failure-first cohort ordering, permit eviction/reuse and cancellation during cohort recovery have less direct test coverage, with no defect found by static review. Local tests do not establish Kubernetes end-to-end throughput at 100 Gbps.

Mark ADR-0004 accepted in the merge-bound revision so the decision and its implementation enter main together after review.
@Coldwings

Copy link
Copy Markdown
Collaborator Author

Final independent review: both reviewers report no blocking objections for the merge-bound revision.

The final increment only changes ADR-0004 and its index from proposed to accepted, so the decision takes effect together with its implementation when this PR merges. Source and tests are identical to the previously independently reviewed revision; both reviewers verified the exact final diff and retained their no-blocker verdicts. All earlier findings are addressed. The documented validation limits remain unchanged.

Final self-review also confirms that the ADR status/index agree and all eight local documentation checks pass. The final revision's CI must finish successfully before merge.

@Coldwings Coldwings left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final self-review: no blocking findings. Verified that the merge-bound ADR status and index both say accepted, and all eight local documentation checks pass. Source/tests/modules are unchanged from the fully validated preceding revision. Both independent reviewers separately verified this exact final head and retain their no-blocker verdicts. Remote CI is checked separately before merge.

@Coldwings
Coldwings merged commit b9332ad into data-accelerator:main Oct 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants