Skip to content

feat(dashboard): raise the hand-written page cap to 3 MiB - #20790

Merged
chenmingwei23 merged 2 commits into
mainfrom
feat/20634-dashboard-page-cap
Oct 16, 2026
Merged

chenmingwei23 merged 2 commits into
mainfrom
feat/20634-dashboard-page-cap

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A hand-written dashboard page of up to 3 MiB can be saved, previewed, applied and drawn.

Issue #20634 asks for a 3 MiB page cap, with the frame and package caps raised to match. A dashboard package refuses a page (view.html) over 128 KiB, and the frame refuses any page over 1,000,000 bytes. A page that inlines a client-side library, such as a 3D renderer, together with its own scene code is several hundred KB to a few MB of layout with no data in it. Pages like that are common, and today none of them can be saved as a package. For example, the unminified three.js r170 build, its OrbitControls, Chart.js 4 and D3 7, with a 3 KB scene, is 1,848,313 bytes, and the same build with TensorFlow.js 4.22 instead is 2,830,086 bytes: over the old frame cap and the old package cap, under the new one.

Why it matters

Anyone who writes their own dashboard page meets this cap as soon as the page inlines a library. As a package the page cannot be saved at all. As a template page it stops working at 1,000,000 bytes.

Not a goal

What changed (motivation -> approach -> change)

A hand-written page of up to 3 MiB can be saved, previewed, applied and drawn. The caps now bound only what a page costs to store, send and draw, because size cannot tell an inlined library from values baked into a layout.

Limit Before After
Frame page (MAX_PAGE_BYTES in dashboard_frame.py) 1,000,000 bytes 3 MiB
Package page (MAX_VIEW_HTML_BYTES in artifact_store/dashboard_package.py) 128 KiB 3 MiB, the frame's own number
Stored package (MAX_PACKAGE_BYTES, same file) 256 KiB 2 x page cap + 128 KiB
Sandbox document stash (_MAX_BYTES in dashboard/handlers/sandbox_doc.py) 4 MiB 8 MiB

A page that passes the page check also passes the package check. The stored package is JSON, which writes a quote or a line break as two bytes, and real minified libraries grow 1 to 7.5% that way, so the package cap is 2 x the page cap + 128 KiB.

The frame cap measures the page, before the host adds about 17 KB of bootstrap and the data island. The sandbox-document stash (dashboard/handlers/sandbox_doc.py) holds the whole document, so it grows from 4 MiB to 8 MiB: room for a page at the cap and a 4 MiB data island, 64 fields of 64 KiB. The gateway's 60 MiB request cap and the artifact store's 25 MiB content cap already fit, and so does the preview tool's 64 KiB html argument (src/kiro_crew/validation.py), which refuses every page whatever its size.

The old refusal said a large page "usually has values baked into it". Size does not show that, so the message now says to inline less and keep values in the fold path. The checks that keep values out are unchanged: every field names its source, every binding is a declared field, the controller pushes only declared fields, and value-shaped keys are refused.

Three scans that were harmless at the old caps now see pages 24 times larger.

  • The member dashboard route (dashboard/handlers/member_dashboard.py) ran the credential scrub and the framing check on the event loop. At 3 MiB the scrub takes between 0.1 s and 1.7 s, by page shape, so both now run in a worker thread.
  • The no-script binding fill (fill_bindings_as_text in dashboard_frame.py) re-read text in three ways: from every < to the end of a page with no > after it, from every - of a tag name such as a-a-a-, and from every repeat of the binding attribute in one start tag. Each is quadratic and holds the GIL: 16 to 31 KiB took one to two seconds. Each start tag is now read once, and a 3 MiB page of any of them takes under 0.1 s.
  • The credential scrub (security/redaction.py, with the new security/pem_scan.py) is changed here because the new cap lets a page of legal size stop it. A page that repeats a PEM private-key header with no END marker after it sent the scan to the end of the page once per header: 64 KiB took 2 s, 256 KiB took 40 s, and a page at the cap would stop the gateway for hours. The scan now looks up the last END marker once and matches a header past it without the full-block tail, which cannot succeed there. Every match is unchanged; 3 MiB of repeated headers takes about 0.4 s.

Backwards compatibility

Compatible: all four limits go up, so every page, package and document accepted today is still accepted. A page over the new cap is refused with the same error, which now names the new limit.

The credential scan returns the same matches. The alternation text is unchanged, and in a differential run the new scan found the same matches as the previous one on 270,000 random texts built from PEM headers, END markers, newlines and other credentials.

The no-script binding fill changes in two edge cases. It no longer reads a start tag that contains a literal <; such a cell keeps the author's own text. And a hyphenated tag name is read whole, so <my-el ...> is closed only by </my-el>, never by </my>, as a browser reads it. All 134 cells in the 14 shipped and fixture pages that carry bindings fill the same as before, and so do 40,000 random well-formed pages.

The package cap can be raised later but not lowered while packages stored under the larger one remain, because the reader applies it too.

The frame cap was pinned equal to another tool's own default; #20634 asks the frame to follow the page cap, so that a page that saves also draws, and test_the_page_cap_is_the_frames_own pins that equality in the pin's place.

The display battery's rules version changes, because security/pem_scan.py is one of its rule sources.

Reader: dashboard_frame.py:fill_bindings_as_text -- crew page, a package page served with scripts withheld -- its one caller is render_package_page; ordinary cells still fill (test_dashboard_package_html_view.py, class TestAScriptsOffPageStillShowsItsValues).

Tests

  • test_dashboard_frame.py: the cap is 3 MiB. A page exactly at it is accepted, and one byte over is refused with the cap in the message.
  • test_dashboard_package_html_view.py: a package whose page is exactly the cap saves, and one byte over is refused naming the cap. A page of quotes at the cap, the 2x worst case, still saves. A stored package exactly at MAX_PACKAGE_BYTES saves and one byte more is refused. The page cap equals the frame's. The no-script fill finishes a page at the cap built from each of five quadratic shapes (unclosed tags, unclosed bound tags, a hyphenated tag name, one tag repeating the attribute, and that with a wrong close tag), under a hang guard and with no time asserted. A literal < in a start tag leaves the cell as written, and a hyphenated tag name is closed only by its own end tag.
  • test_dashboard_templates_builtin.py: a template page exactly at the shared ceiling passes the check that preview, apply and rollback run, and one byte more is refused naming it.
  • test_member_dashboard_package_route.py: a package page at the cap is served through the frame, and the scrub and framing run off the event loop.
  • test_member_dashboard_routes.py: a template page at the cap previews, applies and draws through the real routes, with its scrub off the loop.
  • test_sandbox_doc_channel.py: a document built from a page at the cap plus a 4 MiB data island (64 fields of 64 KiB), which is over the old 4 MiB, is minted and served back unchanged.
  • test_artifact_dashboard_package.py: the stored-form test sets its own cap just under its layout, because the real cap is now sized for a library page. Small pages and every older case are covered by the existing tests, which pass unchanged.
  • test_redaction_pem_scan_linear.py: 26 PEM shapes (full and encrypted blocks, unterminated and nested headers, several blocks, newlines spelled LF, CRLF or a lone carriage return, overlapping markers) keep the spans and the redacted text they had. 30,000 random texts match the plain search loop, and the last-END-marker lookup matches a scan of every offset. A text of 8,000 headers with no END marker is matched without once reaching the full-block tail, which is a count and not a timing, and 3 MiB of repeated headers is redacted under a hang guard.
  • test_security_gate_liveness.py: the security package line budget now covers the 81 new lines, the new module and the choice between the two patterns. test_redaction_jwt_scan_linear.py passes the scan state its helper now takes.

Manual verification

Not run in a browser. The page document was built with the dashboard's own buildSrcdoc in a Node DOM over a 3 MiB page: 3.17 MB in about 0.5 s. To see it draw, put a page of about 3 MiB in the panel-templates folder, preview it and press Keep.

Related Issues

Fixes #20634

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/pem_scan.py

  • src/kiro_crew/security/redaction.py

  • @chenmingwei23 posted a complete reasoning template for fa3d217.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision fa3d2170524e6f761c7b722cd32c394c5a8248c9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of fa3d2170524e6f761c7b722cd32c394c5a8248c9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Every member-dashboard read now re-scrubs and re-frames a page that can be 3 MiB, taking 0.1–1.7 s of GIL-bound regex time. Moving that work to a thread does not take it off the GIL.

Watch

  • Per-read cost grows with the cap, and the thread does not isolate it. The diff wraps the scrub and the framing in await _run(lambda: _template_text_safe(...)) and render_package_page. re holds the GIL while it runs, and the description says so itself: "holds the GIL … which the _run offload cannot help." So a few tabs polling one large page still slow the event loop, and nothing caches the scrubbed page for a version that has not changed.
    Clears when: the scrubbed page is cached by package version or content hash, or scrubbed once at save. Or a measurement shows loop latency stays bounded under concurrent reads of a 3 MiB page.
  • The description takes credit for a fill fix this diff does not contain. It says fill_bindings_as_text "re-read text in three ways… Each start tag is now read once." It also lists that fix's literal-< and hyphenated-tag changes under Backwards compatibility. But this diff does not touch _SIMPLE_BINDING_CELL: the possessive, atomic form is already on base 392a42e6. Only the at-cap tests are new here.
    Clears when: the description says the linear fill already landed on main and that this PR only adds tests at the new cap.

[DESIGN-REVIEWED] fa3d217

Description read

[DESCRIPTION-READ] 11c4a97ceb245934a00eea395d3bbdedca89a12e78cd7e4da70cdbf42c39ad7f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of fa3d2170524e6f761c7b722cd32c394c5a8248c9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The description says this PR makes the no-script binding fill linear and changes two of its edge cases, but the diff only adds tests for a fix the base already has.

Not justified as shipped

  • Item 9 — rides along: the description says "Each start tag is now read once" and "The no-script binding fill changes in two edge cases". The diff never touches fill_bindings_as_text, and base 392a42e6 already carries the possessive _SIMPLE_BINDING_CELL (dashboard_frame.py:1148). The Reader: line therefore describes a take-away this diff does not make.
    Subtraction: delete the fill paragraphs and the Reader: line from ## What changed and ## Backwards compatibility. Keep the tests as pins of base behaviour.
    Clears when: the description stops claiming that this diff changes the fill.

What this change ships

Inventory (9 items) — 8 justified

Intent: let someone save, preview, apply and draw a hand-written dashboard page of up to 3 MiB (#20634). ADDITION.

  1. The frame accepts a page of up to 3 MiB instead of 1,000,000 bytes — justified
  2. A package page may be 3 MiB instead of 128 KiB — justified
  3. The stored-package cap becomes 2 × the page cap + 128 KiB, sized for JSON escaping — justified
  4. The sandbox-document stash grows from 4 MiB to 8 MiB — justified
  5. The oversize refusal no longer claims the page has values baked in — justified
  6. The member dashboard runs the scrub and the framing off the event loop — justified
  7. A credential scan over repeated unterminated PEM headers now runs in linear time (pem_scan.py) — justified
  8. The display battery's rules version changes because of the new rule source — justified
  9. Tests pin the linear no-script fill and how it reads hyphenated tag names — rides along: the fix is already on base, but the description says this PR makes it

[FIRST-PRINCIPLES-REVIEWED] fa3d217

Description read

[DESCRIPTION-READ] 113666b9da98d8fa6306e6d63baaace26ff7dd127a560a403eb8dea1a374f3a6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fa3d2170524e6f761c7b722cd32c394c5a8248c9 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/dashboard/handlers/sandbox_doc.py:69 -- A document containing 5,242,880 é characters passes the character-count check but returns 503 stash_full from _prune, contradicting “refused outright (413)” -> Fix: remove the guaranteed 413 from this comment and the changed artifacts spec.
[GPT-REVIEWED] fa3d217

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fa3d2170524e6f761c7b722cd32c394c5a8248c9: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fa3d2170524e6f761c7b722cd32c394c5a8248c9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fa3d217

Verdict parsed from the review's SHA-scoped output markers for commit fa3d2170524e6f761c7b722cd32c394c5a8248c9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fa3d2170524e6f761c7b722cd32c394c5a8248c9: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 15, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/20634-dashboard-page-cap branch from b640377 to 37ee5c6 Compare October 15, 2026 11:01
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 37ee5c6
  • What rule changed: The PEM private-key branch of the credential scrub (src/kiro_crew/security/redaction.py, with the new src/kiro_crew/security/pem_scan.py). A header that has no END marker after it is matched without the full-block tail, which cannot succeed there. The pattern text is unchanged.
  • Before -> after (what was allowed/blocked, now): The same spans are redacted before and after: a header with its block up to the first END marker, or a truncated tail to the end of the text. Only the cost changes. A text that repeats the header with no END marker was scanned in quadratic time (64 KiB took 2 s, 256 KiB took 40 s) and is now scanned in linear time (3 MiB takes about 0.4 s).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): A missed span would leave part of a private key unredacted in a served page or a tool output. The new scan is checked against the plain pattern search as an oracle on pinned shapes and on random texts, so a changed match shows up as a test failure.
  • Evidence checked (which sandbox result proves it): test/test_redaction_pem_scan_linear.py pins 26 PEM shapes (full, encrypted, unterminated, nested, several blocks, three newline spellings), compares 30,000 random texts with the plain search loop, compares the last-END-marker lookup with a scan of every offset, and counts that a header with no END marker never reaches the full-block tail. A one-off differential run over 270,000 random PEM-like texts found no difference between the old and the new scan.
  • No regression:
    • Tools still work (which tools/commands were run, result): redact_credentials, the display battery and the credential prefilter were exercised by test_redaction_jwt_scan_linear.py, test_security_regex_linearity.py, test_credential_prefilter.py, test_credential_patterns.py, test_display_battery_rules_version.py and test_security.py. All pass.
    • Backward compatible (old config / old data / old callers still work, how checked): The pattern text, the public functions and their arguments are unchanged, and callers of get_credential_patterns() still get the raw pattern. The new module is a rule source of the display battery, so its bytes join the rules version once.
    • Existing tests (which suites passed on this head): The security, redaction, credential and display-battery test files named above, 3328 tests in the related set run at this head, plus the CI Backend Tests for this head.
  • How to undo: Revert the second commit of this PR. The scan then returns to the single combined pattern, with the slow case on a header that no END marker follows.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in the dashboard chat tests (dfc37de -> 5b7b29f) since the first head ran; the related chat tests pass on this head. This push also carries the fixes for the failing checks and the design concern about the sandbox-document stash. CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 15, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

span=83b0acf97324

rebutted: a real page between 1,000,000 bytes and 3 MiB exists, and keeping the frame cap at 1,000,000 would let it save and then refuse to draw it.
Example from public files: three.js r170 (the unminified build, 1,325,462 bytes), its OrbitControls (32,134), Chart.js 4 (208,518) and D3 7 (279,706), with a 3 KB scene, is 1,848,313 bytes. The old frame cap (1,000,000) and the old package page cap (131,072) both refuse it. This head accepts it: validate_page returns 1848313.
Files: https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.cjs , https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/controls/OrbitControls.js , https://cdn.jsdelivr.net/npm/chart.js@4/dist/chart.umd.js , https://cdn.jsdelivr.net/npm/d3@7/dist/d3.min.js
The number and the frame change are the linked issue's own. #20634 asks for a 3 MiB page cap and for the frame limit to rise with it, so that a page that can be saved is always a page the frame accepts.
The old pin tied the frame cap to another tool's default. The frame and the package page cap are now pinned equal by test_the_page_cap_is_the_frames_own, so the two cannot drift apart.
3 MiB stays a bound: Mermaid 11 (3,572,661), Plotly 2.35 (4,558,696) and Babylon.js 7.30 (5,957,369) are over it. A runaway generation is bounded by any finite cap, and the cost of this one is measured: the scrub takes about 0.2 s at 3 MiB and the scans that were quadratic are linear.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

accepted as is: one shared 8 MiB stash can refuse a second near-cap dashboard with a 503 stash_full, and that is the existing refusal, not a new one.
Two documents at the 3 MiB cap with ordinary data are about 6.1 MiB together and both fit. Only a set of documents in flight that passes 8 MiB in total is refused, and the frame shows its retry control.
An entry is single use and leaves the stash when it is served, so room returns as soon as the frames load. A separate per-document cap would be a change to the stash itself, which this PR does not need.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does fa3d2170524e6f761c7b722cd32c394c5a8248c9 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 12 classified, 0 newly refused, 11 still allowed.
  • ubuntu-latest (platform posix): 12 classified, 0 newly refused, 11 still allowed.
  • windows-latest (platform windows): 9 classified, 0 newly refused, 8 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: Nothing in the shell deny path got stricter, and the change names no threat that it newly refuses. Every size cap is raised. MAX_PAGE_BYTES: Final[int] = 3 * 1024 * 1024 (it was 1_000_000). MAX_VIEW_HTML_BYTES = 3 * 1024 * 1024 (it was 128 * 1024). MAX_PACKAGE_BYTES = 2 * MAX_VIEW_HTML_BYTES + 128 * 1024 (it was 256 * 1024). _MAX_BYTES = 8 * 1024 * 1024 (it was 4 * 1024 * 1024). The html max_len=64 * 1024 line in validation.py changes only its comment. The one changed matcher is the PEM private-key branch of the credential redactor. When no END marker follows a header, _credential_search now matches with _CREDENTIAL_PATTERNS_NO_PEM_BLOCK instead of _CREDENTIAL_PATTERNS (if pem_scan.has_no_end_after(text, first, starts[3]): return _CREDENTIAL_PATTERNS_NO_PEM_BLOCK.match(text, first)). The diff says this matches exactly as before and only fixes a quadratic slowdown, so it is not presented as a security fix. The threat it addresses is "a text that repeats the header costs one per header: quadratic, with the GIL held". Because the redactor now runs over 3 MiB dashboard pages, one crafted page could stall the event loop. The tests check this equivalence against a plain search loop.

ADJUDICATED: 18 candidates, 0 confirmed newly-refused

PLATFORM COVERAGE: The diff adds no shell, path, separator, interpreter or environment-variable assumption. Its only platform-sensitive spellings are the line endings in the PEM body (\r?\n and a lone \r), and those live inside the redactor, not in command text. Candidates cover all three spellings:

  • posix: pytest runs of the new and edited redaction and dashboard suites, with env -u and .venv/bin/python. Also grep for the PEM marker spellings, openssl x509 and ssh-keygen -lf on public material, a curl health check against the local dashboard, wc -c on the shipped project-report page, and a scoped git diff of the two security files.
  • windows: .venv\Scripts\python.exe with backslash test paths, an absolute C:\… interpreter and test path, Select-String over pem_scan.py, ssh-keygen.exe with %USERPROFILE%, curl.exe with a NUL sink and double quotes, and (Get-Item …).Length on the template page.
  • any: a python3 -c read of dashboard_frame.MAX_PAGE_BYTES, scripts/deny_diff.py --help, and kirocrew token.

No assumed platform spelling is left without a candidate.

UNADJUDICATED: The changed matcher is the output credential redactor, security/redaction.py::_credential_search → _credential_matches → redact_credentials. None of the three shell tiers in SHELL_DENY_TIERS calls it: is_sensitive_bash_command, audit_bash_exfiltration (which matches only _BASH_EXFIL_PATTERNS/_BASH_EXFIL_RES) and is_denied. So the classifier cannot reach it, and the 0 above says nothing about it. It does make decisions in these places:

  • Blocked-link retention: exfil._credential_clean keeps a blocked link's full URL and path only if redact_credentials leaves them unchanged. It is called from _blocked_link_url_verdict and _exfil_retained_path, and _blocked_link_safe_domain makes the same check on the host.
  • Dashboard read scrub: member_dashboard._safe_body and _template_text_safe, through _serve and _serve_package.
  • Output and chat surfaces: chat_turn/file_changes.py::_flush_file_changes, which rewrites diff-chip bodies in place before they are saved.

If any of these redacted more than before, that would be a new refusal: a blocked link stripped of its URL, or a page or diff chip with text replaced. These are the inputs I would have submitted, none of which I could get a verdict for:

  • A link with a path such as https://docs.example.com/guide/-----END-PRIVATE-KEY-----.
  • The string -----BEGIN PUBLIC KEY----- in a dashboard page.
  • Prose that mentions a private-key header with no END marker, followed by ordinary lines.
  • A file diff that adds a PEM header constant to source code.
  • A CRLF-terminated public certificate block.

The changed branch is reached only at a private-key header with no END marker at or after the header's end. The full-block alternative can never succeed in that position, so in principle nothing should flip. The confirmation is the repo's own randomized comparison with the plain search loop, not this lane.

[SCOPE-REVIEWED] fa3d217

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

The only red check is macOS Tests / Backend Tests (macOS) (1), an on-demand suite that is not required. It fails on test/test_ai_review_shallow_fetch.py::test_the_gpt_reviewer_is_told_the_two_dot_range, which runs the codex-review workflow's sed -i "s/..." file line under bash. BSD sed on macOS needs a suffix after -i, so the command exits non-zero and the test raises CalledProcessError.

This PR changes no .github file and nothing in that test. The sed line comes from #20583, so any PR whose macOS shard 1 runs that test fails the same way. Macos shards 2, 3 and 4 pass.

@chenmingwei23
chenmingwei23 force-pushed the feat/20634-dashboard-page-cap branch from 37ee5c6 to 46b50de Compare October 15, 2026 11:47
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 46b50de
  • What rule changed: The PEM private-key branch of the credential scrub (src/kiro_crew/security/redaction.py, with the new src/kiro_crew/security/pem_scan.py). A header that has no END marker after it is matched without the full-block tail, which cannot succeed there. The pattern text is unchanged.
  • Before -> after (what was allowed/blocked, now): The same spans are redacted before and after: a header with its block up to the first END marker, or a truncated tail to the end of the text. Only the cost changes. A text that repeats the header with no END marker was scanned in quadratic time (64 KiB took 2 s, 256 KiB took 40 s) and is now scanned in linear time (3 MiB takes about 0.4 s).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): A missed span would leave part of a private key unredacted in a served page or a tool output. The new scan is checked against the plain pattern search as an oracle on pinned shapes and on random texts, so a changed match shows up as a test failure.
  • Evidence checked (which sandbox result proves it): test/test_redaction_pem_scan_linear.py pins 26 PEM shapes (full, encrypted, unterminated, nested, several blocks, three newline spellings), compares 30,000 random texts with the plain search loop, compares the last-END-marker lookup with a scan of every offset, and counts that a header with no END marker never reaches the full-block tail. A one-off differential run over 270,000 random PEM-like texts found no difference between the old and the new scan.
  • No regression:
    • Tools still work (which tools/commands were run, result): redact_credentials, the display battery and the credential prefilter were exercised by test_redaction_jwt_scan_linear.py, test_security_regex_linearity.py, test_credential_prefilter.py, test_credential_patterns.py, test_display_battery_rules_version.py and test_security.py. All pass.
    • Backward compatible (old config / old data / old callers still work, how checked): The pattern text, the public functions and their arguments are unchanged, and callers of get_credential_patterns() still get the raw pattern. The new module is a rule source of the display battery, so its bytes join the rules version once.
    • Existing tests (which suites passed on this head): The security, redaction, credential and display-battery test files named above, 3328 tests in the related set run at this head, plus the CI Backend Tests for this head.
  • How to undo: Revert the second commit of this PR. The scan then returns to the single combined pattern, with the slow case on a header that no END marker follows.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Rebased: main moved by five commits (5b7b29f -> 58a1885) since the last head ran, among them the portable sed fix for the macOS shard (#20712). The scoped tests for the files the green-age check named pass on the merged tree. CI re-running.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

needs-a-decision (maintainer): the stored-package cap is 2 x the page cap + 128 KiB (6.1 MiB), and the reader applies the same number to a stored package.
So the cap can be raised later but not lowered while packages written under the larger value remain. Lowering it would stop those packages from loading.
The PR body (Backwards compatibility) and the MAX_PACKAGE_BYTES comment say so. Please confirm at review that this one-way door is acceptable.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

needs-a-decision (maintainer): 6.1 MiB is derived, not chosen. It is 2 x the 3 MiB page cap + 128 KiB, because the stored JSON writes a quote or a line break as two bytes.
It is a one-way door as the PR states: the reader applies the same cap, so it can be raised but not lowered while larger packages are stored. Please confirm at review that this is acceptable.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 15, 2026
@chenmingwei23
chenmingwei23 enabled auto-merge (squash) October 15, 2026 12:35
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 16, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/20634-dashboard-page-cap branch from ff0eef6 to 8643e60 Compare October 16, 2026 07:46
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 8643e60
  • What rule changed: The PEM private-key branch of the credential scrub (src/kiro_crew/security/redaction.py, with the new src/kiro_crew/security/pem_scan.py). A header that has no END marker after it is matched without the full-block tail, which cannot succeed there. The pattern text is unchanged.
  • Before -> after (what was allowed/blocked, now): The same spans are redacted before and after: a header with its block up to the first END marker, or a truncated tail to the end of the text. Only the cost changes. A text that repeats the header with no END marker was scanned in quadratic time (64 KiB took 2 s, 256 KiB took 40 s) and is now scanned in linear time (3 MiB takes about 0.4 s).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): A missed span would leave part of a private key unredacted in a served page or a tool output. The new scan is checked against the plain pattern search as an oracle on pinned shapes and on random texts, so a changed match shows up as a test failure.
  • Evidence checked (which sandbox result proves it): test/test_redaction_pem_scan_linear.py pins 26 PEM shapes (full, encrypted, unterminated, nested, several blocks, three newline spellings), compares 30,000 random texts with the plain search loop, compares the last-END-marker lookup with a scan of every offset, and counts that a header with no END marker never reaches the full-block tail. A one-off differential run over 270,000 random PEM-like texts found no difference between the old and the new scan.
  • No regression:
    • Tools still work (which tools/commands were run, result): redact_credentials, the display battery and the credential prefilter were exercised by test_redaction_jwt_scan_linear.py, test_security_regex_linearity.py, test_credential_prefilter.py, test_credential_patterns.py, test_display_battery_rules_version.py and test_security.py. All pass.
    • Backward compatible (old config / old data / old callers still work, how checked): The pattern text, the public functions and their arguments are unchanged, and callers of get_credential_patterns() still get the raw pattern. The new module is a rule source of the display battery, so its bytes join the rules version once.
    • Existing tests (which suites passed on this head): The six security and redaction test files named above, the PEM scan test, the liveness test, the eight dashboard test files this PR touches, and seven test files that main's latest page-scrub and suggestions changes added or extended, 23 files in all. 3350 tests passed and 1 was skipped, run at this head, plus the CI Backend Tests for this head.
  • How to undo: Revert the second commit of this PR. The scan then returns to the single combined pattern, with the slow case on a header that no END marker follows.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Rebased: main's #19866 (the suggestions page for a crewmate with no adopted page) edited the same lines of member_dashboard.py as this PR. Both changes are kept. The await _run(...) call that moves the scrub off the event loop stays on its line, and main's comment block and its body["suggestions"] line follow it unchanged.

git range-diff against the previous head shows the first commit differing only in the context lines around that spot, and the second commit identical. The security package line budget needed no change: the measured count is still 29,492. The related tests pass on this head together with the tests main added for #19866. CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 16, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/20634-dashboard-page-cap branch from 8643e60 to 33c036c Compare October 16, 2026 16:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 16, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 33c036c
  • What rule changed: Two things. (1) The PEM private-key branch of the credential scrub (src/kiro_crew/security/redaction.py, with the new src/kiro_crew/security/pem_scan.py). A header that has no END marker after it is matched without the full-block tail, which cannot succeed there. The pattern text is unchanged. (2) The hand-written dashboard page cap is raised to 3 MiB (MAX_PAGE_BYTES in src/kiro_crew/dashboard_frame.py), with the matching package and frame ceilings. This head is rebased onto main: the no-script fill regex (_SIMPLE_BINDING_CELL) is now exactly main's tokenizer-style version from fix(dashboard): no-script fill reads a tag the way a tokenizer does (closes #21005) #21153, and this PR no longer changes it. The test package line budget is 29,573: main's 29,492 plus 81 lines for the PEM scan (56 for pem_scan.py, 25 for the choice between the two patterns).
  • Before -> after (what was allowed/blocked, now): The same spans are redacted before and after: a header with its block up to the first END marker, or a truncated tail to the end of the text. Only the cost changes. A text that repeats the header with no END marker was scanned in quadratic time (64 KiB took 2 s, 256 KiB took 40 s) and is now scanned in linear time (3 MiB takes about 0.4 s). Pages up to 3 MiB are now accepted instead of 1,000,000 bytes; larger pages are still refused. The no-script fill reads a start tag the way main already does, so its behavior on this head equals main's.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): A missed span would leave part of a private key unredacted in a served page or a tool output. The new scan is checked against the plain pattern search as an oracle on pinned shapes and on random texts, so a changed match shows up as a test failure. A page near the cap that is slow to scan would hold the gateway; the timing tests below run the real fill on a page at the ceiling.
  • Evidence checked (which sandbox result proves it): test/test_redaction_pem_scan_linear.py pins 26 PEM shapes (full, encrypted, unterminated, nested, several blocks, three newline spellings), compares 30,000 random texts with the plain search loop, compares the last-END-marker lookup with a scan of every offset, and counts that a header with no END marker never reaches the full-block tail. A one-off differential run over 270,000 random PEM-like texts found no difference between the old and the new scan. The page-cap tests in test_dashboard_frame.py, test_dashboard_package_html_view.py and test_member_dashboard_routes.py exercise a page at the frame ceiling through preview, apply and draw, and include the tests that fix(dashboard): skip raw-text element contents in the page skeleton scan #21141 and fix(dashboard): no-script fill reads a tag the way a tokenizer does (closes #21005) #21153 added.
  • No regression:
    • Tools still work (which tools/commands were run, result): redact_credentials, the display battery and the credential prefilter were exercised by test_redaction_jwt_scan_linear.py, test_security_regex_linearity.py, test_credential_prefilter.py, test_credential_patterns.py, test_display_battery_rules_version.py and test_security.py, run on this head. All pass.
    • Backward compatible (old config / old data / old callers still work, how checked): The pattern text, the public functions and their arguments are unchanged, and callers of get_credential_patterns() still get the raw pattern. The new module is a rule source of the display battery, so its bytes join the rules version once. Pages that fit under the old 1,000,000-byte cap are unaffected by the larger cap.
    • Existing tests (which suites passed on this head): Run locally at this head, with no full suite: test_dashboard_frame.py, test_dashboard_package_html_view.py, test_redaction_pem_scan_linear.py, test_redaction_jwt_scan_linear.py, test_security_gate_liveness.py, test_artifact_dashboard_package.py, test_member_dashboard_package_route.py, test_sandbox_doc_channel.py, test_dashboard_templates_builtin.py, test_dashboard_template_office.py and test_member_dashboard_routes.py (786 passed); plus the six security and redaction files named above (2007 passed, 1 skipped). CI runs the rest on this head.
  • How to undo: Revert the second commit of this PR to return the scrub to the single combined pattern, and revert the first commit to return the page cap to 1,000,000 bytes.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 16, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/20634-dashboard-page-cap branch from 33c036c to 02e0786 Compare October 16, 2026 16:58
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 02e0786
  • What rule changed: Two things. (1) The PEM private-key branch of the credential scrub (src/kiro_crew/security/redaction.py, with the new src/kiro_crew/security/pem_scan.py). A header that has no END marker after it is matched without the full-block tail, which cannot succeed there. The pattern text is unchanged. (2) The hand-written dashboard page cap is raised to 3 MiB (MAX_PAGE_BYTES in src/kiro_crew/dashboard_frame.py), with the matching package and frame ceilings. This head is rebased onto current main (the 12 commits since the last head did not touch src/kiro_crew/security or src/kiro_crew/dashboard_frame.py): the no-script fill regex (_SIMPLE_BINDING_CELL) is now exactly main's tokenizer-style version from fix(dashboard): no-script fill reads a tag the way a tokenizer does (closes #21005) #21153, and this PR no longer changes it. The test package line budget is 29,573: main's 29,492 plus 81 lines for the PEM scan (56 for pem_scan.py, 25 for the choice between the two patterns).
  • Before -> after (what was allowed/blocked, now): The same spans are redacted before and after: a header with its block up to the first END marker, or a truncated tail to the end of the text. Only the cost changes. A text that repeats the header with no END marker was scanned in quadratic time (64 KiB took 2 s, 256 KiB took 40 s) and is now scanned in linear time (3 MiB takes about 0.4 s). Pages up to 3 MiB are now accepted instead of 1,000,000 bytes; larger pages are still refused. The no-script fill reads a start tag the way main already does, so its behavior on this head equals main's.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): A missed span would leave part of a private key unredacted in a served page or a tool output. The new scan is checked against the plain pattern search as an oracle on pinned shapes and on random texts, so a changed match shows up as a test failure. A page near the cap that is slow to scan would hold the gateway; the timing tests below run the real fill on a page at the ceiling.
  • Evidence checked (which sandbox result proves it): test/test_redaction_pem_scan_linear.py pins 26 PEM shapes (full, encrypted, unterminated, nested, several blocks, three newline spellings), compares 30,000 random texts with the plain search loop, compares the last-END-marker lookup with a scan of every offset, and counts that a header with no END marker never reaches the full-block tail. A one-off differential run over 270,000 random PEM-like texts found no difference between the old and the new scan. The page-cap tests in test_dashboard_frame.py, test_dashboard_package_html_view.py and test_member_dashboard_routes.py exercise a page at the frame ceiling through preview, apply and draw, and include the tests that fix(dashboard): skip raw-text element contents in the page skeleton scan #21141 and fix(dashboard): no-script fill reads a tag the way a tokenizer does (closes #21005) #21153 added.
  • No regression:
    • Tools still work (which tools/commands were run, result): redact_credentials, the display battery and the credential prefilter were exercised by test_redaction_jwt_scan_linear.py, test_security_regex_linearity.py, test_credential_prefilter.py, test_credential_patterns.py, test_display_battery_rules_version.py and test_security.py, run on this head. All pass.
    • Backward compatible (old config / old data / old callers still work, how checked): The pattern text, the public functions and their arguments are unchanged, and callers of get_credential_patterns() still get the raw pattern. The new module is a rule source of the display battery, so its bytes join the rules version once. Pages that fit under the old 1,000,000-byte cap are unaffected by the larger cap.
    • Existing tests (which suites passed on this head): Run locally at this head, with no full suite: test_security_gate_liveness.py, test_dashboard_frame.py, test_dashboard_package_html_view.py, test_redaction_pem_scan_linear.py, test_member_dashboard_routes.py and test_member_dashboard_package_route.py (475 passed). The wider set (the artifact, sandbox document, template and six security and redaction files, 2,793 passed and 1 skipped in all) passed on the previous head, and the rebase since then brought in no change under the security package or the frame module. CI runs the rest on this head.
  • How to undo: Revert the second commit of this PR to return the scrub to the single combined pattern, and revert the first commit to return the page cap to 1,000,000 bytes.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Oct 16, 2026
A hand-written dashboard page that inlines a client-side library, such as
a 3D renderer, together with its own scene code is several hundred KB to
a few MB of layout with no data in it. The 128 KiB page cap in a
dashboard package refused every such page, and the frame drew nothing
over 1,000,000 bytes.

The page cap is now 3 MiB in the package and in the frame. The package
cap is twice that plus 128 KiB, because the stored JSON writes a quote or
a line break in the page as two bytes. The page cap bounds what a page
costs to store, send and draw; it no longer claims to detect values
baked into the layout, which a size cannot tell from a library.

The sandbox document stash that holds the composed page is 8 MiB instead
of 4 MiB, so a page at the cap keeps room for the largest data island a
package can hold.

Two scans that were harmless at the old caps now see pages 24 times as
large. The member dashboard route scrubs and frames a stored page off the
event loop. The no-script binding fill reads each start tag once, where
it used to read on to the end of the page from every unclosed tag, which
was quadratic.

Fixes #20634
…r time

The credential pattern spells a PEM private key as a header followed by
a full block, the body up to the first END marker, or a truncated tail.
The full block fails only after reading to the end of the text, so a
header that no END marker follows costs one such read, and a text that
repeats the header costs one per header: quadratic, with the GIL held.
With the dashboard page cap at 3 MiB, a page of repeated headers could
stop the gateway for hours while it was redacted.

The scan now looks up the text's last END marker once and matches a
header past it with the same alternation minus the full-block tail,
which cannot succeed there. The pattern text and every match are
unchanged. The new module is a rule source of the display battery, so
its bytes join the rules version.

Part of #20634
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: fa3d217
  • What rule changed: Two things. (1) The PEM private-key branch of the credential scrub (src/kiro_crew/security/redaction.py, with the new src/kiro_crew/security/pem_scan.py). A header that has no END marker after it is matched without the full-block tail, which cannot succeed there. The pattern text is unchanged. (2) The hand-written dashboard page cap is raised to 3 MiB (MAX_PAGE_BYTES in src/kiro_crew/dashboard_frame.py), with the matching package and frame ceilings. This head is rebased onto current main (the 12 commits since the last head did not touch src/kiro_crew/security or src/kiro_crew/dashboard_frame.py): the no-script fill regex (_SIMPLE_BINDING_CELL) is now exactly main's tokenizer-style version from fix(dashboard): no-script fill reads a tag the way a tokenizer does (closes #21005) #21153, and this PR no longer changes it. The test package line budget is 29,573: main's 29,492 plus 81 lines for the PEM scan (56 for pem_scan.py, 25 for the choice between the two patterns).
  • Before -> after (what was allowed/blocked, now): The same spans are redacted before and after: a header with its block up to the first END marker, or a truncated tail to the end of the text. Only the cost changes. A text that repeats the header with no END marker was scanned in quadratic time (64 KiB took 2 s, 256 KiB took 40 s) and is now scanned in linear time (3 MiB takes about 0.4 s). Pages up to 3 MiB are now accepted instead of 1,000,000 bytes; larger pages are still refused. The no-script fill reads a start tag the way main already does, so its behavior on this head equals main's.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): A missed span would leave part of a private key unredacted in a served page or a tool output. The new scan is checked against the plain pattern search as an oracle on pinned shapes and on random texts, so a changed match shows up as a test failure. A page near the cap that is slow to scan would hold the gateway; the timing tests below run the real fill on a page at the ceiling.
  • Evidence checked (which sandbox result proves it): test/test_redaction_pem_scan_linear.py pins 26 PEM shapes (full, encrypted, unterminated, nested, several blocks, three newline spellings), compares 30,000 random texts with the plain search loop, compares the last-END-marker lookup with a scan of every offset, and counts that a header with no END marker never reaches the full-block tail. A one-off differential run over 270,000 random PEM-like texts found no difference between the old and the new scan. The page-cap tests in test_dashboard_frame.py, test_dashboard_package_html_view.py and test_member_dashboard_routes.py exercise a page at the frame ceiling through preview, apply and draw, and include the tests that fix(dashboard): skip raw-text element contents in the page skeleton scan #21141 and fix(dashboard): no-script fill reads a tag the way a tokenizer does (closes #21005) #21153 added.
  • No regression:
    • Tools still work (which tools/commands were run, result): redact_credentials, the display battery and the credential prefilter were exercised by test_redaction_jwt_scan_linear.py, test_security_regex_linearity.py, test_credential_prefilter.py, test_credential_patterns.py, test_display_battery_rules_version.py and test_security.py, run on this head. All pass.
    • Backward compatible (old config / old data / old callers still work, how checked): The pattern text, the public functions and their arguments are unchanged, and callers of get_credential_patterns() still get the raw pattern. The new module is a rule source of the display battery, so its bytes join the rules version once. Pages that fit under the old 1,000,000-byte cap are unaffected by the larger cap.
    • Existing tests (which suites passed on this head): Run locally at this head, with no full suite: test_security_gate_liveness.py, test_dashboard_frame.py, test_dashboard_package_html_view.py, test_redaction_pem_scan_linear.py, test_member_dashboard_routes.py, test_member_dashboard_package_route.py and test_member_dashboard_package_route_live_page.py (475 passed). The two page-cap route tests now live in the live-page file that main's latest test split created. The wider set (the artifact, sandbox document, template and six security and redaction files, 2,793 passed and 1 skipped in all) passed on an earlier head, and the rebases since then brought in no change under the security package or the frame module. CI runs the rest on this head.
  • How to undo: Revert the second commit of this PR to return the scrub to the single combined pattern, and revert the first commit to return the page cap to 1,000,000 bytes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard package: raise the 128 KiB page cap and 256 KiB package cap for hand-written pages

2 participants