Skip to content

feat(source): POST /api/source/issues lists and searches GitHub and GitLab issues - #34742

Merged
bolichen97 merged 2 commits into
mainfrom
feat/source-issues-query
Nov 11, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
feat/source-issues-query

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let apps and the dashboard list and search one repository's or project's GitHub and GitLab issues through the gateway, with the same sign-in, owner gate and redaction as the single-issue read.

Today the gateway can read one issue by URL (POST /api/source/issue) but has no way to list or search issues. The new app-sdk issue components (IssueList, IssueCard, useIssues) need a query endpoint with a fixed contract, so they can be built against it in parallel.

Why it matters

Without a list endpoint each app that shows issues has to bring its own GitHub or GitLab client and credentials, outside the gateway's owner gate, host allowlist and redaction. One gateway route keeps credentials inside gh/glab and gives every app the same validated, sanitized shape.

Not a goal

  • Jira and Linear. The contract names them; they answer 400 provider_not_supported here and land later, one provider function each.
  • The frontend components. Another PR builds IssueList / IssueCard / useIssues against the types added here.
  • Changing Issue Radar. Its own app-side clients are untouched.

What changed (motivation → approach → change)

Goal → one owner-only route that validates every field before any provider call and fixes the scope itself → source_providers/issue_search.py plus a handler and route.

  • Route and auth. POST /api/source/issues (api_issue_search in handlers/source_providers.py, registered in routes/chat.py) uses the same _authorize_owner_request, audit helper and _provider_error_response mapping as api_issue_source. Audit operation: source.issues.list. The app-token scope check admits the path only when a manifest declares /api/source/issues, then the owner gate refuses app tokens, as for /api/source/pull-request; the segment-based grant rule keeps /api/source/issue and /api/source/issues apart (tested, unchanged).
  • Validation. Every field is type-checked, length-capped and free of control characters. Scope must be a GitHub owner/repo or a GitLab project path. host must be github.com / gitlab.com or on the gateway's existing github_hosts / gitlab_hosts allowlist. Logins follow Issue Radar's login rules. Refusals never echo the submitted value.
  • Scope cannot be widened. GitHub: the gateway writes repo:<scope> is:issue itself, quotes free text word by word, and refuses raw terms that name repo/org/user/owner or use OR/AND/NOT/parentheses; results are also dropped unless their repository_url is the scoped repo, and pull requests are dropped. GitLab: the read is always projects/<scope>/issues; raw is parsed as key=value pairs against an allowlist of narrowing filters, and the gateway's scope=all, paging and ordering come first. Arguments go to gh api/glab api as one path argument through the existing runner, so no field becomes a CLI flag.
  • Sanitized output. issue_search.py validates, fetches and projects, and never imports the redactor. cache.search_issues (beside the single-issue, checks and contributor reads) runs _redact_provider_data on the raw provider rows before projection (an excerpt joins lines, and a PEM key is only recognised across them) and again on the projected list (excerpts stripped, newlines kept); only then are excerpts flattened and cut. URLs are built from the validated host and scope, not from provider html_url/web_url; avatar URLs go through projection._safe_https_url. The excerpt is the body's first ~280 characters, stripped from the whole body, redacted, then cut at a word. Bodies are fetched whole (no jq cut), so both redaction passes see each secret entire. Only the endpoint's own refusals echo text; any other error answers provider_error; GitLab paging headers must be ASCII digits.
  • Provider dispatch. _PROVIDERS maps a provider id to one validate, one fetch and one project function, so Jira and Linear are one entry each.
  • Paging. nextCursor is the next page number. GitHub stops at its 1,000-result window (a last partial page reads the window's final 100); GitLab reads X-Next-Page / X-Total from glab api --include (total is null when GitLab omits it). Results are cached for 30 seconds; refresh: true bypasses it.
  • Types and docs. IssueSearchRequest, IssueSearchResponse, IssueSummary, IssuePerson and IssueSearchProvider in website/src/types/index.sources.ts (re-exported from index.ts), matching the contract; an Issue Search section in docs/app-kit/api-reference.md; an API-only Issue search row in docs/feature-map/README.md.
  • Excerpt strip and validators are plain string code. No regular expressions in the module (per review), and each pass is linear, so a body of repeated ![, [ or < is cheap.

Backwards compatibility

Compatible: adds a new route and new exported types; no existing route, field or type changes.

Removes nothing: the diff only adds a route, a provider module, types and docs.

Known limits

Left as is in this PR (local Opus review advisories, not upheld as blocking):

  • A GitHub page whose full issue bodies pass the 4 MiB output cap (near-maximum bodies at a high limit) fails as a 503 provider_error instead of returning results; marked shortcut: in _fetch_github_issues. Lowering limit avoids it.
  • cut_excerpt returns an empty excerpt when the first 281 characters of the stripped body hold no whitespace (for example a body that opens with a very long URL). Cosmetic.
  • A raw GitHub term whose key starts with a quoted run ("x"repo:other/r) skips the scope-qualifier refusal. Results stay scoped: rows outside the repository are dropped by the repository_url check.

Tests

test/test_source_issue_search.py (142 cases):

  • Auth: the denial set matches /api/source/issue, the owner read is audited as source.issues.list, and the app-token grant is segment-bounded both ways (/api/source/issue vs /api/source/issues).
  • Boundary: invalid scope, host, labels, text, raw, cursor, limit, state, sort and order are refused before any provider call; a refusal never echoes the value; raw repo:other/repo, -repo:, org:, user:, owner:, OR and parentheses are refused; --hostname as scope, host or login is refused, and in text or raw it stays a word inside the encoded q path argument, never a CLI argument; a non-allowlisted self-managed GitLab host is refused; GitLab raw cannot set scope, per_page or a repeated filter.
  • Nearby inputs: empty results (both providers), paging past the last page, a closed-only query, a label with spaces, a GitLab nested group, an allowlisted GitHub Enterprise and self-managed GitLab host, assignee: 'me' resolved to the GitLab username, assignee: 'me' with no signed-in user failing closed, signed-out GitHub mapping like the single-issue read.
  • Output: pull requests dropped, out-of-repo rows dropped, shape matches IssueSummary, a secret-shaped string in a GitHub or GitLab title or body comes back redacted in place through POST /api/source/issues, the cached copy is the redacted one, a PEM key in a GitHub or GitLab body is redacted before the excerpt joins its lines, an indented key whose END line lies past character 4,000 is redacted (red when the body arrives cut), a token split by **, backticks, a link or a multi-line comment in a link is redacted wherever it sits in the body, a token joined by the markdown strip is redacted (removing either redact pass turns at least one red), bodies are fetched without a jq cut, no excerpt keeps a cut token fragment, a non-ASCII-digit GitLab header is ignored and an unexpected provider ValueError is not echoed, excerpt stripping and cut, the link and image strip stopping at the next bracket (red on a pattern that crosses it), cache and refresh.

test/test_source_issue_search_string_parity.py (33 cases) keeps each replaced regex as an oracle and checks the string code agrees with it: every validator, every strip pass and the whole excerpt, on edge and adversarial inputs (Unicode digits, empty, long no-space runs, PEM headers with emphasis, (/) in urls, CRLF, Unicode spaces) plus a seeded generated corpus. Changing a bound, a digit test or the emphasis order turns it red.

Edited existing test: test_link_strip_tries_stop_at_the_next_bracket asserted _LINK_RE/_IMAGE_RE.match(text) is None, and those objects are gone. It now asserts the same property (the strip try at offset 0 fails, so the opener is kept) through _strip_bracket_links on the same 4 inputs, which are also in the parity corpus. No other test changed.

Determinism: 20 runs at -n0 in a shuffled order (seed 20261110) and 10 runs at -n 4 --dist loadgroup beside test/test_source_providers_issues.py and test/test_source_providers_endpoints.py, all green.

Proof: not provable -- other: new feature, no fix to revert; the new tests exercise a route that does not exist without this diff

Manual verification

N/A — unit coverage sufficient: both provider reads are exercised through the real handler and runner with gh/glab output faked at the subprocess boundary.

Screenshots / video

Why no screenshot: the only frontend change is new exported TypeScript types; nothing renders differently.

Related Issues

None.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 10, 2026 19:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The PR adds an API route and TypeScript types but no rendered control, so the missing blind read leaves no evidence gap.

[UX-REVIEWED] 57e238e

Description read

[DESCRIPTION-READ] bf5f14c7ee0ee33bb268d800fb47f1f3f52a37712e3f415921506abfdf7fcce0

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the route only adds things, uses the same owner gate and redaction as the single-issue read, and checks results against the scope it was given.

Notes

  • GitHub raw uses a deny-list of qualifiers, while GitLab uses an allow-list. Only the repository_url drop keeps GitHub results inside the repo, and the quoted-key bypass listed in Known limits already gets past the deny-list. An allow-list of narrowing qualifiers would match GitLab and make that drop a second check rather than the only one.

[DESIGN-REVIEWED] 57e238e

Description read

[DESCRIPTION-READ] f7e3ab151edf7575829c3c6d7691b37a34125bd058c62bb1dc915bc4c1e4c6bd

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Two items add surface that the Goal doesn't need: placeholder Jira/Linear values, and a hand-written markdown stripper used only for excerpts.

Notes

  • Item 4 — zero consumers: ## Not a goal excludes "Jira and Linear", yet _PLANNED_PROVIDERS = frozenset({"jira", "linear"}) and IssueSearchProvider = 'github' | 'gitlab' | 'jira' | 'linear' ship as public contract. 0 constructors of either value outside tests (grepped jira|linear in the non-test hunks). Subtraction: delete _PLANNED_PROVIDERS and the provider_not_supported code, and narrow the type to 'github' | 'gitlab'. Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 5 — oversized: strip_markdown and its 13 helpers (_strip_fences … _strip_emphasis, _drop_partial_word) exist only to turn the excerpt into plain text. They are why the excerpt needs a second redact pass ("a token joined by the markdown strip"). The Goal asks for a "validated, sanitized shape", not plain-text rendering. Subtraction: return the redacted body's first ~280 characters raw, and delete the strip helpers and the second _redact_provider_data pass. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (10 items) — 8 justified

Intent: apps and the dashboard can list and search one repo's or project's GitHub/GitLab issues through the gateway's owner gate and redaction. ADDITION.

  1. Owner-only POST /api/source/issues lists and searches one GitHub repo's issues — justified
  2. The same route covers GitLab projects, including allowlisted self-managed hosts — justified
  3. A manifest grant for /api/source/issues, separate from /api/source/issue — justified
  4. Jira/Linear answer 400 provider_not_supported; the public type names them — zero consumers
  5. Excerpt is the markdown-stripped first ~280 characters, cut at a word — oversized
  6. Results cached for 30 seconds; refresh: true bypasses it — justified
  7. Page-number cursor; GitHub stops at its 1,000-result window — justified
  8. Raw scope-widening qualifiers refused; PRs and out-of-repo rows dropped — justified
  9. New audit operation source.issues.list — justified
  10. Exported IssueSearch*/IssueSummary/IssuePerson TS types — justified

[FIRST-PRINCIPLES-REVIEWED] 57e238e

Description read

[DESCRIPTION-READ] c5b993cb264f61f98824365aee4b98bd0e527ffd5e9193f881f0194dca57dea9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 57e238e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 57e238e

Verdict parsed from the review's SHA-scoped output markers for commit 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 57e238ea1eb1dafc7f16e2fb54d4ed5b619b3999 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@kyleseaman
kyleseaman force-pushed the feat/source-issues-query branch from ec21a1f to f2efacf Compare November 10, 2026 21:37
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 10, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own.

Adds an owner-only POST /api/source/issues (GitHub/GitLab list and search). Auth matches api_issue_source. The scope is fixed by the gateway and checked again: GitHub refuses repo/org/user/owner and boolean terms in raw and keeps only rows whose repository_url ends with the scoped repo; GitLab reads only projects//issues and raw keys must be on an allowlist. Hosts must be on the existing allowlists, and rows are redacted before and after projection, then byte-capped, before caching. The GPT lane's PEM finding is real only for a key whose header was deliberately written with markdown emphasis AND whose base64 lines are under 40 characters (64-character lines are caught by the bare-secret check). It exposes at most about 280 characters to the owner, and the same body comes back whole and unredacted through the existing single-issue read on main, so this is not a regression. Non-blocking.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit f2efacf19d · 1958 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@kyleseaman
kyleseaman force-pushed the feat/source-issues-query branch from f2efacf to 39e7605 Compare November 10, 2026 23:12
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 11, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on 39e7605 for one point. The route itself looks right to me.

Auth is the same as the existing source routes. api_issue_search calls _authorize_owner_request(request, "source.issues.list") first, the same gate as api_issue_source on main: owner match once an owner is configured, the signed local identity otherwise, app tokens and unsigned callers refused. It is registered beside /api/source/issue in routes/chat.py, provider calls go through runner._run_json / _run_provider with the allowlisted host, and the rows are redacted before and after the projection and byte-capped. I found no unauthenticated path. The scope is written by the gateway (repo: first, results filtered by repository_url; GitLab is always the one project's endpoint), and the raw-query checks only keep the query inside that scope.

New regular expressions in product code. This repo keeps regex out of product code. issue_search.py adds import re and about 17 patterns: _CURSOR_RE, _HTML_COMMENT_RE, _FENCE_RE, _IMAGE_RE, _LINK_RE, _HTML_TAG_RE, _LINE_PREFIX_RE, _RULE_RE, _EMPHASIS_RE, _SPACE_RE, the inline re.search(r"\s\S*\Z", ...) in _drop_partial_word, _GITHUB_LOGIN_RE, _GITHUB_REPO_RE, _QUOTED_RE, _GITLAB_SEGMENT_RE, _GITLAB_LOGIN_RE and _HEADER_INT_RE. Some names match patterns elsewhere in the tree, but these are new copies in a new module, not moves. Plain string code covers each of them:

  • Numbers (_CURSOR_RE, _HEADER_INT_RE): s.isascii() and s.isdigit() plus a length check, and s[0] != "0" for the cursor.
  • Logins, repo names and path segments: a frozenset of allowed characters, all(c in ALLOWED for c in s), a first-character check and a length bound.
  • Whitespace (_SPACE_RE, _drop_partial_word): " ".join(text.split()), and cut at the last whitespace found by a reverse scan or rsplit(None, 1).
  • _QUOTED_RE: _split_terms already walks the quotes, so the same loop can drop quoted spans.
  • The markdown strip: one pass over splitlines(keepends=True) with startswith checks for fences, rules, headings, list and quote markers, plus str.find scans for <!--/-->, ![...](...), [...](...) and <tag>, and str.replace for **, __, ~~ and backticks. That also removes the backtracking concern the _IMAGE_RE comment works around.

Regex in the tests is fine and out of scope. Once the patterns are gone I will re-read the module and the tests.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Thanks for this. The review found nothing that blocks on this head. The PR adds a row to docs/feature-map/README.md, and a change to that document needs a maintainer to sign off before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/).

What the review checked on this head:

  • The new POST /api/source/issues route uses the same owner gate as the single-issue read. App tokens, unsigned callers and non-owners get a 403, and each denial is audited.
  • The gateway sets the scope itself, and that holds. GitHub search always starts with repo:<scope> is:issue. Rows outside that repository, and pull requests, are dropped. GitLab reads stay on the project endpoint, and raw keys must be on an allowlist. Result URLs are rebuilt from the validated host and scope.
  • Redaction runs on the raw provider rows, runs again after the markdown strip, and runs before the excerpt cut. Tests fail if either pass is removed.
  • The tests in the new test file pass at this head. They have no sleeps, no fixed ports and no clock waits, and the cache is cleared around each test.

Done well: a refused query never echoes the submitted value, and the validation tests check that the provider fake was never called, not only that the status was 400.

Optional:

  • feat(app-sdk): issue list components (useIssues, IssueList, IssueCard) #34732 adds its own IssueSummary, IssuePerson, IssueQuery and IssueQueryResponse to the same type files. Whichever PR merges second will conflict. One PR could own these types while the other imports them, with the merge order named in both bodies.
  • No caller sets raw yet, and it carries most of the new parsing code plus the scope gap the body lists under Known limits. You could ship it later with the first caller that needs it.
  • No test checks that two queries differing only in host or raw get different cache keys. Dropping either field from IssueQuery.cache_key leaves every test green.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 11, 2026
Kyle Seaman added 2 commits November 11, 2026 15:08
… its cut, replace regexes with string code, say issue search refuses app tokens

The portability gate reads a literal '/' split/join as filesystem path assembly; a GitLab project path is a URL path, so it uses a named separator as links.py does. The internal-content scan reads whole PEM markers in source lines; the test fixtures now assemble them from their dashes. The excerpt is stripped with its newlines kept, redacted, and only then flattened and cut. Per review, issue_search.py uses plain string code instead of regular expressions for its validators and markdown strip, with the same behaviour, checked against the old patterns kept as oracles in test_source_issue_search_string_parity.py. The feature map and app-kit reference say the endpoint answers the owner's dashboard session and refuses an app token, as /api/source/pull-request does.
@kyleseaman
kyleseaman force-pushed the feat/source-issues-query branch from 39e7605 to 57e238e Compare November 11, 2026 15:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 11, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Nov 11, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, my point is fixed. I re-read the whole diff, tests included.

  • Regex: issue_search.py no longer imports re. The validators use _charset_match over ASCII frozensets, quoted spans are dropped with str.find, and the markdown strip is plain string scans that stay linear. cache.py keeps its existing import re, which this PR does not touch.
  • Parity: test_source_issue_search_string_parity.py keeps the old patterns as test-only oracles and checks every replacement against them on edge cases, a seeded 4,000-item corpus and long adversarial runs. That is a good way to prove the rewrite changed nothing.
  • Auth and exposure (unchanged since my last read): api_issue_search calls _authorize_owner_request(request, "source.issues.list") first, the same gate as api_issue_source, and it is registered beside /api/source/issue. The test compares the two routes' denials case by case, and app-token grants stay path-segment bounded. The gateway writes the scope (repo: first, rows filtered by repository_url; GitLab always reads the one project's endpoint), provider calls pin the allowlisted host, and rows are redacted before and after the strip and byte-capped before caching.

The red checks are cancelled duplicates or an earlier readiness run, each with a newer pass on this head.

@bolichen97
bolichen97 merged commit 48ef6a9 into main Nov 11, 2026
98 of 111 checks passed
@bolichen97
bolichen97 deleted the feat/source-issues-query branch November 11, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants