Skip to content

tracking: upgrade gh-aw MCP server to get real thread IDs for review-responder #114

Open
Open
tracking: upgrade gh-aw MCP server to get real thread IDs for review-responder#114
Labels
soonComing up — act on this when upstream or conditions are ready

Description

@microsasa
microsasa
opened on Mar 15, 2026

Bug

The Review Responder agent cannot resolve review threads because it hallucminates thread node IDs. The GitHub MCP tools don't expose PRRT_ thread node IDs to the agent, so the agent fabricates plausible-looking IDs based on the example format in the tool description. These fake IDs fail when the handler calls the GraphQL resolveReviewThread mutation.

With required_conversation_resolution: true, unresolved threads block auto-merge. This breaks the autonomous pipeline for any PR where Copilot leaves inline comments.

Evidence

Our PRs

PR Responder ran? Replies posted? Threads resolved? Outcome
#91 Yes Yes (1/1) No — resolve failed Manually resolved
#106 Yes Yes (2/2) No — resolve failed Closed, re-dispatched
#112 Yes Yes (3/3) No — resolve failed Stuck

PR #112 Logs (run 23120336057)

Agent emitted 8 safe-output messages in correct order (resolve before push):

Message 1/8: add_labels ✅
Message 2/8: resolve_pull_request_review_thread ❌ — Could not resolve to a node with the global id of 'PRRT_kwDOOIiPrc6djnf7'
Message 3/8: reply_to_pull_request_review_comment ✅
Message 4/8: resolve_pull_request_review_thread ❌ — Could not resolve to a node with the global id of 'PRRT_kwDOOIiPrc6djnf_'
Message 5/8: reply_to_pull_request_review_comment ✅
Message 6/8: reply_to_pull_request_review_comment ✅
Message 7/8: resolve_pull_request_review_thread ❌ (same error pattern)
Message 8/8: push_to_pull_request_branch ✅

Thread ID Comparison

Agent provided: PRRT_kwDOOIiPrc6djnf7 (prefix kwDOOIiPrc)
Actual thread IDs: PRRT_kwDORibwsM50cYP7 (prefix kwDORibwsM)

Completely different base node IDs. The agent fabricated the IDs — they don't correspond to any real objects.

Meanwhile, reply_to_pull_request_review_comment succeeded for all 3 threads. The reply tool uses a different identifier (comment-level ID) that the agent can discover from the MCP tools.

gh-aw's Own Smoke Tests Confirm This

gh-aw's smoke tests consistently skip "Resolve Review Thread" because the agent can't discover thread IDs:

Smoke #20998 (2026-03-14): ⚠️ skipped — could not obtain PRRT node ID via available tools
Smoke #17210 (2026-02-20): ⚠️ Skipped (PRRT_ thread IDs unavailable without GraphQL)
Smoke #19689 (2026-03-05): ⚠️ Skipped — GraphQL not available to fetch PRRT_ node IDs

This has been a known gap for at least a month.

Root Cause Analysis

What the GitHub MCP Server Does

In github/github-mcp-server (pkg/github/pullrequests.go):

  1. get_review_comments method queries review threads via GraphQL:

    type reviewThreadNode struct {
    ID          githubv4.ID        // <-- PRRT_ node ID IS queried
    IsResolved  githubv4.Boolean
    IsOutdated  githubv4.Boolean
    Comments    struct { ... }
}

  2. The response is converted via convertToMinimalReviewThreadsResponse() before being returned to the agent. It's unclear whether this conversion preserves the thread ID field.

  3. The tool description for resolve_pull_request_review_thread says:

    "Get thread IDs from pull_request_read with method get_review_comments."

The Gap

Either:

  • convertToMinimalReviewThreadsResponse() strips the ID field (so the agent never sees it)
  • Or the gh-aw agent doesn't use the MCP server's get_review_comments (uses a different, older tool that doesn't return thread IDs)
  • Or the agent calls the right tool but doesn't parse the thread ID from the response

Whatever the cause, the agent can see thread content (comments, line numbers, file paths) but NOT thread node IDs. It then fabricates IDs based on the PRRT_kwDO... example in the tool description.

Impact

The autonomous pipeline is broken for any PR with Copilot inline comments:

  1. Implementer creates PR ✅
  2. Copilot reviews with inline comments ✅
  3. Review Responder addresses comments, replies ✅
  4. Review Responder fails to resolve threads ❌
  5. Quality Gate approves ✅
  6. Auto-merge blocked by unresolved threads ❌

Every such PR requires manual thread resolution.

Workaround

Manually resolve threads via GraphQL:

# Get real thread IDs
gh api graphql -f query='query { repository(owner:"OWNER",name:"REPO") { pullRequest(number:N) { reviewThreads(first:20) { nodes { id isResolved } } } } }'

# Resolve each
gh api graphql -f query='mutation { resolveReviewThread(input:{threadId:"PRRT_xxx"}) { thread { isResolved } } }'

When to Revisit

Monitor:

  • github/github-mcp-server for changes to convertToMinimalReviewThreadsResponse() or reviewThreadNode serialization
  • github/gh-aw smoke tests — when test 14 "Resolve Review Thread" starts passing instead of being skipped
  • gh-aw releases for changelog mentions of thread resolution fixes

Related

Activity

  1. microsasa commented on Mar 15, 2026

    @microsasa
    microsasa
    on Mar 15, 2026
    OwnerAuthor

    Update: MCP Server Already Supports Thread Resolution

    github/github-mcp-server#1768 — "Add support for resolving/unresolving PR review threads" — was closed as completed on 2026-03-13.

    The issue confirms: get_review_comments already returns thread IDs in the ID field (PRRT_kwDO...). The resolve/unresolve mutations were added.

    The gap is in gh-aw, not the MCP server. gh-aw either uses an older MCP server version that doesn't have this, or its tool layer doesn't expose the MCP server's resolve capability to the agent. gh-aw's own smoke tests still skip thread resolution as of 2026-03-14.

    Investigating whether we can override the MCP server version gh-aw uses.

  2. microsasa commented on Mar 15, 2026

    @microsasa
    microsasa
    on Mar 15, 2026
    OwnerAuthor

    MCP Server Version Analysis

    gh-aw pins ghcr.io/github/github-mcp-server:v0.32.0 (released 2026-03-06).

    The thread resolution feature (PR github/github-mcp-server#1919, closes #1768) merged 2026-03-13 — one week AFTER v0.32.0 was released. The fix is NOT in v0.32.0. It will ship in the next MCP server release.

    Even after a new MCP server release, gh-aw needs to update its pinned version and release a new gh-aw version. Then we need to upgrade our local gh-aw CLI.

    Chain of dependencies

    1. github-mcp-server releases v0.33.0+ (includes resolve thread)
    2. gh-aw updates pinned version from v0.32.0 to v0.33.0+
    3. gh-aw releases new version
    4. We upgrade gh-aw CLI (gh extension upgrade github/gh-aw)
    5. We recompile lock files (gh aw compile)

    Cannot override

    The MCP server version is hardcoded in the compiled lock file — it's a Docker image reference in the workflow YAML. We can't override it without editing lock files (which are auto-generated and overwritten on compile).

    Workarounds (both viable now)

    1. Rescue workflow: Query real thread IDs via gh api graphql, check if replied + outdated, resolve them. Runs as a plain GitHub Actions workflow, not through gh-aw.
    2. Add bash tool to review-responder: Allow gh api graphql so the agent can query real thread IDs during its run instead of hallucinating them.

  3. microsasa commented on Mar 15, 2026

    @microsasa
    microsasa
    on Mar 15, 2026
    OwnerAuthor

    Upstream filed: github/gh-aw#21130 — requesting MCP server upgrade from v0.32.0 to v0.33.0+ to fix thread resolution.

  4. microsasa
    added
    bugSomething isn't working
    on Mar 16, 2026
  5. microsasa
    changed the title [-]bug: review-responder hallucminates thread IDs — resolve_pull_request_review_thread always fails[/-] [+]tracking: upgrade gh-aw MCP server to get real thread IDs for review-responder[/+] on Mar 16, 2026
  6. microsasa
    added 7 commits that reference this issue on Mar 16, 2026

    fix: give review-responder bash tool for real thread ID lookup (#117)

    0776026

    fix: give review-responder bash tool for real thread ID lookup

    8681516

    fix: give review-responder bash tool for real thread ID lookup

    e9ec50b

    fix: give review-responder bash tool for real thread ID lookup

    42c286c

    fix: give review-responder bash tool for real thread ID lookup

    4249dc2

    fix: give review-responder bash tool for real thread ID lookup

    ac0cecf

    fix: instruct review-responder to query real thread IDs via GraphQL

    87c7473
  7. microsasa
    added
    eternal
    and removed
    bugSomething isn't working
    on Mar 16, 2026
  8. microsasa
    added 4 commits that reference this issue on Mar 17, 2026

    refactor: remove thread resolution from review-responder

    d2fba1d

    feat: pipeline orchestrator v1 — thread resolver

    6cb70fd

    feat: pipeline orchestrator v1 — thread resolver

    6e5b39a

    feat: pipeline orchestrator v1 — thread resolver

    0aed4af

  9. microsasa commented on Apr 3, 2026

    @microsasa
    microsasa
    on Apr 3, 2026
    OwnerAuthor

    Workaround implemented

    This issue is effectively worked around by the shared/fetch-review-comments.md pre-fetch component (added in PR #186).

    How it works: Instead of relying on the MCP server to expose thread IDs to the agent, a workflow step runs gh api graphql before the agent starts. It queries all review threads (with real PRRT_ node IDs) via the GraphQL API and writes them to /tmp/gh-aw/review-data/unresolved-threads.json. The agent reads this file instead of calling MCP tools for thread data.

    The responder has been successfully replying to and resolving threads for weeks using this approach. The upstream MCP server fix (exposing thread IDs via get_review_comments) would make the workaround unnecessary, but the pipeline is not blocked by this anymore.

  10. microsasa commented on Apr 3, 2026

    @microsasa
    microsasa
    on Apr 3, 2026
    OwnerAuthor

    Upstream investigation (April 3, 2026)

    github/github-mcp-server — NOT FIXED

    The bug is a two-PR collision:

    • PR #2062 (Feb 23, shipped in v0.32.0): Intentionally dropped the thread ID field from MinimalReviewThread to reduce context size by 14.5%. The field was considered unused.
    • PR #1919 (Mar 13, post-v0.32.0): Added resolve_thread/unresolve_thread tools — which require the PRRT_ ID that #2062 removed.

    Current code: The PRRT_ ID is fetched from GraphQL in reviewThreadNode but stripped in convertToMinimalReviewThread() — the MinimalReviewThread struct in minimal_types.go has no ID field.

    Pending fix: PR #2245 (opened Mar 20, still open/unmerged) is a 2-line fix — adds ID string to MinimalReviewThread and populates it. No new releases since v0.32.0 (Mar 6).

    Related issues:

    • #2229 (Mar 17) — feature request for exposing PRRT_ node IDs
    • #1768 (Jan 8) — original request, closed by #1919

    github/gh-aw — NOT FIXED

    • 30+ releases since March 15 (v0.58.3 → v0.65.7), none mention thread resolution
    • Smoke tests still skip "Resolve Review Thread" — confirmed in runs from March 31 (#23698) and April 2 (#24103)
    • Still pins github-mcp-server:v0.32.0 (the broken version)
    • Issue #21130 requests bumping to v0.33.0+ once the MCP server ships the fix

    What needs to happen upstream

    1. MCP server merges PR #2245 and ships v0.33.0+
    2. gh-aw bumps their pinned MCP server version from v0.32.0 to v0.33.0+
    3. We upgrade gh-aw and remove our shared/fetch-review-comments.md workaround

    We're at least two upstream releases away from this being fixed natively. Our pre-fetch workaround remains necessary.

  11. microsasa
    added
    soonComing up — act on this when upstream or conditions are ready
    and removed
    eternal
    on Apr 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    soonComing up — act on this when upstream or conditions are ready

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions