Skip to content

miner(discover): supplementWithDiscoveryIndex's dedup set is never updated during its own filter pass #10334

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

packages/loopover-miner/lib/discover-cli.ts's supplementWithDiscoveryIndex (around line 190)
merges hosted discovery-index candidates into the local fan-out results, dropping any candidate
that duplicates an issue already present in fanOut.issues:

const seen = new Set(fanOut.issues.map((issue) => dedupeKey(issue.repoFullName, issue.issueNumber)));
const supplemented = aiAllowed
  .filter((candidate) => !seen.has(dedupeKey(candidate.repoFullName, candidate.issueNumber)))
  .map((candidate) => ({ ...candidate, assignees: [...(candidate.assignees ?? [])], labels: [...candidate.labels] }) as RawCandidateIssue);
if (supplemented.length === 0) return fanOut;
return { ...fanOut, issues: [...fanOut.issues, ...supplemented] };

seen is built exactly once, from fanOut.issues, before the .filter() call, and is never added
to during that filter. The filter therefore only guards against a candidate duplicating something
already in the LOCAL fan-out — it does nothing to catch two candidates that duplicate each
other
within the SAME discovery-index response. If the hosted discovery index ever returns two
entries for the same (repoFullName, issueNumber) pair (a real possibility for a hosted, shared,
eventually-consistent index that this file's own doc comments already acknowledge as a
best-effort/fail-open source), both pass the filter, both get appended to fanOut.issues, and the
run proceeds with a duplicated candidate.

This has concrete downstream effects: recordDiscoveryTelemetry's droppedAiBanned count and the
overall fanOut.issues.length become inflated, and every later stage that keys off issue identity
(ranking, signal-tracking writes via contribution-profile-cache, portfolio-queue enqueue) sees
the same (repo, issue) pair twice in one run.

Requirements

  • In supplementWithDiscoveryIndex, update the seen set as candidates are accepted during the
    filter/map pass, so a duplicate within aiAllowed itself is caught the same way a duplicate
    against fanOut.issues already is. Do not change the function's existing "local results always
    win" precedence for candidates that duplicate fanOut.issues — this issue is only about
    duplicates within the hosted response itself.
  • Preserve the existing behavior for every other branch in this function (the
    isDiscoveryPlaneEnabled early return, the aiPolicyAllowed !== false filter and its
    droppedAiBanned telemetry, the supplemented.length === 0 early return) exactly as-is.

Deliverables

  • supplementWithDiscoveryIndex no longer appends two entries for the same
    (repoFullName, issueNumber) pair when the hosted discovery-index response itself contains a
    duplicate.
  • A new regression test in the existing discover-cli test file asserting that a mocked
    queryDiscoveryIndex response containing two candidates with the same repoFullName +
    issueNumber results in exactly one entry being appended to fanOut.issues, and that a
    duplicate of an existing fanOut.issues entry is still dropped in favor of the local result
    (the pre-existing behavior, to guard against a regression while fixing the new case).

Test Coverage Requirements

This repo's Codecov patch gate is 99%+ (branch-counted) on packages/loopover-miner/lib/**. Add
the new regression test under test/unit/** (this package's tests live in the shared root test/
directory; find and extend the existing discover-cli test file rather than creating a new one). The
new branch you add to update seen during the filter/map pass must be exercised by the new test.

Expected Outcome

A hosted discovery-index response containing an internal duplicate produces exactly one candidate
in the fan-out output, matching the same dedup guarantee supplementWithDiscoveryIndex already
provides against duplicates of the local fan-out.

Links & Resources

  • packages/loopover-miner/lib/discover-cli.tssupplementWithDiscoveryIndex (~line 190),
    dedupeKey (the existing helper to reuse).

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions