Skip to content

Remove NRL docs and Perflab baseline reporting - #2432

Merged
jioffe502 merged 5 commits into
mainfrom
jioffe502/remove-legacy-benchmark-harness
Aug 19, 2026
Merged

Remove NRL docs and Perflab baseline reporting#2432
jioffe502 merged 5 commits into
mainfrom
jioffe502/remove-legacy-benchmark-harness

Conversation

@jioffe502

@jioffe502 jioffe502 commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • remove the NRL harness README, guides, design-history page, nightly README, and BO767 walkthrough
  • remove the Perflab/release-baseline loader and Slack side-by-side comparison block
  • remove the --reference-file and release_references compatibility surfaces entirely
  • remove the associated baseline test fixture and nightly configuration example
  • keep the harness runner, benchmark runfiles, artifact/history/portal flows, nightly launcher, and ordinary Slack posting intact

Why

Perflab is the canonical home for performance results and comparisons. This removes the ad hoc baseline presentation from NRL while preserving the harness implementation for the separate code decision.

Validation

  • git diff --check passes
  • focused Slack tests pass: 12 passed
  • focused Slack tests pass: 12 passed
  • no Perflab, release-reference, or reference-file integration remains in the repository
  • replacement benchmarking docs link to the internal NRB repository
  • the harness runner and nightly orchestration remain in the tree
  • the full nightly launcher test file cannot run on this macOS host because its unchanged Bash 4+ [[ -v "$name" ]] construct is rejected by system Bash 3.2

@jioffe502
jioffe502 force-pushed the jioffe502/remove-legacy-benchmark-harness branch from 3df166a to f6ffc92 Compare August 18, 2026 18:39
@jioffe502 jioffe502 changed the title Remove legacy benchmark harness after NRB migration Remove legacy NRL harness documentation Aug 18, 2026
@jioffe502 jioffe502 changed the title Remove legacy NRL harness documentation Remove NRL docs and Perflab baseline reporting Aug 18, 2026
@jioffe502
jioffe502 marked this pull request as ready for review August 18, 2026 19:20
@jioffe502
jioffe502 requested review from a team as code owners August 18, 2026 19:20
@greptile-apps

greptile-apps Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR removes the local NRL harness documentation and Perflab release-reference reporting while retaining benchmark execution, nightly orchestration, artifacts, and ordinary Slack reporting.

  • Deletes the release-reference loader, CLI option, environment configuration, Slack comparison block, and associated tests.
  • Replaces local harness documentation with links to the separate NRB repository.
  • Removes obsolete guides, walkthroughs, expected-result documentation, and nightly documentation.

Confidence Score: 3/5

The PR should not merge until the replacement benchmark documentation is reachable by its intended audience and the outstanding public-interface compatibility violation is resolved.

Public documentation now points exclusively to an internal GitLab repository, while the CLI and Python reference-reporting inputs remain removed without the compatibility treatment required by the repository rules.

Files Needing Attention: nemo_retriever/README.md, nemo_retriever/docs/cli/benchmarking.md, nemo_retriever/src/nemo_retriever/harness/cli.py, nemo_retriever/src/nemo_retriever/harness/slack.py

Important Files Changed

Filename Overview
nemo_retriever/README.md Replaces deleted local harness guides with an internal GitLab link that is inaccessible to the public README audience.
nemo_retriever/docs/cli/benchmarking.md Redirects end-to-end benchmarking readers to the same internal-only NRB repository.
nemo_retriever/src/nemo_retriever/harness/cli.py Removes reference-file loading and the public CLI option; the previously reported rule-governed compatibility break remains outstanding.
nemo_retriever/src/nemo_retriever/harness/slack.py Removes release-reference rendering and public function parameters while preserving ordinary Slack payload generation.
ops/retriever-nightly/run-nightly.sh Removes release-reference configuration propagation while retaining nightly execution and ordinary Slack posting.
Prompt To Fix All With AI
### Issue 1
nemo_retriever/README.md:834-835
**Replacement benchmark link is inaccessible**

If a reader lacks access to NVIDIA's internal `gitlab-master.nvidia.com` domain, the sole replacement link for the deleted harness guides cannot be opened, leaving that reader without a usable path to the canonical end-to-end benchmark documentation.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (5): Last reviewed commit: "Merge upstream/main into legacy harness ..." | Re-trigger Greptile

Comment thread nemo_retriever/src/nemo_retriever/harness/cli.py
Comment thread nemo_retriever/README.md Outdated
Comment thread nemo_retriever/src/nemo_retriever/harness/cli.py
@jioffe502
jioffe502 merged commit 0f4a58a into main Aug 19, 2026
8 checks passed
Comment thread nemo_retriever/README.md
Comment on lines +834 to +835
End-to-end Retriever experiments and benchmark orchestration are maintained in
the [NeMo Retriever Benchmark (NRB) repository](https://gitlab-master.nvidia.com/charlesb/nemo-retriever-benchmark/).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Replacement benchmark link is inaccessible

If a reader lacks access to NVIDIA's internal gitlab-master.nvidia.com domain, the sole replacement link for the deleted harness guides cannot be opened, leaving that reader without a usable path to the canonical end-to-end benchmark documentation.

Rule Used: When this PR changes user-facing code, configurati... (source)

Knowledge Base Used: Benchmark Harness

Prompt To Fix With AI
This is a comment left during a code review.
Path: nemo_retriever/README.md
Line: 834-835

Comment:
**Replacement benchmark link is inaccessible**

If a reader lacks access to NVIDIA's internal `gitlab-master.nvidia.com` domain, the sole replacement link for the deleted harness guides cannot be opened, leaving that reader without a usable path to the canonical end-to-end benchmark documentation.

**Rule Used:** When this PR changes user-facing code, configurati... ([source](.greptile))

**Knowledge Base Used:** [Benchmark Harness](https://app.greptile.com/nvidia-public-github/-/custom-context/knowledge-base/nvidia/nemo-retriever/-/docs/benchmark-harness.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants