Skip to content

[Health-1.1] Fix Arista storage leak in generic healthz check test - #6104

Open
arnavanand-lgtm wants to merge 5 commits into
openconfig:mainfrom
arnavanand-lgtm:fix-healthz-cleanup
Open

arnavanand-lgtm wants to merge 5 commits into
openconfig:mainfrom
arnavanand-lgtm:fix-healthz-cleanup

Conversation

@arnavanand-lgtm

Copy link
Copy Markdown
Contributor

Description:
This change addresses a critical storage leak on test devices (specifically Arista EOS) where leftover artifacts from gnoi.Healthz() rapidly consume 100% of the available persistent storage on /mnt/flash/.

RCA / Background:

  • The system_generic_health_check_test issues a Healthz gNOI check which instructs the device to gather system support bundles and diagnostics, saving these large files into device storage.
  • The test natively relies on the gnoiClient.Healthz().Acknowledge() RPC call to notify the device to purge these artifacts once the test completes.
  • However, Arista routers frequently return blank or invalid Healthz Event IDs. Because of this, the standard Acknowledge operation fails, leaving massive tech-support bundles stranded in /mnt/flash/persist/healthz/.
  • Over time, this exhausts 100% of the disk, surfacing as baffling, downstream peripheral failures (e.g., SSH daemons silently failing to save password settings due to no space left on device).

Fix / Update:
Implemented a robust "Manual Sweep" fallback clause directly after the Acknowledge routine. If the primary Acknowledge() RPC fails (due to the missing Event ID), the test will iterate over the returned artifact IDs and issue a direct gnoiClient.File().Remove(...) explicit filesystem RPC to manually sweep the files off the disk, permanently guarding against the storage leak.

Note: The manual sweep is wrapped in a strict vendor check (dut.Vendor() == ondatra.ARISTA) to remain safely vendor-neutral across the wider OpenConfig topology.

@arnavanand-lgtm
arnavanand-lgtm requested a review from a team as a code owner September 29, 2026 06:53
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical storage exhaustion issue on Arista devices caused by failed cleanup of gnoi.Healthz() artifacts. By introducing a manual filesystem sweep as a fallback for failed Acknowledge operations, the change ensures that persistent storage is correctly reclaimed, preventing downstream system failures due to disk space depletion.

Highlights

  • Storage Leak Resolution: Implemented a manual cleanup fallback for Arista devices to ensure health check artifacts are removed when the standard gNOI Acknowledge RPC fails.
  • Vendor-Specific Safety: Added a conditional check for Arista hardware to ensure the manual filesystem sweep only executes on affected devices, maintaining vendor neutrality.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@OpenConfigBot

OpenConfigBot commented Sep 29, 2026 •

Copy link
Copy Markdown

Pull Request Functional Test Report for #6104 / 724766a

Virtual Devices

Device Test Status Test Documentation Job Raw Log
Arista cEOS status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Cisco 8000E status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Cisco XRd status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Juniper ncPTX status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Nokia SR Linux status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Openconfig Lemming status
Health-1.1: pending authorization
Health-1.1: Generic Health Check

Hardware Devices

Device Test Status Test Documentation Raw Log
Arista status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Cisco status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Juniper status
Health-1.1: pending authorization
Health-1.1: Generic Health Check
Nokia status
Health-1.1: pending authorization
Health-1.1: Generic Health Check

Help

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a cleanup mechanism in TestComponentStatus to acknowledge Healthz events and free up flash space on the DUT. If the acknowledgment fails, a fallback manual sweep is executed for Arista devices using the gNOI File system RPC to remove the artifacts. The review feedback highlights two key improvements: first, to prevent potential accidental directory deletion, you should explicitly verify that the artifact ID is not empty before constructing the file path; second, to adhere to the repository style guide, you should replace the hardcoded vendor check (dut.Vendor() == ondatra.ARISTA) with a defined deviation.

t.Logf("Warning: Failed to acknowledge Healthz event %v: %v", ackReq.Id, ackErr)
// Fallback clause for devices that bug out on Acknowledge (e.g. Arista with blank Event IDs).
// We will manually sweep the artifacts directly through gNOI File system RPC.
if dut.Vendor() == ondatra.ARISTA {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the Repository Style Guide, hardcoded vendor checks (like dut.Vendor() == ondatra.ARISTA) should be avoided in test files. Instead, platform-specific workarounds or exceptions should be implemented as deviations.

Please define a new deviation (e.g., ManualHealthzSweepRequired) in internal/deviations/deviations.go and internal/deviations/deviations.textproto, and use it here.

Suggested change
if dut.Vendor() == ondatra.ARISTA {
if deviations.ManualHealthzSweepRequired(dut) {
References
  1. Usage in Tests: Access deviations via deviations.DeviationName(dut). (link)
  2. Avoid using 'dut.Vendor()' for vendor-specific logic or configurations in tests. Instead, use the 'deviations' package to maintain test abstraction and portability across different vendors.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use right deviation and why is this specific to Arista ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @navaneethyv ,

Good question — we used dut.Vendor() == ondatra.ARISTA directly (similar to vendorCoreFilePath in this file) because this is only post-test flash housekeeping using an EOS-specific path (/mnt/flash/persist/healthz) rather than changing any test validation or operational intent.

That said, if you'd prefer tracking the Healthz.Acknowledge fallback formally as a deviation in metadata.proto / metadata.textproto, I'm happy to add it. Let me know which approach you prefer!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants