Repository navigation
[Health-1.1] Fix Arista storage leak in generic healthz check test - #6104
arnavanand-lgtm wants to merge 5 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical storage exhaustion issue on Arista devices caused by failed cleanup of gnoi.Healthz() artifacts. By introducing a manual filesystem sweep as a fallback for failed Acknowledge operations, the change ensures that persistent storage is correctly reclaimed, preventing downstream system failures due to disk space depletion. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Pull Request Functional Test Report for #6104 / 724766aVirtual Devices
Hardware Devices
|
There was a problem hiding this comment.
Code Review
This pull request introduces a cleanup mechanism in TestComponentStatus to acknowledge Healthz events and free up flash space on the DUT. If the acknowledgment fails, a fallback manual sweep is executed for Arista devices using the gNOI File system RPC to remove the artifacts. The review feedback highlights two key improvements: first, to prevent potential accidental directory deletion, you should explicitly verify that the artifact ID is not empty before constructing the file path; second, to adhere to the repository style guide, you should replace the hardcoded vendor check (dut.Vendor() == ondatra.ARISTA) with a defined deviation.
| t.Logf("Warning: Failed to acknowledge Healthz event %v: %v", ackReq.Id, ackErr) | ||
| // Fallback clause for devices that bug out on Acknowledge (e.g. Arista with blank Event IDs). | ||
| // We will manually sweep the artifacts directly through gNOI File system RPC. | ||
| if dut.Vendor() == ondatra.ARISTA { |
There was a problem hiding this comment.
According to the Repository Style Guide, hardcoded vendor checks (like dut.Vendor() == ondatra.ARISTA) should be avoided in test files. Instead, platform-specific workarounds or exceptions should be implemented as deviations.
Please define a new deviation (e.g., ManualHealthzSweepRequired) in internal/deviations/deviations.go and internal/deviations/deviations.textproto, and use it here.
| if dut.Vendor() == ondatra.ARISTA { | |
| if deviations.ManualHealthzSweepRequired(dut) { |
References
- Usage in Tests: Access deviations via deviations.DeviationName(dut). (link)
- Avoid using 'dut.Vendor()' for vendor-specific logic or configurations in tests. Instead, use the 'deviations' package to maintain test abstraction and portability across different vendors.
There was a problem hiding this comment.
Use right deviation and why is this specific to Arista ?
There was a problem hiding this comment.
Hi @navaneethyv ,
Good question — we used dut.Vendor() == ondatra.ARISTA directly (similar to vendorCoreFilePath in this file) because this is only post-test flash housekeeping using an EOS-specific path (/mnt/flash/persist/healthz) rather than changing any test validation or operational intent.
That said, if you'd prefer tracking the Healthz.Acknowledge fallback formally as a deviation in metadata.proto / metadata.textproto, I'm happy to add it. Let me know which approach you prefer!
Description:
This change addresses a critical storage leak on test devices (specifically Arista EOS) where leftover artifacts from
gnoi.Healthz()rapidly consume 100% of the available persistent storage on/mnt/flash/.RCA / Background:
system_generic_health_check_testissues aHealthzgNOI check which instructs the device to gather system support bundles and diagnostics, saving these large files into device storage.gnoiClient.Healthz().Acknowledge()RPC call to notify the device to purge these artifacts once the test completes.Acknowledgeoperation fails, leaving massive tech-support bundles stranded in/mnt/flash/persist/healthz/.Fix / Update:
Implemented a robust "Manual Sweep" fallback clause directly after the
Acknowledgeroutine. If the primaryAcknowledge()RPC fails (due to the missing Event ID), the test will iterate over the returned artifact IDs and issue a directgnoiClient.File().Remove(...)explicit filesystem RPC to manually sweep the files off the disk, permanently guarding against the storage leak.Note: The manual sweep is wrapped in a strict vendor check (
dut.Vendor() == ondatra.ARISTA) to remain safely vendor-neutral across the wider OpenConfig topology.