Repository navigation
[RT-7.9] Fix ECMP load balancing test flakiness by randomizing UDP ports in multipath_bgp_ecmp_protocol_nexthop - #6173
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses intermittent test failures in the BGP ECMP load balancing test (RT-7.9) caused by insufficient entropy in the generated traffic flows. By randomizing UDP port selection and slightly relaxing the packet loss tolerance, the test now achieves stable and balanced load distribution across multiple paths on hardware testbeds. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Pull Request Functional Test Report for #6173 / ecf3d5cVirtual Devices
Hardware Devices
|
There was a problem hiding this comment.
Code Review
This pull request updates the BGP ECMP test by adjusting the packet loss tolerance percentage and randomizing the UDP source and destination ports using a new randRange helper function. Feedback on the changes points out that instantiating a new rand.Source seeded with the current time inside randRange can cause identical port sequences when called in rapid succession. It is recommended to use the global rand generator instead to ensure proper randomization.
|
Hi @AmrNJ @navaneethyv @jayakumarramisetty, could you please help review this PR or assign someone who can take a look? Thanks! |
Summary
This pull request updates
multipath_bgp_ecmp_protocol_nexthop_test(RT-7.9) by randomizing both source and destination UDP port keys in the traffic generator to ensure adequate 5-tuple entropy for ECMP hash distribution, and adjusts the packet loss tolerance to accommodate traffic ramp-up.Problem & Motivation
In
multipath_bgp_ecmp_protocol_nexthop_test(Test PlanRT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop), the test asserts balanced ECMP load distribution across 3 nexthop paths within strict tolerance bounds (lbToleranceFms = 20, representing ±3% allowable deviation from ideal).The traffic generator in
configureOTG()was previously generating flows using a static source port (SrcPort = 14) and sequentially incremented destination ports (1024..11024). Using a fixed source port combined with sequential destination ports creates low entropy across packet 5-tuples, leading to hash polarization on hardware switches. This resulted in an unbalanced traffic distribution across paths (e.g., one path receiving ~21% instead of ~33%), causing the test to intermittently fail:Detailed Changes
Port Key Randomization:
randRange(1024, 65535, 1000)for both IPv4 and IPv6 traffic flows.randRange(start, end uint32, count int) []uint32helper function to generate non-correlated, distributed port values.Traffic Loss Tolerance:
lossTolerancePctfrom 0% to 0.5% to account for transient packet buffer drops during traffic generator ramp-up, while keeping the strict load balance tolerance (lbToleranceFms = 20) intact.Verification
RT-7.9 Pass 1.log
RT-7.9 Pass 2.log
RT-7.9 Pass 3.log