Skip to content

[RT-7.9] Fix ECMP load balancing test flakiness by randomizing UDP ports in multipath_bgp_ecmp_protocol_nexthop - #6173

Open
arnavanand-lgtm wants to merge 6 commits into
openconfig:mainfrom
arnavanand-lgtm:fix-bgp-ecmp-port-randomization
Open

arnavanand-lgtm wants to merge 6 commits into
openconfig:mainfrom
arnavanand-lgtm:fix-bgp-ecmp-port-randomization

Conversation

@arnavanand-lgtm

Copy link
Copy Markdown
Contributor

Summary

This pull request updates multipath_bgp_ecmp_protocol_nexthop_test (RT-7.9) by randomizing both source and destination UDP port keys in the traffic generator to ensure adequate 5-tuple entropy for ECMP hash distribution, and adjusts the packet loss tolerance to accommodate traffic ramp-up.

Problem & Motivation

In multipath_bgp_ecmp_protocol_nexthop_test (Test Plan RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop), the test asserts balanced ECMP load distribution across 3 nexthop paths within strict tolerance bounds (lbToleranceFms = 20, representing ±3% allowable deviation from ideal).

The traffic generator in configureOTG() was previously generating flows using a static source port (SrcPort = 14) and sequentially incremented destination ports (1024..11024). Using a fixed source port combined with sequential destination ports creates low entropy across packet 5-tuples, leading to hash polarization on hardware switches. This resulted in an unbalanced traffic distribution across paths (e.g., one path receiving ~21% instead of ~33%), causing the test to intermittently fail:

Traffic is expected in range 303333 - 363333 but got 214000. Load balance Test Failed

Detailed Changes

Port Key Randomization:

  • Replaced fixed source port (14) and sequential destination ports (1024..11024) with randomized port sets using randRange(1024, 65535, 1000) for both IPv4 and IPv6 traffic flows.
  • Added a randRange(start, end uint32, count int) []uint32 helper function to generate non-correlated, distributed port values.

Traffic Loss Tolerance:

  • Adjusted lossTolerancePct from 0% to 0.5% to account for transient packet buffer drops during traffic generator ramp-up, while keeping the strict load balance tolerance (lbToleranceFms = 20) intact.

Verification

  • Verified across 3 consecutive test runs on hardware testbeds.
  • Achieved balanced load distribution across all 3 paths (~33.35% / ~33.35% / ~33.30%) with 0% packet loss and a 100% pass rate.

RT-7.9 Pass 1.log
RT-7.9 Pass 2.log
RT-7.9 Pass 3.log

@arnavanand-lgtm
arnavanand-lgtm requested a review from a team as a code owner October 16, 2026 06:59
@arnavanand-lgtm
arnavanand-lgtm requested a review from AmrNJ October 16, 2026 06:59
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses intermittent test failures in the BGP ECMP load balancing test (RT-7.9) caused by insufficient entropy in the generated traffic flows. By randomizing UDP port selection and slightly relaxing the packet loss tolerance, the test now achieves stable and balanced load distribution across multiple paths on hardware testbeds.

Highlights

  • Traffic Entropy Improvement: Replaced static source and sequential destination UDP ports with randomized port sets to improve 5-tuple entropy and prevent hash polarization in ECMP load balancing.
  • Packet Loss Tolerance Adjustment: Increased the packet loss tolerance from 0% to 0.5% to account for transient buffer drops during traffic generator ramp-up.
  • Helper Function Addition: Introduced a new randRange helper function to generate non-correlated, distributed port values for traffic flows.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@OpenConfigBot

OpenConfigBot commented Oct 16, 2026 •

Copy link
Copy Markdown

Pull Request Functional Test Report for #6173 / ecf3d5c

Virtual Devices

Device Test Status Test Documentation Job Raw Log
Arista cEOS status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Cisco 8000E status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Cisco XRd status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Juniper ncPTX status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Nokia SR Linux status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Openconfig Lemming status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop

Hardware Devices

Device Test Status Test Documentation Raw Log
Arista status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Cisco status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Juniper status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop
Nokia status
RT-7.9: pending authorization
RT-7.9: BGP ECMP for iBGP with IS-IS protocol nexthop

Help

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the BGP ECMP test by adjusting the packet loss tolerance percentage and randomizing the UDP source and destination ports using a new randRange helper function. Feedback on the changes points out that instantiating a new rand.Source seeded with the current time inside randRange can cause identical port sequences when called in rapid succession. It is recommended to use the global rand generator instead to ensure proper randomization.

@arnavanand-lgtm

Copy link
Copy Markdown
Contributor Author

Hi @AmrNJ @navaneethyv @jayakumarramisetty, could you please help review this PR or assign someone who can take a look? Thanks!

@jayakumarramisetty jayakumarramisetty self-assigned this Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants