Skip to content

Keysight AI DCB collective communications - #491

Open
lthompson-keysight wants to merge 2 commits into
open-traffic-generator:masterfrom
Keysight:keysight-ccb
Open

lthompson-keysight wants to merge 2 commits into
open-traffic-generator:masterfrom
Keysight:keysight-ccb

Conversation

@lthompson-keysight

@lthompson-keysight lthompson-keysight commented Sep 3, 2026 •

Copy link
Copy Markdown

Vendor-agnostic collective communication benchmarking model.

Feature Overview

  • Related Issue: N/A
  • Brief Description:
    Add support for benchmarking collective communications systems
    Generated artifacts are built by CI from the sources in this commit.

Feature Details

config body: /collective_communications
set_control_state body: /collective_communications
get_state body: /collective_communications
get_metrics body: /collective_communications


Code snippets

"""
Basic sample for running a collective benchmark trial via the OTG API.
"""
"""
Basic sample for running a collective benchmark trial via the OTG API.
"""

import time
from typing import Literal

import pandas as pd
from pandas.core.frame import DataFrame

import snappi
from snappi import (
    Api,
    CollectiveCommunicationsStateStopped,
    Config,
    MetricsResponse,
)

from keysight_dse_common.schemas.summary_metrics_schema import (
    SummaryMetricsColumnIds,
)

# -------------- BEGIN Implementation-specific ---------------

CHASSIS_ADDRESS = "1.2.3.4"
PORTS_PER_CARD = 4
# Chassis-scheme port locations select the Keysight hardware platform.
CHASSIS_LOCATION_SCHEME = "com.keysight.chassis:"


def impl_rank_to_location(rank_id: int) -> str:
    card_index, port_index = divmod(rank_id, PORTS_PER_CARD)
    return (
        f"{CHASSIS_LOCATION_SCHEME}{CHASSIS_ADDRESS};{card_index + 1};{port_index + 1}"
    )


# -------------- END: Implementation-specific ---------------

# Metric types selectable under the single collective application.
SingleCollectiveMetric = Literal[
    "summary",
    "data_chunk",
    "flows",
    "qpairs",
    "ports",
    "aggregation_of_iterations",
    "iteration",
    "packet_drop",
]

DSE_SERVER_ADDRESS = "localhost:50001"
NUM_HOSTS = 4
XPUS_PER_HOST = 8
IP_ADDRESS_TEMPLATE = "32.0.{rank}.1"
IP_GATEWAY_TEMPLATE = "32.0.{rank}.254"
IP_PREFIX = 24

# Workload-level RoCEv2 transport settings, applied uniformly to every rank
# pair using this transport.
ROCEV2_RDMA_MESSAGE_SIZE = 1 << 17  # 128 KiB
ROCEV2_VERB = "write"
ROCEV2_REUSE_QPS = True
TRANSPORT_QPS_PER_RANK_PAIR = 1

SUMMARY_METRICS_CSV = "summary_metrics.csv"
PORT_METRICS_CSV = "port_metrics.csv"


def set_up_infrastructure(mut_config: Config) -> None:
    infra = mut_config.collective_communications.infrastructure
    infra.choice = infra.SIMPLE_IMPLICIT

    rank_id = 0
    for host_index in range(NUM_HOSTS):
        device = mut_config.devices.add()
        device.name = f"host{host_index}-device"

        for xpu_index in range(XPUS_PER_HOST):
            # Register test chassis port that will be used for this rank's traffic
            port = mut_config.ports.add()
            port.name = f"host{host_index}-nic{xpu_index}-port"
            # port location is used by the implementation to identify
            # both the platform and the specific port within that platform
            port.location = impl_rank_to_location(rank_id)

            # Configure the IPv4 settings for the port
            ethernet = device.ethernets.add()
            ethernet.name = f"host{host_index}-nic{xpu_index}-eth"
            ethernet.connection.choice = ethernet.connection.PORT_NAME
            ethernet.connection.port_name = port.name
            ipv4 = ethernet.ipv4_addresses.add()
            ipv4.name = f"{ethernet.name}-ipv4"
            ipv4.address = IP_ADDRESS_TEMPLATE.format(rank=rank_id)
            ipv4.prefix = IP_PREFIX
            ipv4.gateway = IP_GATEWAY_TEMPLATE.format(rank=rank_id)
            # RoCEv2 runs over this address: reference it from the host's RoCEv2 stack
            device.rocev2.ipv4_interfaces.add(ipv4_name=ipv4.name)

            rank_id += 1

    # Configure Layer 1 settings for all ports
    layer1 = mut_config.layer1.add()
    layer1.name = "all-ports-l1"
    layer1.port_names = [port.name for port in mut_config.ports]
    layer1.speed = layer1.SPEED_400_GBPS
    layer1.signaling = layer1.PAM4_106_GBPS
    layer1.fec_mode = layer1.RS_FEC_KP4


def set_up_benchmark(mut_config: Config) -> None:
    cc = mut_config.collective_communications

    single_collective = cc.workload.single_collective

    single_collective.collective_algorithm.algorithm_type = (
        single_collective.collective_algorithm.ALL_REDUCE_VECTOR_HALVING_DOUBLING
    )

    single_collective.datasizes.list.size_bytes = [
        1 << 27,
        1 << 30,
        1 << 32,
    ]  # 128 MiB, 1 GiB, 4 GiB

    single_collective.iterations = 10

    rocev2 = single_collective.transport.rocev2
    rocev2.rdma_message_size = ROCEV2_RDMA_MESSAGE_SIZE
    rocev2.verb = rocev2.WRITE
    rocev2.reuse_qps = ROCEV2_REUSE_QPS
    rocev2.qps_per_rankpair = TRANSPORT_QPS_PER_RANK_PAIR


def run_and_wait_for_result(
    api: Api, poll_interval_s: float = 2.0
) -> CollectiveCommunicationsStateStopped:
    control_state = api.control_state()
    execution = control_state.collective_communications.execution
    execution.state = execution.START
    api.set_control_state(control_state)

    states_request = api.states_request()
    states_request.collective_communications.choice = (
        states_request.collective_communications.EXECUTION
    )

    while True:
        ai_state = api.get_states(states_request).collective_communications.execution
        if (
            ai_state.choice == ai_state.STOPPED
            and ai_state.stopped.choice != ai_state.stopped.UNRUN
        ):
            return ai_state.stopped
        time.sleep(poll_interval_s)


def print_result(stopped: CollectiveCommunicationsStateStopped) -> None:
    match stopped.choice:
        case stopped.SUCCESS:
            warnings = stopped.success.warnings or []
            print(f"Run succeeded with {len(warnings)} warning(s):")
            for warning in warnings:
                print(f"  - {warning}")
        case stopped.FAILURE:
            print("Run failed:")
            for warning in stopped.failure.errors or []:
                print(f"  - {warning}")
        case stopped.MANUAL:
            print("Run was stopped manually.")
        case _:
            print(f"Run stopped with unexpected reason: {stopped.choice}")


def fetch_collective_communications_metrics(
    api: Api,
    metric_selection: SingleCollectiveMetric,
    column_names: list[str],
    pagination_start: int,
    pagination_count: int,
) -> MetricsResponse:
    """Fetch collective benchmark metrics of the selected type for the given page of rows."""
    request = api.metrics_request()
    request.choice = request.COLLECTIVE_COMMUNICATIONS

    cc = request.collective_communications
    cc.choice = cc.SINGLE_COLLECTIVE

    single_collective = cc.single_collective
    single_collective.choice = metric_selection
    getattr(single_collective, metric_selection).column_names = column_names

    request.pagination.start = pagination_start
    request.pagination.count = pagination_count

    return api.get_metrics(request)


def fetch_port_metrics(
    api: Api,
    column_names: list[str],
    pagination_start: int,
    pagination_count: int,
) -> MetricsResponse:
    """Fetch single-collective port metrics for the given page of rows."""
    request = api.metrics_request()

    ports = request.collective_communications.single_collective.ports
    ports.column_names = column_names

    request.pagination.start = pagination_start
    request.pagination.count = pagination_count

    return api.get_metrics(request)


# Maps each CollectiveCommunications.Summary.Metric field to its summary schema column id.
SUMMARY_FIELD_TO_COLUMN_ID = {
    "data_size_bytes": SummaryMetricsColumnIds.data_size,
    "collective": SummaryMetricsColumnIds.collective,
    "iterations": SummaryMetricsColumnIds.iterations,
    "avg_collective_time_ms": SummaryMetricsColumnIds.completion_time,
    "algo_bw_gbps": SummaryMetricsColumnIds.algo_bw,
    "bus_bw_gbps": SummaryMetricsColumnIds.bus_bw,
    "ideal_percentage": SummaryMetricsColumnIds.ideal,
    "pfc_rx": SummaryMetricsColumnIds.pfc_fx,
    "ecn_ce_rx": SummaryMetricsColumnIds.ecn_ce_rx,
    "frames_retx": SummaryMetricsColumnIds.frames_retx,
    "total_time_s": SummaryMetricsColumnIds.total_time,
    "duty_cycle_percentage": SummaryMetricsColumnIds.duty_cycle,
}


def summary_metrics_to_dataframe(response: MetricsResponse) -> pd.DataFrame:
    """Convert the collective benchmark summary metrics in a response into a DataFrame
    whose columns are the summary schema display names (SummaryMetricsColumnIds)."""
    summary_metrics = (
        response.collective_communications_metrics.single_collective.summary
    )

    schema_columns = [
        column_id.value for column_id in SUMMARY_FIELD_TO_COLUMN_ID.values()
    ]
    rows = [
        {
            column_id.value: getattr(summary_metric, field_name)
            for field_name, column_id in SUMMARY_FIELD_TO_COLUMN_ID.items()
        }
        for summary_metric in summary_metrics
    ]
    return pd.DataFrame(rows, columns=schema_columns)


def port_metrics_to_dataframe(response: MetricsResponse) -> pd.DataFrame:
    """Convert the single-collective port metrics in a response into a DataFrame with one
    row per port and one column per returned metric field."""
    port_metrics = response.collective_communications_metrics.single_collective.ports
    return pd.DataFrame(
        [port_metric.serialize(port_metric.DICT) for port_metric in port_metrics]
    )


def main() -> None:
    api = snappi.api(location=DSE_SERVER_ADDRESS, transport="grpc")

    config = api.config()
    set_up_infrastructure(config)
    set_up_benchmark(config)
    api.set_config(config)

    print_result(run_and_wait_for_result(api))

    summary_metrics_response = fetch_collective_communications_metrics(
        api,
        metric_selection="summary",
        column_names=[],
        pagination_start=0,
        pagination_count=100,
    )
    summary_dataframe: DataFrame = summary_metrics_to_dataframe(
        summary_metrics_response
    )
    summary_dataframe.to_csv(SUMMARY_METRICS_CSV, index=False)

    port_metrics_response: MetricsResponse = fetch_port_metrics(
        api,
        column_names=["port", "frames_tx", "frames_rx", "bytes_tx", "bytes_rx"],
        pagination_start=0,
        pagination_count=100,
    )
    port_dataframe: DataFrame = port_metrics_to_dataframe(port_metrics_response)
    port_dataframe.to_csv(PORT_METRICS_CSV, index=False)


if __name__ == "__main__":
    main()

Comment thread collective_communications/workload.yaml Resolved
Comment thread device/rocev2/qps.yaml Resolved
@lthompson-keysight lthompson-keysight self-assigned this Sep 9, 2026
Comment thread port/rocev2.yaml Resolved
@lthompson-keysight
lthompson-keysight marked this pull request as draft September 10, 2026 00:21
@lthompson-keysight
lthompson-keysight marked this pull request as ready for review September 10, 2026 00:22
@lthompson-keysight
lthompson-keysight marked this pull request as draft September 10, 2026 00:23
@ajbalogh

Copy link
Copy Markdown
Contributor

I tried to view the changes as a whole using the redocly link in this conversation and didn't see any changes.

@apratimmukherjee

Copy link
Copy Markdown
Contributor

I have updated the Redocli link to https://redocly.github.io/redoc/?url=https://raw.githubusercontent.com/Keysight/keysight-ccb/refs/heads/keysight-ccb/artifacts/openapi.yaml&nocors#tag/Configuration . It should now show the tree view .
Took a quick look at changes in the common areas , at first glance most look like things we can handle/reject
e.g. emulated interface having ability to be bound to multiple ports [ bound to a LAG with multiple member ports is what we normally do similar to this ] which looks little counter-intuitive to me for normal routing scenarios -> but we should be able to simply reject that with an error in implementations where it will not make sense . Might need little more details in such areas to inform the user that this choice is for what purpose and when it should be rejected as an invalid choice by the impl. ( e.g any L2/L3 scenario ) . Will try to look in more detail later .

@lthompson-keysight

lthompson-keysight commented Nov 19, 2026 •

Copy link
Copy Markdown
Author

Changes made for addressing PR review #2

(note: any mention of 'removal' refers to removing/canceling a proposed addition. We're not removing
anything that was already in the upstream OTG API)

General

  • Switched back to using contiguous field IDs instead of reserving spaces like 100-200 etc.
  • Removed miscellaneous unreferenced schemas that had been let in accidentally
  • Removed infrastructure related models
  • Removed impairments

Base OTG Model

  • Improved and clarified aspects of L1 configuration:
    • PFC: added the enabled_pfc_priorities field and clarified the meaning of the pfc_class_X fields
    • FEC: clarified how the fec_mode field interacts with the legacy auto_negotiation.rs_fec setting
    • Clarified default behavior for signaling and FEC mode
  • Metrics pagination:
    • Added explicit defaults for pagination parameters (start, count etc.)
    • Clarified behavior for legacy clients that do not support pagination
  • Clarified error and unknown key handling for CustomSettings
  • Removed rule on forbidding leading/trailing whitespace for object names
  • Removed explicit limit for capture_max_file_size
  • Removed Ethernet.Connection.port_names

Collective Communications

  • Simplifications and removals:
    • Removed placeholders for features not part of the current PR (workload emulation, generate_config)
    • Simplified Algorithm structure and removed custom algorithm support for now
    • Removed 'TCP' QP negotiation option
    • Removed Impairments
    • Infrastructure
      • Removed generate_bindings
      • Removed PlatformConfig and CollectiveCommunications.platforms
      • Added custom_settings to NicBinding
      • Removed low level infrastructure choice
  • Control:
    • Corrected awkward definition of get_states request
    • Clarified start/stop and other lifecycle semantics
  • Process groups:
    • Many clarifications regarding how process groups are defined and function:
      • Clarified that rank order matters for collective algorithms
      • Clarified which is the order used and how it can be controlled
      • Clarified how the root rank is chosen for tree algorithms
      • Clarified process group overlap and scoping limitations
    • Simplified declaration of ranges
    • Added "flow generation scope" as a separate but related concept for background flow and point-to-point
      flow generation. Clarified the extent of the "default" or "maximal" scopes.
  • Transport settings:
    • Remodel/rename and document Transport.choice so that it reads as transport global settings rather than a selector; remove TCP value
  • Single collective:
    • Made the definition of the inter-iteration delay consistent to those for mix-of-collectives in terms
      of unit, defaults and limits. Notably the base unit is now microseconds instead of ms.
  • Mix of Collectives:
    • Added name fields for workloads and operations
    • Added workload, workload_iteration and operation columns to the results to ensure that all rows can
      be matched to their origin in the workload_set->workload->operation structure
    • Added workload_iteration rollup table, to fit between the workload_set_iteration and operation_iteration
      rollups
  • Metrics:
    • Removed query filters pending a redesign
    • Clarified unit and description for GBps values
    • Clarified definition of "ideal percentage" and made the definition uniform across single-collective/MOC
    • Clarified various formulas and scopes of aggregation
    • Clarified meaning and reference of timestamps
    • Remove Source NIC, Destination NIC from several metrics
  • Background traffic
    • Renamed line_rate_percent to line_rate_percentage
    • Fixed description of line_rate_percentage to refer to port speed instead of NIC speed

@apratimmukherjee apratimmukherjee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few possible points for cleanup:

  • Field UIDs 101/102 are still used on shared schemas. The change list says contiguous IDs were restored, but these existing schemas still use 101/102. These numbers become permanent once released, so please use the next free uid:

Config.collective_communications / Config.artifacts: 101/102 → 12/13
Capture.max_file_size: 101 → 7
Rocev2.PerPortSettings.enable_rx_reordering: 101 → 4
Rocev2.AckAndNak.rnr_nak_retry_count / ack_request_interval: 101/102 → 6/7
Rocev2.DCQCN.rate_decrement_coefficient: 101 → 15

  • Config.artifacts is undefined (only a placeholder), so it can be removed, together with Placeholder.Object. It can be added back later with defined fields if it's really needed.

  • UID gaps in the new CC schemas. Removing fields left gaps, for example CollectiveCommunications.State.Stopped uses only 2/4/6, and CollectiveCommunications has no uid 3. These aren't released yet, so renumbering is enough.

  • The sample code is out of date. The code in the PR description still uses cc.platforms, platform_type, physical_binding.platform_name and the metric choices iterations_summary / packet_drop. None of these exist in the model any more.

@ajbalogh ajbalogh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem

  • OTG models are vendor-neutral. Existing sections such as flows and stateful_flows reference objects defined in ports and devices by name through x-constraint.
    • collective_communications should follow the same pattern.
  • Every InfrastructureRef instead resolves against components the schema never declares.
    • A user has no way to determine from the model what values to supply for these properties.
    • Each implementation must define its own valid values, so different servers can accept, reject or bind the same config differently. This breaks the model-driven, vendor-neutral contract the repository sets out to provide.

Proposal

  • Remove InfrastructureRef, RankBinding.infrastructure_ref, RankBinding.nic_refs and NicBinding.
  • RankBinding: rank_id and nics[], each with ethernet_name (x-constraint: /components/schemas/Device.Ethernet/properties/name) and optional custom_settings.
    • Host membership follows from each Ethernet's parent Device.
    • The test port follows from each Ethernet's connection.port_name.
    • Document both conventions in the RankBinding description.
  • Result: existing x-constraint tooling catches dangling references, and no new vocabulary is required.

Note

  • Update the PR sample to follow the latest model changes.

@lthompson-keysight

lthompson-keysight commented Nov 19, 2026 •

Copy link
Copy Markdown
Author

Changes made for addressing @apratimmukherjee PR review #3

  • make x-field-uids contiguous
  • remove config/artifacts
  • set default iteration_delay to 0
  • updated sample in PR description

@ajbalogh comments being evaluated

Simplified infrastructure

Snapshot of Keysight's model work, from internal commit 9a0ea1ada4b4, based on upstream
67a8aa5. Generated artifacts are built by CI from the sources in this commit.

Co-authored-by: Cristian Dinu <cristian-mircea.dinu@keysight.com>
Co-authored-by: Gabe Righi <gabriel.righi@keysight.com>
@lthompson-keysight

Copy link
Copy Markdown
Author

Problem

  • OTG models are vendor-neutral. Existing sections such as flows and stateful_flows reference objects defined in ports and devices by name through x-constraint.

    • collective_communications should follow the same pattern.
  • Every InfrastructureRef instead resolves against components the schema never declares.

    • A user has no way to determine from the model what values to supply for these properties.
    • Each implementation must define its own valid values, so different servers can accept, reject or bind the same config differently. This breaks the model-driven, vendor-neutral contract the repository sets out to provide.

Proposal

  • Remove InfrastructureRef, RankBinding.infrastructure_ref, RankBinding.nic_refs and NicBinding.

  • RankBinding: rank_id and nics[], each with ethernet_name (x-constraint: /components/schemas/Device.Ethernet/properties/name) and optional custom_settings.

    • Host membership follows from each Ethernet's parent Device.
    • The test port follows from each Ethernet's connection.port_name.
    • Document both conventions in the RankBinding description.
  • Result: existing x-constraint tooling catches dangling references, and no new vocabulary is required.

Note

  • Update the PR sample to follow the latest model changes.

Simplified Infrastructure to use an automatic assignment algorithm per discussion, and eliminated chakra references.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants