OVN NB drift on router gateways: missing localnet uplink port + unrepairable router-peer LSP corruption

[cardoe]

Summary

Investigating a "no outbound connectivity" report on gateway (staging, PUBLICNET) turned up two separate OVN NB drift bugs that left a router's external gateway completely non-functional, with no automated tooling able to detect or repair either one.

What we found

1. Missing OVN localnet uplink port, with no reconciliation path

routers.create_uplink_port/delete_uplink_port (neutron_understack/routers.py) create/delete a raw type=localnet Logical_Switch_Port directly via ovn_client()._nb_idl.create_lswitch_port(...), bypassing Neutron's revision-number tracking entirely (empty external_ids, no ovn_revision_numbers row).

Neutron's core OVN maintenance worker (check_for_inconsistencies / get_deleted_resources) can only detect drift for resources that were ever inserted into ovn_revision_numbers — confirmed by reading neutron/db/ovn_revision_numbers_db.py:get_deleted_resources, which filters ovn_revision_numbers rows with standard_attr_id IS NULL. Since this LSP was never in that table, the maintenance worker has no way to know it exists, let alone repair it if it goes missing.

In this incident, the localnet port for PUBLICNET's VLAN segment (1806/f20-5-network) was simply gone. Nothing noticed. Outbound traffic through the router's SNAT had no physical uplink to leave on.

2. Router gateway's switch-side peer LSP had type/options blanked, and neutron-ovn-db-sync-util --repair couldn't fix it

The switch-side Logical_Switch_Port peer for the router's gateway LRP (name = the Neutron port ID, external_ids correctly showing device_owner=network:router_gateway) had type="" and options={} instead of type=router + options:router-port=.../nat-addresses/exclude-lb-vips-from-garp. This is set by SetLRouterPortInLSwitchPortCommand (core Neutron OVN driver, commands.py), called from create_router_port/_update_lrouter_port. We don't have a confirmed trigger for how it got blanked (candidate: ovn_client.update_port()'s upsert-style path skipping the type=router re-assertion — see ovn_client.py:709-719 — though the is_ovn_provider_router() gate should have passed for this non-flavored router, so this isn't fully proven).

More importantly: running neutron-ovn-db-sync-util's full repair did not fix it. ovn_db_sync.py:_is_router_port_changed (used by sync_routers_and_rports to decide whether a router port needs re-syncing) only compares networks and ipv6_ra_configs — the LRP-side data. It never inspects the switch-side peer LSP's type/options at all. Since this router's LRP-side data was never corrupted (only the switch-side peer was), the sync tool concluded "no change needed" and silently skipped the exact field that was broken. The repair run visibly refreshed external_ids/revision_number (via the generic ports-sync pass) but left type/options untouched.

Both were fixed manually via direct ovn-nbctl writes (not via any existing tool):

# 1. recreate the missing localnet uplink port
ovn-nbctl lsp-add <switch> uplink-<segment_id> -- \
  lsp-set-type uplink-<segment_id> localnet -- \
  lsp-set-addresses uplink-<segment_id> unknown -- \
  set Logical_Switch_Port uplink-<segment_id> tag=<vlan> \
    options:network_name=<physnet> options:mcast_flood_reports=true \
    options:mcast_flood=false options:localnet_learn_fdb=false

# 2. restore the router-gateway peer LSP's type/options
ovn-nbctl set Logical_Switch_Port <router_port_id> \
  type=router \
  options:router-port=lrp-<router_port_id> \
  options:nat-addresses=router \
  options:exclude-lb-vips-from-garp=true

Ask

• For (1): either make create_uplink_port/delete_uplink_port go through Neutron's revision-tracked port-creation path so the existing maintenance-worker machinery covers them for free, or add a dedicated periodic reconciliation task in neutron-understack that walks router-owned networks and recreates whichever half (Neutron trunk-subport vs. OVN localnet LSP) is missing.
• For (2): this is upstream/core Neutron behavior (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py), not neutron-understack — worth deciding whether we carry a local patch, or report/patch upstream, so _is_router_port_changed also compares the switch-side peer LSP's type/options, not just the LRP's networks.

Note: we weren't able to confirm the exact trigger event for (2) — it happened around the same time as an unrelated RabbitMQ queue-durability cleanup + neutron-server restart, but that's circumstantial, not confirmed from logs.

Environment

• rax-staging-iad3-staging
• Router: 1533c5be-e717-45d3-a082-f84f18063016
• Network: PUBLICNET (b420e863-7d26-4c87-8974-76f6c8750640), VLAN 1806 / f20-5-network

[cardoe]

Update: confirmed root cause

Found direct log evidence in neutron-ovn-maintenance-worker's own log history (the pod has never restarted since its first boot, so the full trail back to startup was still available).

Timeline (all UTC, 2026-09-09):

• 17:03:36Z — the neutron-ovn-maintenance-worker Deployment is created for the very first time ever in this environment (part of the RabbitMQ queue-durability cleanup work).
• 17:14:04Z — it settles onto its current, still-running pod.
• 17:14:13.314Z — its OVN NB/SB OVSDB IDL client logs connected.
• 17:14:24.8–24.9Z (~11.5s later) — its startup ovn_db_sync pass logs, back-to-back, for 9 ports:

  WARNING ...ovn_db_sync... Port found in Neutron but not in OVN NB DB, port_id=<id>
  WARNING ...ovn_db_sync... Creating the port <id> in OVN NB DB
  

Those 9 ports are every port Neutron knew about on the PUBLICNET network at that moment — all 8 router_gateway ports across different routers (including the router in question's gateway port, 7eedfa49-...), plus the shared trunk-subport port 9e8ab558-....

What actually happened: 11.5 seconds is not enough time for a freshly-opened OVSDB IDL connection to finish its initial full-table transfer from the NB server on a fleet this size. The startup sync pass ran its "does this port exist in OVN?" check against a still-populating local IDL cache, got a false negative for every port on this switch, and "fixed" the false positive by calling the generic (non-router-aware) create_port() path for each one — overwriting the existing, correct rows with generic defaults. That's exactly what stripped type=router / options:router-port=.../nat-addresses/exclude-lb-vips-from-garp off the gateway's switch-side peer LSP. The missing localnet uplink LSP (a separate, untracked raw OVN object on the same switch, not itself a Neutron-tracked port) is very likely a sibling casualty of the same race window, though it doesn't appear directly in this log trail.

Confirmed: nobody ran manual ovn-nbctl against the NB DB. This was triggered purely by turning on neutron-ovn-maintenance-worker for the first time. It's a startup-ordering race in core Neutron's ovn_db_sync bootstrap (not neutron-understack): the sync pass can begin before its own IDL has finished its initial sync from the NB server, and a false "not found" during that window causes it to destructively recreate live rows via a path that doesn't preserve router-specific type/options.

This replaces the "trigger unconfirmed, possibly RabbitMQ-related" language in the original writeup — the trigger is now confirmed and reproducible: any environment enabling neutron-ovn-maintenance-worker for the first time against a busy, populated OVN NB DB is at risk of this same false-positive-driven corruption, independent of the RabbitMQ incident that happened to coincide with it here.

[cardoe]

Correction to the previous root-cause update

The previous comment overclaimed the mechanism. On closer review (credit to a second independent review that caught this), one specific claim doesn't hold up, and the root cause is narrower than stated.

What was wrong: the claim that the OVN NB IDL connection hadn't finished replicating its initial state yet, causing a stale client-side cache to read a false "not found" for live rows, which then got silently overwritten.

This is disproven by the actual code path. AddLSwitchPortCommand.run_idl() (with may_exist=True, used by the generic port-create path) only checks the local IDL cache via idlutils.row_by_value(...). If found there, it returns early and touches nothing. If not found, it falls through to a real OVSDB insert. Checked against the live NB schema: Logical_Switch_Port has "indexes": [["name"]] -- a server-enforced unique index. If the row had genuinely still existed server-side while only the client's cache was stale, that insert should have been rejected with a duplicate-index error on commit, not silently succeeded. Since it clearly succeeded (the port now exists with the corrupted fields, no error logged), the row must have been genuinely absent server-side at that moment -- not just missing from a lagging client-side cache.

Also checked: whether neutron-ovn-db-sync-util (run manually, does not reproduce this) constructs its OVN NB IDL connection differently from the automatic MaintenanceWorker startup path (which did produce the corruption). It doesn't -- both go through OvsdbNbOvnIdl.from_worker(worker.MaintenanceWorker), which takes the identical BaseOvnIdl.from_server() + set_lock() branch either way. So that's not the differentiator either.

What the evidence actually establishes, and no more:

• The startup sync pass, at 2026-09-09T17:14:24.8-24.9Z, genuinely found 9 specific ports on one network (all of them tied to router external-gateway connections) absent from OVN NB DB, and "repaired" them via the generic, non-router-aware create_port() path -- consistent with the observed corruption (type/options wiped on the router-gateway ports' switch-side peer LSPs).
• Why those 9 rows were genuinely absent from OVN NB DB at that exact moment is not established by this evidence and remains unconfirmed. It coincided with a broader window of other restarts/cleanup activity in the environment, but that connection is circumstantial, not proven -- and a race tied to that broader chaotic window wouldn't necessarily reproduce in an isolated, calm manual retest later, which is consistent with (but doesn't prove) that explanation either.
• The missing raw localnet LSP (the neutron-understack-created object, separate GitHub issue) is not connected to this log trail at all -- sync_networks_ports_and_dhcp_opts only walks Neutron-tracked ports, and that object was never one.

What still stands: the underlying gap being tracked here -- MaintenanceWorker's automatic startup sync can, for reasons not yet fully root-caused, "repair" a batch of resources via a path that doesn't preserve router-specific fields, with no operator in the loop to catch it -- is real and directly evidenced. The proposed safety-threshold mitigation (refuse to auto-repair a suspicious fraction of a resource type during the automatic startup sync specifically, never during a deliberate operator-run repair) still stands on its own merits as a defense-in-depth measure, independent of nailing the exact reason the rows were absent.

[cardoe]

Follow-up audit: trigger window narrowed, exact deleting transaction still unconfirmed

I audited the retained cluster state, the deployed Neutron 28.0 code, and the three maintenance-worker rollouts.

Confirmed

• There were three maintenance-worker ReplicaSets:
  • revision 1 created at 17:03:36Z
  • revision 2 created at 17:09:10Z
  • revision 3 created at 17:14:04Z
• All three used the same image, command, and configuration hashes. Revisions 2 and 3 were rollout restarts, not configuration changes.
• The first worker startup coincides with a large ovn-northd recomputation beginning at approximately 17:04:37Z.
• Revision 3's startup sync at 17:14:24Z classified all nine Neutron-tracked ports on PUBLICNET as missing and attempted to create them through the generic port path.
• The deployed AddLSwitchPortCommand uses may_exist=True and returns without modifying an LSP when it finds the row. Therefore the warning “Creating the port” is emitted before the command executes and does not, by itself, prove that revision 3 overwrote an existing row.
• The manual sync utility and maintenance worker use the same IDL/synchronization implementation. A manual repair being harmless now is expected because the replacement rows currently exist and have the external IDs required by the synchronizer.
• The generic repair path is not router-aware. If a router gateway/interface LSP is genuinely absent, it recreates the base Neutron port but does not restore the switch-side peer's type=router and router-specific options.
• Core synchronization cannot repair the raw UnderStack uplink-* LSP because it has empty external IDs and is not a Neutron-tracked port.

Strongest remaining hypothesis

The affected set is significant: the nine tracked ports plus the missing raw uplink-* port correspond to essentially the entire LSP membership of the PUBLICNET logical switch. That pattern is more consistent with the PUBLICNET logical switch—or its complete ports reference set—being removed/recreated than with nine independent port failures.

The most likely sequence is:

1. PUBLICNET and its ports were functioning before the Helm upgrade.
2. Revision 1 ran the first automatic repair around 17:04:37Z.
3. Something during that first repair/rollout removed the PUBLICNET switch or its port references, causing the non-root LSP rows to be garbage-collected.
4. Later worker startups encountered genuinely absent rows.
5. Revision 3 generically recreated the tracked ports, leaving router peer LSPs incomplete. The untracked uplink could not be recreated.

Step 3 is still an inference, not a confirmed root cause.

Why the exact cause cannot be recovered retrospectively

The revision 1 and revision 2 pods were deleted. Their kubelet log directories have already been removed, Kubernetes events have expired, no centralized log collector is deployed, and the relevant OVN Raft history has been compacted. There is also no pre-upgrade OVN NB backup. The surviving server logs show the transaction/recomputation windows but not the row-level operations or client responsible.

How to confirm and close the gap while retaining repair mode

A controlled replay should use a disposable PUBLICNET-like logical switch with multiple router gateway ports, the shared trunk subport, and the raw uplink. Before starting/restarting the worker:

1. Save OVN NB/SB and Neutron DB snapshots.
2. Record or monitor row-level OVSDB transactions.
3. Retain logs from every maintenance-worker revision.
4. Replay the same three worker startups/rollouts.
5. Compare the logical switch UUID and its full ports set before and after each transaction.

Regardless of whether that replay reproduces the original deletion, repair needs two safety improvements:

• refuse or pause an automatic repair that would recreate an anomalously large fraction of one switch's ports, while continuing to allow normal small repairs;
• never treat a router-owned port as fully repaired by the generic L2 create_port() path—perform router-aware reconciliation or leave it for the router synchronizer.

UnderStack also needs an independent reconciler for its raw uplink-* LSP because core Neutron has no inventory record from which to restore it.

This supersedes the earlier cache-race explanation: activation of the maintenance worker is the confirmed incident window, but the precise transaction that made the PUBLICNET LSPs genuinely absent remains unconfirmed until a monitored replay captures it.