Real-BMC robustness: iLO 5 session revocation, iDRAC 8 retry pauses, one-way tags, OEM states (#145 #146 #147 #150)

[bertysentry]

Fixes #145, fixes #147, fixes #150. Refs #146 (see below).

The four real-BMC robustness issues from the lab survey and the Codex review of #148, in one PR as for the earlier clusters. All four touch the request path (connector, message handlers, queue, runners).

#146: HP iLO answers D4h in the middle of a walk

The issue suggested sending the Get SDR again. Low-level probes on the lab iLO (ilo-hp-ceph, actually an iLO 5, ProLiant Gen10) show that this cannot work:

• Once the D4h comes, the session is revoked: every command of the session gets D4h, Set Session Privilege Level included. The same Get SDR sent 61 times over 22 s never succeeded; a new session works.
• The revocations come from outside the client: a broken management tool in the lab deletes the IPMI sessions every two minutes.
• The iLO applies a deletion at a whole minute of the session's age. Revocations came at 60.2–60.7 s, 121–123 s and 182 s.
• With a dedicated metricshub account, sessions were still revoked at whole minutes of their age (61 s, 121.5 s, 182 s), whatever the keep-alive command. That tool probably deletes every session it can see.
• A first series of experiments suggested that the keep-alive (Get Channel Authentication Capabilities since Fix the connection and manager lifecycle (#126, #94, #95, #98) #148) shortened sessions to 60 s. The runs with the dedicated account disproved it: the experiments had been phase-locked to the deletion sweep, each one starting right after the previous one died. The second commit's docs no longer claim it.

In this PR:

• A Get Sensor Reading refused with any completion code costs only that sensor (WARN naming the sensor and the host), not the whole walk.
• The keep-alive is a Get Device ID, as ipmitool sends. That change came with the One-way IPMI messages do not reserve their tag: a late reply can be taken for the reply of a later request of the same command #150 rework of Connection.KeepAlive and is not needed for the iLO.
• Troubleshooting describes the revoked-session symptom (0xD4 on commands that worked earlier in the session).
• Not in this PR: surviving a revoked session, by opening a new session and resuming the walk at the same record. That is the only library-side cure; it is not planned, and HP iLO 4 answers Get SDR with InsufficientPrivilege (D4h) intermittently and the whole walk fails #146 is closed as caused by the lab connector.

Branch runs on the iLO:

• After its reset, with the shared root account: 3 of 4 collections succeeded (12 FRUs, 107 sensors). In the 4th, the session was revoked 62 s into the FRU walk.
• With the dedicated account: both branch collections succeeded, while one of the two main collections had its FRU walk revoked at 61 s.

#147: Dell iDRAC 8 collections over two minutes

Probes with transport retries off show:

• the iDRAC answers C3h within 15 to 200 ms;
• it loses about 1.5% of the replies, and the request sent again at once succeeds.

The cost was the random pause of up to idleTime (4 s) before every resend, on top of the 5 s timeouts.

Fix:

• IpmiConnector resends at once after a lost reply or a C3h. The pause stays for the codes that say the BMC is busy (node busy, out of resources, initialization in progress).
• Every try takes a fresh tag. Resending under the tag of a request that was just answered raced with the receiver releasing that tag, and the resend could then wait forever: the pause used to hide this race, and the review found it.
• The retry WARN is one line, without a stack trace.

FRU walk times with no lost reply in the run:

Card	main	this branch
idrac-f200-9bx1dd3	17.8 s	10.6 / 13.1 s
idrac-f200-cbx1dd3	21.7 / 15.7 s	10.9 / 12.3 s
idrac-f200-bbx1dd3	19.6 s	10.2 / 12.0 s
idrac-ecs1 (iDRAC 8, 8-19 lost replies per run)	98.7 / 99.9 / 111.2 s	82.2 / 42.9 / 50.7 s

The remaining time on a lossy iDRAC is the 5 s per-message timeout, which #101 would make configurable. Trade-off: without the pause, the 4 tries of a message cover about 21 s instead of about 27 s on average. Both versions failed one walk during a 50 s outage of an F200.

Not done: giving up a FRU on its first C3h (the issue's second suggestion). Retrying C3h without the pause costs less than 100 ms per FRU, so a special case isn't worth it.

#150: one-way messages hold their tag

• MessageHandler queues a one-way IPMI message like any request (MessageQueue.add(request, oneWay), QueueElement.isOneWay()). Its tag stays reserved until its reply or its timeout. The reply is dropped and the timeout is not reported. Like any request, it takes one of the 8 slots of the window meanwhile.
• The SOL ACK-only packets are sent one-way but the BMC never acknowledges them (IPMI 2.0 section 15.9). They keep an unqueued tag (SolMessageHandler.takeTag): queueing them would fill the SOL window and stall the receive thread, which the review caught.
• Connection.KeepAlive, a public class added in this unreleased 1.2.03 cycle (Fix the connection and manager lifecycle (#126, #94, #95, #98) #148), becomes a private Get Device ID coder.

#145: OEM and reserved codes logged as errors; 70h-7Eh not OEM

• The "Invalid value" fallbacks of the record decoders log at DEBUG. That covers EntityId, DeviceType, SensorType, SensorUnit, ChassisType, FruMultiRecordType, ManagementAccessRecordType and ReadingType. The returned values do not change, so the text output does not change either.
• Event/reading types 70h-7Fh are OEM (Table 42-1, ReadingType.isOem()):
  • their sensors report the raw state bytes (name=0xHHLL), where main printed name=Unknown;
  • an OEM sensor with no state asserted reports no state, like any discrete sensor;
  • ReadingType.parseInt(), and so getStatesAsserted() and the SEL record event, returns UnknownOEMEvent for these types.

Lab regression test (all 40 cards)

Each card ran chassis status (login), getFrus(), getSensors() and the text conversion, once on main (fdd1738) and once on this branch:

• Logins and features exercised: cipher suites 3 and 17, skipAuth on the XCC, and the hex BMC key on the Cisco C240. The new keep-alive runs on every collection longer than 30 s.
• Hardware: Lenovo IMM/XCC, GIGABYTE, Dell iDRAC 6/8/F200, Cisco IMC, Fujitsu iRMC S5, Supermicro and HP iLO 5.
• Results: all 39 cards other than the iLO collect fine on the branch, with the same FRU and sensor counts; for the iLO, see HP iLO 4 answers Get SDR with InsufficientPrivilege (D4h) intermittently and the whole walk fails #146 above.
• Text output: it differs from main only in the intended OEM states (DDR4_P1_AB_PRS=0x8009 instead of Unknown, Err Reg Pointer=0x8001, vFlash=0x8021), apart from reading drift.
• Logs: no ERROR lines are left.

Review

A review in six lenses, with an independent verifier per finding, confirmed 17 findings, fixed in this PR: the SOL ACK queueing, the same-tag resend race, idle OEM sensors, and several doc corrections. One correction is pre-existing: the D4h enum constant is spelled InsufficentPrivilege, not InsufficientPrivilege as the docs said. A second pass on these fixes confirmed 8 minor findings (tests that could pass by chance or did not pin the one-way routing, doc wording), fixed in the second commit.

Found along the way and filed separately: #151, sensor and FRU names that carry the NUL padding of their SDR record (iRMC, iLO, Supermicro).

Tests

• mvn verify: 113 tests; checkstyle, PMD, CPD and SpotBugs clean.
• New tests:
  • fake-BMC session tests showing that a lost reply and a C3h are resent without pause;
  • a one-way message holds its tag and slot until its reply, and its timeout is not reported;
  • the keep-alive is queued as a one-way Get Device ID, and the SOL ACK-only packets are not queued (checked to fail without the override);
  • OEM reading types 70h-7Fh, active and idle;
  • a refused Get Sensor Reading costs only that sensor.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-09T19:41:34.740462Z	49c6232	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 283cdf35f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reuse the request tag after a lost reply

When a reply is lost after the BMC has already executed a state-changing command, resending through sendMessage() assigns a new IPMI request sequence number, so the BMC treats it as a new request rather than a duplicate and may execute the operation twice. The existing MessageHandler.retryMessage() contract explicitly preserves the original tag for duplicate detection; the lost-reply path needs equivalent reservation/race handling, while retries after an actual transient response may still require a fresh tag.

Useful? React with 👍 / 👎.

[bertysentry]

No change for this one: the lost-reply path already resends under a fresh tag on main, so this PR does not change it. Since #140, MessageQueue.processObsoleteMessage() removes a timed-out request and releases its tag before it notifies the listeners, so the retry(tag) that followed found no queued element (MessageHandler.retryMessage() returned -1) and sendMessage() took a new tag. The same happened after a C3h, once the 0-4 s pause had let the receiver remove the answered request. This PR only drops the same-tag resend in the remaining window, when the request was answered but not yet removed, where the resend could wait forever. Keeping the original tag across a lost reply, so that a BMC could recognize a duplicated state-changing command, would mean keeping timed-out requests queued, which #140 removed (#77/#78). That is a separate, pre-existing question and could be its own issue if wanted.

🤖 Addressed by Claude Code