Add Arkime OT protocol field normalizations for Dashboards cross-filtering

[sercanokur]

Summary Feature Issue: normalize fields for OT protocols supported by Arkime #986

Malcolm Dashboards and Arkime use different field names for the same Zeek OT protocol attributes. Zeek logs are indexed under nested zeek.* fields, while Arkime sessions use top-level protocol fields defined in db.pl (sessions3Update). This mismatch prevents reliable pivoting and cross-filtering between Dashboards visualizations and Arkime session views.

This PR aligns Malcolm with Arkime's OT field naming for eight protocols: BACnet, C12.22, DNP3, EtherNet/IP (ENIP/CIP), Modbus, OPC UA, S7comm, and Synchrophasor. Changes are additive only: original zeek.* fields are preserved; new top-level Arkime-style fields are merged alongside them.

Motivation

Operators often start an investigation in Malcolm Dashboards (Zeek-indexed data) and pivot into Arkime for PCAP/session context, or vice versa. Without shared field names, filters and drill-downs do not carry across tools. Arkime already documents these mappings in sessions3Update; this PR brings Malcolm's ingest template, Logstash pipeline, and OT dashboards in line with that reference.

Changes

1. OpenSearch composable template (dashboards/templates/composable/component/arkime.json)

Added index mappings for OT protocol blocks mirroring Arkime db.pl sessions3Update definitions, including:

• bacnet.* (function, apduType, service, objectName, …)
• c1222.* (calledAp*, callingAp*, securityMode, responseControl, …)
• dnp3.funcName
• enip.* / cip.* (command, service, class, deviceType, product, status, vendor, …)
• modbus.* (funccode, exccode, transactionid, unitid)
• opcua.* (endpointUrl, securityPolicyUri)
• s7comm.* (errorClass, errorCode, func, funcName, opcode, pduref, rosctr)
• synchrophasor.* (cmd, dataRate, frameType, idcode, numPMU, station)

2. Zeek Logstash pipeline (logstash/pipelines/zeek/1200_zeek_mutate.conf)

Within each protocol's existing log_source branch, added mutate merge filters that copy Zeek nested fields to Arkime top-level names, for example:

Protocol	Zeek source	Arkime target
BACnet	zeek.bacnet.bvlc_function	bacnet.function
BACnet	zeek.bacnet.pdu_type	bacnet.apduType
BACnet	zeek.bacnet.pdu_service	bacnet.service
C12.22	zeek.c1222.mechanism_name	c1222.securityMode
DNP3	zeek.dnp3.fc_request / fc_reply	dnp3.funcName
ENIP	zeek.enip.enip_command	enip.command
Modbus	zeek.modbus.func	modbus.funccode
OPC UA	zeek.opcua_binary.endpoint_url	opcua.endpointUrl
S7comm	zeek.s7comm.func	s7comm.func
Synchrophasor	zeek.synchrophasor.frame_type	synchrophasor.frameType

DNP3 mutate ID fix: Arkime merge filters use IDs suffixed with _arkime (mutate_merge_zeek_dnp3_fc_request_arkime, mutate_merge_zeek_dnp3_fc_reply_arkime) to avoid colliding with existing mutate IDs in 1300_zeek_normalize.conf. Without distinct IDs, the malcolm-zeek pipeline fails to start (duplicate filter ID), blocking all Zeek ingest.

No regressions to existing normalize logic: 1300_zeek_normalize.conf is unchanged. DNP3 documents still receive event.result / event.action from the existing normalize path while also gaining dnp3.funcName for Arkime alignment.

3. OT dashboard JSON (7 dashboards)

Updated field references from zeek.* to Arkime field names in Discover/visualization configs:

Dashboard	File
BACnet	2bec1490-eb94-11e9-a384-0fcf32210194.json
C12.22	db51fc00-673b-11f0-8230-37b935257dd0.json
DNP3	870a5862-6c26-4a08-99fd-0c06cda85ba3.json
EtherNet/IP	29a1b290-eb98-11e9-a384-0fcf32210194.json
Modbus	152f29dc-51a2-4f53-93e9-6e92765567b8.json
OPC UA Binary	dd87edd0-796a-11ec-9ce6-b395c1ff58f4.json
S7comm	e76d05c0-eb9f-11e9-a384-0fcf32210194.json
Synchrophasor	2cc56240-e460-11ed-a9d5-9f591c284cb4.json

Non-OT dashboards and other Logstash pipelines (Suricata, beats, filescan, enrichment, etc.) are not modified.

Testing performed

Local Malcolm stack (Docker Compose) with docker-compose-dev.yml to bind-mount the updated Logstash pipeline.

Pipeline health

• Confirmed all Logstash pipelines start cleanly (malcolm-zeek healthy).
• Verified zero duplicate mutate IDs between 1200_zeek_mutate.conf and 1300_zeek_normalize.conf.
• No Logstash ERROR / duplicate-ID failures after the DNP3 _arkime suffix fix.

Ingest verification (OpenSearch arkime_sessions3-*)

Representative OT PCAPs processed via Zeek offline (zeek-offline -r … local) and uploaded to /zeek/upload/:

Protocol	Sample PCAP	Arkime fields verified
BACnet	bacnet_example.pcap	bacnet.service, bacnet.apduType, bacnet.function, bacnet.objectName
DNP3	dnp3_example.pcap	dnp3.funcName (+ existing event.result, zeek.dnp3.*)
ENIP/CIP	enip_cip_example.pcap	enip.command, cip.service, cip.class, identity fields
Modbus	modbus_example.pcap	modbus.funccode, modbus.unitid, …
OPC UA	open62541_client-server_mainloop.pcap	opcua.endpointUrl, opcua.securityPolicyUri
S7comm	snap7.pcap, s7comm_plus.pcap	s7comm.func, s7comm.funcName, s7comm.rosctr, …
C12.22	CISA sample PCAPs	c1222.calledAp*, c1222.callingAp*, c1222.responseControl
Synchrophasor	C37.118_1PMU_TCP.pcap	synchrophasor.cmd, frameType, idcode, numPMU, dataRate

28 of 31 expected normalized fields populated across protocols.

Regression checks

• Suricata alerts: 698 event.dataset:alert documents unchanged.
• Conn logs: 386 event.dataset:conn documents with standard source/destination/network fields.
• Zeek + Arkime coexistence: e.g. Modbus docs contain both zeek.modbus.func and modbus.funccode.
• DNP3 dual mapping: dnp3.funcName, zeek.dnp3.fc_*, and event.result/event.action present on the same documents.

Fields not observed (PCAP limitation, not pipeline failure)

These Arkime fields depend on Zeek log columns absent in our test captures:

• c1222.securityMode (requires mechanism_name in Zeek log)
• c1222.calledApInvocationId
• synchrophasor.station (may require ZEEK_SYNCHROPHASOR_DETAILED=true)

Test plan for reviewers

• Rebuild/restart Logstash with dev bind-mount or image rebuild; confirm malcolm-zeek pipeline starts with no duplicate mutate ID errors.
• Ingest one PCAP per protocol (see table above) and confirm Arkime fields appear in OpenSearch while zeek.* fields remain.
• Open each OT dashboard in Dashboards; verify visualizations render with new field names.
• Pivot from a Dashboards OT visualization into Arkime and confirm filters apply on matching field names.
• Spot-check non-OT traffic (conn, alert, http if available) to confirm no ingest regression.

Deployment notes

• Logstash changes require pipeline reload (dev compose bind-mount or image rebuild).
• Dashboard JSON changes require dashboards-helper re-import.
• Composable template changes apply on next index template rollover / reindex as per Malcolm's normal OpenSearch workflow.

Out of scope

• arkime/wise/source.zeeklogs.js updates (Wise field mapping) — can be a follow-up if desired.

[sylvesterkaczmarek]

I think the Modbus normalization has a type mismatch that may only show up after the updated component template is applied to a fresh index.

This change merges [zeek][modbus][func] into [modbus][funccode], but Malcolm currently treats zeek.modbus.func as a symbolic function name, for example the existing Modbus visualization filters it with *WRITE* and *EXCEPTION. In this same PR, however, modbus.funccode is mapped as long, which matches Arkime's integer function-code field.

The mutate merge operation also turns a scalar source into an array when the destination is initially absent, so a normal Zeek value here becomes a string-valued array under a field declared as long. Once an index is created with the new template, that should produce a mapping/parsing failure rather than an Arkime-compatible function code. Testing against an already-created arkime_sessions3-* index could mask this because the new component mapping does not take effect until rollover/reindex.

Could this map from an actual numeric Modbus function-code field instead, if one is available, or otherwise keep the symbolic Zeek value in a keyword-compatible field rather than modbus.funccode? A regression that ingests a Modbus record into a fresh index created from the updated template would pin the type contract.

[sercanokur]

Thanks @sylvesterkaczmarek, this is a solid catch. You're right that zeek.modbus.func is a symbolic value (function name), not numeric, while modbus.funccode is declared long to mirror Arkime's integer function-code field — and since mutate { merge => } boxes a scalar into an array when the destination field doesn't already exist, that mismatch would only surface once a fresh index is built from the updated component template. My testing ran against an already-existing arkime_sessions3-* index, so the old mapping was still in effect and masked exactly the failure mode you describe.

I'll test this against a freshly created index (rollover/reindex from the updated template) to confirm the mapping/parsing failure, and look into whether there's an actual numeric Modbus function-code field available from Zeek to map instead — otherwise I'll move the symbolic value into a keyword-compatible field rather than modbus.funccode. Will report back here once I've dug into it.

[sercanokur]

Fixed in 1718b1d — you were right about the type mismatch.

The original merge would have produced string/array values under a long mapping once the updated component template is applied to a fresh index. To resolve this:

• Removed the raw func/exception → funccode/exccode merges from 1200_zeek_mutate.conf
• Added Zeek-name → numeric-code lookup tables and centralized the conversion in 1300_zeek_normalize.conf (including _EXCEPTION suffix stripping and numeric exception_code passthrough)
• Coerced trans_id/unit_id to integers before writing modbus.transactionid/modbus.unitid
• Added scripts/test-modbus-arkime-normalization.py to lock down the type contract going forward

Also rebased onto current upstream/main to resolve the dashboard export conflicts. Original zeek.modbus.* fields and the existing event.action/event.result normalization are unchanged.

[mmguero]

I really do plan on getting to this for the september release. my tasking has been crazy the last few months, but I'm hoping to carve out time to review and merge this. thanks for your patience.

[sylvesterkaczmarek]

Normalization converts Zeek Modbus function names into integer modbus.funccode values, but the updated dashboard still applies string wildcards such as *WRITE*/*EXCEPTION* and Vega indexof(..., "REG") checks to that field. Those expressions worked on zeek.modbus.func but cannot classify the numeric field, so the Modbus panels will filter or label traffic incorrectly. Retain/use a normalized function-name field (or map codes to labels in the dashboard) and add a dashboard regression.