clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast

[Lt-Flash]

Overview

This PR introduces clusterer_controller, a new OpenSIPS module that drives the existing clusterer module's topology automatically at runtime. Rather than replacing the clusterer, it acts as a control plane on top of it — dynamically adding and removing nodes, electing a master, and managing sharing tag failover, without requiring a database, static node lists, or manual configuration.

The clusterer module itself is unchanged in its normal mode of operation. When use_controller=0 (the default), the clusterer behaves exactly as before — DB-backed or static neighbor config, standard sharing tag event routes and MI commands, no multicast involvement whatsoever. The controller is strictly opt-in.

Peer discovery is done via authenticated, encrypted UDP multicast (AES-256-GCM by default; XChaCha20-Poly1305 + Argon2id when the module is built against libsodium). Nodes announce themselves on a shared multicast group and elect a master deterministically (master / backup / member roles, see below), and the master assigns node IDs and maintains the member list. All of this is fed into the clusterer module at runtime through a new clusterer_ctrl API, so every existing clusterer consumer (dialog, usrloc, dispatcher, etc.) works without any changes to how they use the clusterer.

By default, multicast operates within a single LAN segment. For multi-site deployments, discovery can be extended across routable networks using multicast routing protocols such as PIM-SM (Protocol Independent Multicast — Sparse Mode) or similar services, provided the network infrastructure supports it.

The result is a fully identical config across all nodes — the only difference between machines is the BIN socket IP address.

---

How it works

Each node joins a multicast group and broadcasts a JOIN_REQ. The master responds with a KEY_GRANT (handing the joiner the cluster session key) and NODE_ASSIGN packets that allocate a node_id to every member and feed it into the clusterer module via the clusterer_ctrl API. The master periodically broadcasts MEMBER_LIST so late joiners and re-joiners converge quickly, and a 1-second MASTER_ALIVE keepalive lets peers detect master failure within ~3 seconds.

Each cluster has three roles: master (the active coordinator), backup (the standby that takes over on master failure — always the highest-IP non-master) and member. Election is deterministic across nodes and needs no negotiation round-trips. The master_stickiness parameter (default 1) keeps a live master in place: a higher-IP node that joins becomes the backup rather than preempting the master, so joins cause no handover (the previous backup simply drops to member if the newcomer has a higher IP). With master_stickiness=0 the highest-IP node always becomes master.

On master failure — graceful GOODBYE or keepalive timeout — the backup is promoted immediately. Because the session key is generated once at cluster bootstrap and preserved across every master change, the new master reuses the key every member already holds: no re-keying and no re-join storm.

Split-brain is prevented and healed by three cooperating mechanisms:

1. Prevention at join time — simultaneously-starting nodes see each other's JOIN_REQs, so at the join deadline a node that has seen a higher-IP starter defers self-promotion (bounded) and joins it instead. Only the highest-IP starter becomes master, so independent-key lone masters never form in the first place.
2. Same-key yield — two masters that share a session key (e.g. after a partition heals) read each other's MASTER_ALIVE; the lower-IP one yields.
3. Divergent-key merge — masters bootstrapped independently hold different session keys and can't read each other's MASTER_ALIVE, so each emits a bootstrap-key MASTER_BEACON (readable by any correctly-configured node). On hearing a beacon from a superior partition (larger member count, ties broken by higher IP), a node re-joins that master and adopts its key, converging the whole cluster onto one master and key.

Because node identity and peer lists are injected into clusterer at runtime, no DB reads or static neighbor modparams are needed. All existing clusterer consumers continue to work transparently.

---

New module: clusterer_controller

Key parameters

• cluster (repeatable, required) — defines a cluster to participate in. Value is a comma-separated key=value string:

  • id=N — positive integer cluster identifier, must match the cluster_id used by clusterer consumers (dialog, usrloc, etc.)
  • multicast=A.B.C.D:PORT — IPv4 multicast group and UDP port. Address must be in the 224.0.0.0/4 range. Two clusters on the same LAN can share the same address with different ports, or use entirely different addresses — both isolate traffic correctly.
  • password=STRING — per-cluster shared secret; falls back to the global password modparam if not set
  • bin_socket=bin:IP:PORT — BIN socket to advertise to clusterer peers for this cluster; required when multiple clusters are defined
  • manage_shtags=0|1 — per-cluster override for automatic sharing tag failover; falls back to the global manage_shtags modparam if not set, regardless of modparam order in the config file

• password (default: 3eCrEt*5629) — the shared cluster secret. A per-cluster bootstrap key is derived from it with a memory-hard KDF (scrypt, or Argon2id in the libsodium build); the session key is agreed via X25519 ECDH (see Security). All nodes in a cluster must use the same password. Change this in production.

• manage_shtags (default: 1) — global default for automatic sharing tag management. Can be overridden per cluster via the manage_shtags= key in the cluster string.

• master_stickiness (default: 1) — controls whether a live master is kept when a higher-IP node joins. 1 (sticky) = the master stays and the higher-IP joiner becomes the backup, minimising handovers; 0 = the highest-IP node always becomes master. Global default, overridable per cluster via the master_stickiness= key in the cluster string.

• query_time (default: 5) — heartbeat interval in seconds. Controls the election window (3 × query_time) and the peer purge window (6 × query_time). Valid range: 1–60.

• on_config_mismatch (default: reject) — policy when a node's consistency-critical settings (manage_shtags, master_stickiness, query_time) differ from the running cluster: reject (refuse the join, node shuts down), warn (join but log CONFIG MISMATCH), or adopt (adopt the cluster's settings at runtime). See Security.

• my_ip (default: auto-detected) — pin the controller identity to a specific local IPv4 address. Useful on multi-homed hosts.

• interface (default: auto-detected) — pin the controller to a named network interface (e.g. eth0). Ignored if my_ip is set.

Automatic sharing tag management (manage_shtags)

When manage_shtags=1 (the default), the controller master is the single authority for sharing tag state in the clusterer module. No script-level event routes or MI commands are needed to handle failover:

• Startup: all local sharing tags are forced to backup state regardless of the =active value in the clusterer config. This prevents a restarting node from stealing the active tag from a running cluster.
• Bootstrap: the first node to start alone activates its backup tags exactly once, after waiting query_time × 3 seconds with no master response.
• Failover: when any node departs, the master immediately promotes its own backup tags to active for that cluster.
• Rejoin: a returning node always starts in backup state and never reclaims the active tag from the current holder.
• Blocking: while manage_shtags=1, the clusterer_set_tag_active MI command and $shtag() script variable setter are blocked for controller-managed clusters, preventing accidental conflicts with automatic failover.

manage_shtags can be set globally and overridden per cluster, regardless of the order modparams appear in the config file:

# Global on, one cluster opts out
modparam("clusterer_controller", "manage_shtags", 1)
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444,manage_shtags=0")

# Global off, one cluster opts in
modparam("clusterer_controller", "manage_shtags", 0)
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,manage_shtags=1")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444")

Minimal config (identical on all nodes, only BIN socket IP differs)

socket=bin:10.22.23.191:3857

loadmodule "proto_bin.so"

loadmodule "clusterer.so"
modparam("clusterer", "use_controller", 1)
modparam("clusterer", "sharing_tag",   "vip1/1=active")
modparam("clusterer", "ping_interval", 4)
modparam("clusterer", "ping_timeout",  1500)

loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster",  "id=1,multicast=239.0.90.1:3333")
modparam("clusterer_controller", "password", "MyStr0ngPassw0rd!")

Multiple clusters

Clusters are isolated by their multicast address and/or port, so you can use the same multicast address with different ports, or entirely different addresses — both work:

# Same multicast address, different ports (same LAN segment)
socket=bin:10.0.1.10:5566
socket=bin:10.0.2.10:5567

modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.1.10:5566")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444,bin_socket=bin:10.0.2.10:5567")

# Or different addresses entirely
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.1.10:5566")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.2:3333,bin_socket=bin:10.0.2.10:5567")

Each cluster runs an independent controller worker process with its own election, membership, and sharing tag state. Packets from different clusters never mix: every packet carries a 2-byte cluster_id in its cleartext header, and a node silently drops any packet whose cluster_id differs from its own before decryption — so multiple clusters can even share the same multicast address and port without interfering, and foreign traffic never trips the wrong-password machinery. Two clusters merge only if they are configured identically (same address, port, cluster_id and password).

MI commands

• cl_ctr_list_members — list the current members of each cluster with their node_id, status (master / backup / member) and BIN sockets.
• cl_ctr_node_info node_id=N — full info for a specific node across all clusters.
• cl_ctr_list_config — list every configured cluster and its resolved effective settings (multicast, my_ip, bin_socket, query_time, master_stickiness, manage_shtags, shtag_mode, member_count). The password is never exposed. Useful for confirming a per-cluster override took effect.
• cl_ctr_shtag_force cluster_id=N node_id=M — pin the active sharing tag to a chosen node, overriding automatic master-driven allocation (master-only). The override is carried in MEMBER_LIST, survives master fail-over, and auto-clears if the forced node departs. shtag_mode then reports override:M.
• cl_ctr_shtag_auto cluster_id=N — clear the override and resume automatic, master-driven sharing-tag allocation (master-only).

Script variables

Read-only pseudo-variables expose live cluster state to the routing script (read from shared memory, so available in every process — SIP workers included), so a check like only the master runs this job needs no MI round-trip. Each takes an optional cluster id: the bare form ($cl_ctr_role) resolves the sole configured cluster, while an explicit id ($cl_ctr_role(2)) reads any specific cluster — so a node that participates in several clusters can query each one independently. With more than one cluster defined the bare form returns NULL and logs a one-time warning, so the cluster must be named.

• $cl_ctr_role — master / backup / member / joining
• $cl_ctr_is_master — 1 / 0
• $cl_ctr_master_ip, $cl_ctr_backup_ip — current master / backup IP (NULL if none)
• $cl_ctr_node_id — this node's node_id in that cluster (a node may hold different ids in different clusters)
• $cl_ctr_my_ip, $cl_ctr_members — controller identity IP; live member count
• $cl_ctr_shtag_mode (auto/forced), $cl_ctr_forced_node — sharing-tag allocation mode and pinned node id

# single-cluster node: the bare form
if ($cl_ctr_is_master)
    route(master_only_job);

# multi-cluster node: address each cluster by id
if ($cl_ctr_is_master(1))                 # controller cluster 1
    route(dialog_master_duty);
if ($cl_ctr_role(2) == "backup")          # a different cluster
    route(warm_standby_prep);
xlog("cl2: role=$cl_ctr_role(2) master=$cl_ctr_master_ip(2) members=$cl_ctr_members(2) myid=$cl_ctr_node_id(2)\n");

The convention is queries are nouns (variables), actions are verbs (MI commands): this node's own state is read through the variables above and changed through MI (cl_ctr_shtag_force / cl_ctr_shtag_auto). Script action functions (verbs — e.g. cl_ctr_set_maintenance() for a node to change its own state from the routing logic) are planned together with node maintenance mode; see the Roadmap.

Input validation. The optional cluster-id key is parsed at config load: a non-numeric, negative, zero, or out-of-range value (capped at 9 digits so it can never overflow into a valid-looking id) is rejected with a descriptive error and fails the config check rather than binding to the wrong cluster. The per-peer functions below are equally defensive at runtime — an out-of-range, negative, or non-numeric argument passed via $var resolves to not-found rather than faulting.

Script functions

Per-peer lookups take two arguments (cluster_id, node_id) and so are functions (a comma inside a pvar's parens is ambiguous once the pvar is used as a function argument). Booleans return true/false for if(); value lookups write an output variable:

• cl_ctr_node_is_master(cluster_id, node_id) — true if that node is master
• cl_ctr_node_present(cluster_id, node_id) — true if node_id is a live member
• cl_ctr_get_node_role(cluster_id, node_id, $out) — writes master/backup/member
• cl_ctr_get_node_ip(cluster_id, node_id, $out) — writes that node's IP

if (cl_ctr_node_present(1, 3) && cl_ctr_node_is_master(1, 3)) {
    cl_ctr_get_node_ip(1, 3, $var(ip));
    xlog("node 3 ($var(ip)) leads cluster 1\n");
}

Security

• Each packet's payload is sealed with an AEAD, behind a 2-byte cleartext magic (a key-tier selector, readable before decryption), a 2-byte cluster_id, and the nonce — everything meaningful (type, seq, IPs, pubkeys, salts, member lists) is inside the sealed payload. The magic and cluster_id are bound into the AEAD tag as AAD, so a captured packet can't be re-stamped onto another cluster_id on a shared multicast+password group. The AEAD is AES-256-GCM (WolfSSL) by default, or XChaCha20-Poly1305 when built against libsodium (its 192-bit nonce removes any random-nonce collision concern) — a build-time choice; all nodes in a cluster must match, and the active suite is logged at startup.
• A bootstrap key protects only the admission handshake and the split-brain beacon; it is derived from the shared password with a memory-hard KDF — scrypt (N=2¹⁶) or Argon2id in the libsodium build, with a per-cluster salt, derived once at startup — so a password captured from a bootstrap packet can't be brute-forced cheaply offline. The session key used for all other traffic is derived (HKDF-SHA256) from a random master salt agreed via per-worker X25519 ECDH — the salt is never sent in the clear, and the join nonce binds each wrap to a single exchange. A startup warning fires if the password is the default or has low estimated entropy.
• Replay protection via per-source 32-bit monotonic sequence numbers, with no dependency on clock synchronisation.
• Rate limiting (256 source slots, 20 pkt/s) applied before any decryption attempt to blunt CPU-exhaustion floods on the multicast group.
• Join authentication: the master sends an encrypted, unforgeable JOIN_REJECT after repeated bootstrap-decrypt failures. A wrong-password node can't read that reject, so it also self-detects — while joining it counts undecryptable packets from other peers and, at the join deadline, shuts down ("cannot authenticate — wrong password?") instead of self-promoting into a lone split-brain master. An active member ignores JOIN_REJECT, so a correct-password peer can never be evicted. A full peer table rejects new spoofed-IP joins.
• Rogue-traffic isolation: a node requests a re-key only for an undecryptable session packet from its current master (a legitimate rotation). Undecryptable session packets from any other source (e.g. a wrong-password or malicious node on the group) are ignored, so they can't force the cluster into a re-JOIN churn.
• Config-consistency enforcement: all nodes of a cluster must use identical manage_shtags, master_stickiness and query_time — a per-node mismatch would otherwise cause silent, inconsistent failover/sharing-tag behaviour (e.g. a master with manage_shtags=0 leaves no node holding the active tag). Each node advertises these effective settings in its ALIVE heartbeat and JOIN_REQ; the on_config_mismatch modparam picks the policy when a node tries to join an established cluster with different settings: reject (default) — the master returns a JOIN_REJECT and the misconfigured node shuts down with a clear message; warn — the node joins but a single deduplicated CONFIG MISMATCH warning is logged; adopt — the joining node adopts the running cluster's settings at runtime (and cl_ctr_list_config then reports the adopted values).
• node_id allocation & identity: ids are allocated only by the current master, serialised under the peer-table lock — a joining node never picks its own. The master hands out the lowest unused id from the live peer table, so a node that has failed but not yet timed out still holds its slot and its id is never reused for a different joiner — new nodes get a distinct id even during the failure-detection window. A node that restarts and rejoins from the same address reuses its previous id (and its replay counter is reset), so ids stay stable across restarts. Peers are keyed by source IP, so each node must present a stable, unique source address: two nodes behind one address (e.g. NAT) would share a slot/node_id.
• Trust model — shared secret, not per-node identity: the cluster is a single shared-secret trust domain. Authentication proves only that a peer holds the password; it does not bind a cryptographic identity to a node, and there is no per-node authorisation or revocation. So any holder of the password is a fully trusted member and can legitimately win the highest-IP election and take master — there is no "may be member" vs "may be master" distinction. An attacker without the password cannot influence the election at all (forged/replayed MASTER_ALIVE and beacons fail the AEAD tag or the per-source sequence check and are dropped before any election logic runs); the residual exposure is a malicious insider that already holds the key. Per-node keypairs with enrolment/revocation — distrust one node without re-keying the cluster — are on the Roadmap.

---

Changes to the clusterer module

New controller API (clusterer_ctrl)

A new clusterer_ctrl.c/.h pair exposes a load-time API (load_clusterer_ctrl_binds) through which an external module can drive the clusterer topology at runtime:

• set_my_identity(cluster_id, node_id, bin_url) — register this node's identity post-fork
• add_node / remove_node — add/remove peers at runtime
• update_identity — correct node_id after master assignment
• sync_current_id — sync process-local current_id after fork
• activate_backup_shtags / force_backup_shtags — sharing tag control
• set_shtag_managed — lock a cluster's tags against manual changes

use_controller=1 mode and hybrid topologies

The use_controller modparam registers the controller-driven cluster ids (via modparam("clusterer","cluster_id",N)) as stub clusters at startup; their identity and peer list are filled in post-fork by the controller — no DB or static neighbor config for those. When use_controller=0 (the default) the clusterer is completely unaffected: DB loading, static neighbors, sharing-tag event routes and MI commands all work exactly as before.

Crucially the two are now per-cluster, not all-or-nothing: a single instance can run native clusters (DB-backed or static my_node_info/neighbor_node_info) and controller-managed clusters side by side. Each cluster is flagged controller_managed (from the cluster_id modparam) or native; a controller cluster always behaves as db_mode=0 even when native clusters are DB-backed (cl_db_mode(cl)), the two kinds share one cluster_id namespace, and declaring the same id both ways is rejected. my_node_id/db_url are required only when native clusters exist. This lets a fixed DB-provisioned replication cluster coexist with a zero-config controller HA cluster in one process — see Hybrid Topologies in the admin guide.

Config examples:

Controller-only — two controller clusters sharing one BIN socket:

socket=bin:10.0.0.10:5566
loadmodule "clusterer.so"
modparam("clusterer", "use_controller", 1)
modparam("clusterer", "cluster_id", 1)
modparam("clusterer", "cluster_id", 2)
loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster", "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.0.10:5566")
modparam("clusterer_controller", "cluster", "id=2,multicast=239.0.90.1:3334,bin_socket=bin:10.0.0.10:5566")

Hybrid — a DB-native cluster 10 next to a controller cluster 1:

socket=bin:10.0.0.10:5566
loadmodule "db_mysql.so"
loadmodule "clusterer.so"
# native side: cluster 10 lives in the DB
modparam("clusterer", "db_mode", 1)
modparam("clusterer", "db_url", "mysql://opensips:pass@localhost/opensips")
modparam("clusterer", "my_node_id", 5)          # this node's id in cluster 10
# controller side: cluster 1 is dynamic
modparam("clusterer", "use_controller", 1)
modparam("clusterer", "cluster_id", 1)
loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster", "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.0.10:5566")

modparam("dialog", "dialog_replication_cluster", 1)   # controller-managed (cluster 1)
modparam("dispatcher", "cluster_id", 10)              # DB-native (cluster 10)

Hybrid, no DB — a static native cluster 7 next to a controller cluster 1:

loadmodule "clusterer.so"
modparam("clusterer", "my_node_id", 5)
modparam("clusterer", "my_node_info", "cluster_id=7, url=bin:10.0.0.10:5566")
modparam("clusterer", "neighbor_node_info", "cluster_id=7, node_id=6, url=bin:10.0.0.11:5566")
modparam("clusterer", "use_controller", 1)
modparam("clusterer", "cluster_id", 1)
loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster", "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.0.10:5566")

Here my_node_id=5 is this node's id in the native cluster 7; in cluster 1 the controller assigns an id at runtime, which may differ.

Per-cluster node identity (node_id / cluster_id)

Every clusterer packet already carries a cluster_id, and each cluster keeps its own membership. The change here is how a node's own identity is represented.

Before. A node's identity was a single global current_id (the my_node_id modparam). Classic clusterer works because an operator gives a node the same node_id in every cluster it belongs to — add_node_info() finds "me" in each cluster by matching node_id == current_id, and every runtime check (the "this message is from myself" source check, outgoing id stamps, topology and capability comparisons, cl_get_my_index) reads that one global.

The problem the controller introduces. Each controller cluster's master allocates node_ids independently (lowest free id in that cluster). So the same node can legitimately be node_id=1 in one cluster and node_id=2 in another — the ids diverge. A single global current_id cannot represent that: clusterer_ctrl_sync_current_id() collapsed it to the first cluster's value, so for every other cluster the node mis-identified itself. Concretely, a peer whose id equalled the stale global was treated as "me" (and vice-versa), and outgoing packets were stamped with the wrong source id — silently breaking replication for the divergent cluster. This also affected two controller clusters with different membership, not just hybrids.

The change. Identity is now per-cluster. The authoritative source is cl->current_node->node_id, which already lives in shared memory, exposed through a small accessor cluster_self_id(cl) (returns -1 — matches no node — before this cluster's identity is established). Specifically:

• add_node_info() takes an explicit self_id for the self/peer decision instead of reading the global; native load/provision pass my_node_id (unchanged behaviour), the controller and runtime-topology callers pass the per-cluster id.
• Every runtime read — the source/dest checks, id stamps, topology and capability comparisons, cl_get_my_index — uses cluster_self_id(cl).
• The three BIN receive paths resolve the cluster before the "bad source" self-check, so it compares against this node's id in that cluster.
• msg_add_trailer() stamps the source id per cluster.

The legacy global current_id / GET_CURRENT_ID remains only on the native DB/static provisioning path (where a node uses one id across all its clusters, by the operator's own configuration) and for the get_my_id() API. Result: within a hybrid or multi-controller-cluster instance, cluster_id selects the cluster and node_id is meaningful within that cluster, independently — validated live with a node holding node_id=3 in one cluster and node_id=2 in another, replicating cleanly in both.

---

Testing

Five HA scenarios were tested on a 3-node cluster (10.22.23.191–193) with dialog replication and sharing tag failover:

• Sequential cold start — first node becomes master; with master_stickiness=1 it stays master as higher-IP nodes join, each newcomer taking the backup role and the previous backup dropping to member. Tags activated on the master, others in backup ✅
• Master hard stop (kill -9) — the backup (highest-IP survivor) is promoted to master in ~3s using the preserved key (no re-keying, no election loop), a new backup is designated, and its backup tags are activated ✅
• Master restart (rejoin) — node rejoins as member/backup, master stays put, tags remain on the current master ✅
• Member hard stop — master detects departure and activates its own backup tags for that cluster ✅
• Member restart (rejoin) — node rejoins cleanly, no tag disruption ✅
• master_stickiness=0 — verified a higher-IP joiner takes over as master immediately (the non-sticky path) ✅
• Wrong-password node — a node configured with a different password shuts down at the join deadline ("cannot authenticate … wrong password?") instead of forming a lone split-brain master, and the live cluster is completely undisturbed (zero session-mismatch/demote events); a corrected node then rejoins normally ✅
• Rogue-joiner tool (test/cc_join_reject_test.py) — a standalone script (no node config changes) run from a non-member host on the multicast segment: it sends unauthenticated JOIN_REQs and confirms the master emits a JOIN_REJECT, then floods fake MASTER_ALIVE packets and confirms the cluster ignores them (no re-JOIN churn) ✅
• Sharing-tag override — cl_ctr_shtag_force pins the active tag onto a non-master node (all nodes report override:N), the override survives a master kill -9 (the new master keeps honouring it), auto-clears when the forced node departs, and cl_ctr_shtag_auto restores automatic allocation; error paths (non-master, unknown node/cluster) rejected ✅
• Split-brain prevention & merge — a simultaneous all-node cold start now converges to a single cluster: lower-IP nodes log "deferring self-promotion" and join the highest-IP starter (decrypt-failure warnings dropped from ~700/node to a handful), and an induced divergent-key split is merged via MASTER_BEACON ("superior master … found via beacon") ✅
• libsodium suite — the same HA scenarios (join, failover, rejoin) verified end-to-end with the XChaCha20-Poly1305 + Argon2id build ✅

Full test details are documented in modules/clusterer_controller/doc/clusterer_controller_tests.xml.

---

Limitations (alpha)

• IPv4 only (multicast address must be in 224.0.0.0/4)
• Rough clock agreement between nodes is recommended — the election window is quantized to query_time — but strict NTP is not required for correctness; replay protection is sequence-number based, not time-window based
• Multicast routing must be available on the network (LAN via IGMPv2/v3, or routed via PIM-SM or similar multicast routing protocol for multi-site deployments)
• The bootstrap key uses a memory-hard KDF (scrypt, or Argon2id in the libsodium build) over a fixed per-cluster salt (the password is a shared group secret, so the salt can't be per-node); a strong, high-entropy password is still recommended — the KDF raises the cost of an offline guess but doesn't rescue a weak password
• The crypto suite is a build-time choice (WolfSSL AES-256-GCM/scrypt vs. libsodium XChaCha20-Poly1305/Argon2id); the two wire formats are not interoperable, so a cluster can't mix nodes built with different suites (the active suite is logged at startup to make mismatches visible)

---

Roadmap (not in this PR)

Planned follow-ups (also documented under Planned Features in the admin guide):

• Node maintenance mode — take a node out of duty for a rolling upgrade while it stays in the cluster (keeps replicating, answers pings, stays in cl_ctr_list_members) but is excluded from election and sheds its sharing tags (a cl_ctr_shtag_force pin auto-clears). Two levels: evicted (out of election/tags; the script refuses new work while dialogs finish) and full (additionally marks the node down for clusterer consumers). State is cluster-wide (carried in ALIVE/MEMBER_LIST, survives failover) and runtime-only. Interfaces: MI cl_ctr_maintenance (any node, master propagates), script function cl_ctr_set_maintenance(), read-only vars $cl_ctr_maintenance / $cl_ctr_node_maint(cid,nid) ($cl_ctr_role gains a maintenance value), and event E_CL_CTR_MAINTENANCE.

• Statistics — role, member count, master changes, joins/leaves, JOIN_REJECTs, decrypt failures, config mismatches, split-brain merges via get_statistics/monitoring.

• Events — E_CL_CTR_* raised on state transitions (became master, demoted, node joined/left, split-brain merged, config mismatch, auth reject) for event_route/subscribers.

• Per-node identity & stronger admission crypto — move beyond the single shared secret to per-node keypairs with enrolment and revocation, so one node can be distrusted without re-keying the whole cluster (closing the insider-trust limitation noted under Security). Candidate directions: a standard handshake (Noise) for admission, and a PAKE (CPace/SPAKE2) to remove offline password-guessing on captured bootstrap packets.

(The previously-listed sharing-tag override, resume-auto-tagging, shtag_mode and the read-only pseudo-variables are now implemented in this PR — see the MI commands and Script variables sections above.)

[razvancrainea]

Thanks again for this amazing work! However, since the 3.4 branch is no longer maingained, we will not be able to merge this PR. Thus I am closing it. Please continue your work against the master branch, in PR #4074.