clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast

[Lt-Flash]

OpenSIPS 3.6 port of #4072 (which targets the 3.4 branch). Identical module + clusterer integration; the only code delta versus the 3.4 PR is the socket_info_full listeners-list change in cc_discover_bin_sockets() required by the 3.5+ socket refactor. Built against 3.6.7 and validated on a live 3-node cluster: cluster formation, hard-kill master failover (backup promotes + shtag activates), sticky rejoin, cc_shtag_force/cc_shtag_auto override, and config-mismatch reject — all identical to 3.4. Also includes the generated module README.

Overview

This PR introduces clusterer_controller, a new OpenSIPS module that drives the existing clusterer module's topology automatically at runtime. Rather than replacing the clusterer, it acts as a control plane on top of it — dynamically adding and removing nodes, electing a master, and managing sharing tag failover, without requiring a database, static node lists, or manual configuration.

The clusterer module itself is unchanged in its normal mode of operation. When use_controller=0 (the default), the clusterer behaves exactly as before — DB-backed or static neighbor config, standard sharing tag event routes and MI commands, no multicast involvement whatsoever. The controller is strictly opt-in.

Peer discovery is done via authenticated, encrypted UDP multicast (AES-256-GCM by default; XChaCha20-Poly1305 + Argon2id when the module is built against libsodium). Nodes announce themselves on a shared multicast group and elect a master deterministically (master / backup / member roles, see below), and the master assigns node IDs and maintains the member list. All of this is fed into the clusterer module at runtime through a new clusterer_ctrl API, so every existing clusterer consumer (dialog, usrloc, dispatcher, etc.) works without any changes to how they use the clusterer.

By default, multicast operates within a single LAN segment. For multi-site deployments, discovery can be extended across routable networks using multicast routing protocols such as PIM-SM (Protocol Independent Multicast — Sparse Mode) or similar services, provided the network infrastructure supports it.

The result is a fully identical config across all nodes — the only difference between machines is the BIN socket IP address.

---

How it works

Each node joins a multicast group and broadcasts a JOIN_REQ. The master responds with a KEY_GRANT (handing the joiner the cluster session key) and NODE_ASSIGN packets that allocate a node_id to every member and feed it into the clusterer module via the clusterer_ctrl API. The master periodically broadcasts MEMBER_LIST so late joiners and re-joiners converge quickly, and a 1-second MASTER_ALIVE keepalive lets peers detect master failure within ~3 seconds.

Each cluster has three roles: master (the active coordinator), backup (the standby that takes over on master failure — always the highest-IP non-master) and member. Election is deterministic across nodes and needs no negotiation round-trips. The master_stickiness parameter (default 1) keeps a live master in place: a higher-IP node that joins becomes the backup rather than preempting the master, so joins cause no handover (the previous backup simply drops to member if the newcomer has a higher IP). With master_stickiness=0 the highest-IP node always becomes master.

On master failure — graceful GOODBYE or keepalive timeout — the backup is promoted immediately. Because the session key is generated once at cluster bootstrap and preserved across every master change, the new master reuses the key every member already holds: no re-keying and no re-join storm.

Split-brain is prevented and healed by three cooperating mechanisms:

1. Prevention at join time — simultaneously-starting nodes see each other's JOIN_REQs, so at the join deadline a node that has seen a higher-IP starter defers self-promotion (bounded) and joins it instead. Only the highest-IP starter becomes master, so independent-key lone masters never form in the first place.
2. Same-key yield — two masters that share a session key (e.g. after a partition heals) read each other's MASTER_ALIVE; the lower-IP one yields.
3. Divergent-key merge — masters bootstrapped independently hold different session keys and can't read each other's MASTER_ALIVE, so each emits a bootstrap-key MASTER_BEACON (readable by any correctly-configured node). On hearing a beacon from a superior partition (larger member count, ties broken by higher IP), a node re-joins that master and adopts its key, converging the whole cluster onto one master and key.

Because node identity and peer lists are injected into clusterer at runtime, no DB reads or static neighbor modparams are needed. All existing clusterer consumers continue to work transparently.

---

New module: clusterer_controller

Key parameters

• cluster (repeatable, required) — defines a cluster to participate in. Value is a comma-separated key=value string:

  • id=N — positive integer cluster identifier, must match the cluster_id used by clusterer consumers (dialog, usrloc, etc.)
  • multicast=A.B.C.D:PORT — IPv4 multicast group and UDP port. Address must be in the 224.0.0.0/4 range. Two clusters on the same LAN can share the same address with different ports, or use entirely different addresses — both isolate traffic correctly.
  • password=STRING — per-cluster shared secret; falls back to the global password modparam if not set
  • bin_socket=bin:IP:PORT — BIN socket to advertise to clusterer peers for this cluster; required when multiple clusters are defined
  • manage_shtags=0|1 — per-cluster override for automatic sharing tag failover; falls back to the global manage_shtags modparam if not set, regardless of modparam order in the config file

• password (default: 3eCrEt*5629) — the shared cluster secret. A per-cluster bootstrap key is derived from it with a memory-hard KDF (scrypt, or Argon2id in the libsodium build); the session key is agreed via X25519 ECDH (see Security). All nodes in a cluster must use the same password. Change this in production.

• manage_shtags (default: 1) — global default for automatic sharing tag management. Can be overridden per cluster via the manage_shtags= key in the cluster string.

• master_stickiness (default: 1) — controls whether a live master is kept when a higher-IP node joins. 1 (sticky) = the master stays and the higher-IP joiner becomes the backup, minimising handovers; 0 = the highest-IP node always becomes master. Global default, overridable per cluster via the master_stickiness= key in the cluster string.

• query_time (default: 5) — heartbeat interval in seconds. Controls the election window (3 × query_time) and the peer purge window (6 × query_time). Valid range: 1–60.

• on_config_mismatch (default: reject) — policy when a node's consistency-critical settings (manage_shtags, master_stickiness, query_time) differ from the running cluster: reject (refuse the join, node shuts down), warn (join but log CONFIG MISMATCH), or adopt (adopt the cluster's settings at runtime). See Security.

• my_ip (default: auto-detected) — pin the controller identity to a specific local IPv4 address. Useful on multi-homed hosts.

• interface (default: auto-detected) — pin the controller to a named network interface (e.g. eth0). Ignored if my_ip is set.

Automatic sharing tag management (manage_shtags)

When manage_shtags=1 (the default), the controller master is the single authority for sharing tag state in the clusterer module. No script-level event routes or MI commands are needed to handle failover:

• Startup: all local sharing tags are forced to backup state regardless of the =active value in the clusterer config. This prevents a restarting node from stealing the active tag from a running cluster.
• Bootstrap: the first node to start alone activates its backup tags exactly once, after waiting query_time × 3 seconds with no master response.
• Failover: when any node departs, the master immediately promotes its own backup tags to active for that cluster.
• Rejoin: a returning node always starts in backup state and never reclaims the active tag from the current holder.
• Blocking: while manage_shtags=1, the clusterer_set_tag_active MI command and $shtag() script variable setter are blocked for controller-managed clusters, preventing accidental conflicts with automatic failover.

manage_shtags can be set globally and overridden per cluster, regardless of the order modparams appear in the config file:

# Global on, one cluster opts out
modparam("clusterer_controller", "manage_shtags", 1)
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444,manage_shtags=0")

# Global off, one cluster opts in
modparam("clusterer_controller", "manage_shtags", 0)
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,manage_shtags=1")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444")

Minimal config (identical on all nodes, only BIN socket IP differs)

socket=bin:10.22.23.191:3857

loadmodule "proto_bin.so"

loadmodule "clusterer.so"
modparam("clusterer", "use_controller", 1)
modparam("clusterer", "sharing_tag",   "vip1/1=active")
modparam("clusterer", "ping_interval", 4)
modparam("clusterer", "ping_timeout",  1500)

loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster",  "id=1,multicast=239.0.90.1:3333")
modparam("clusterer_controller", "password", "MyStr0ngPassw0rd!")

Multiple clusters

Clusters are isolated by their multicast address and/or port, so you can use the same multicast address with different ports, or entirely different addresses — both work:

# Same multicast address, different ports (same LAN segment)
socket=bin:10.0.1.10:5566
socket=bin:10.0.2.10:5567

modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.1.10:5566")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444,bin_socket=bin:10.0.2.10:5567")

# Or different addresses entirely
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.1.10:5566")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.2:3333,bin_socket=bin:10.0.2.10:5567")

Each cluster runs an independent controller worker process with its own election, membership, and sharing tag state. Packets from different clusters never mix: every packet carries a 2-byte cluster_id in its cleartext header, and a node silently drops any packet whose cluster_id differs from its own before decryption — so multiple clusters can even share the same multicast address and port without interfering, and foreign traffic never trips the wrong-password machinery. Two clusters merge only if they are configured identically (same address, port, cluster_id and password).

MI commands

• cc_list_members — list the current members of each cluster with their node_id, status (master / backup / member) and BIN sockets.
• cc_node_info node_id=N — full info for a specific node across all clusters.
• cc_list_config — list every configured cluster and its resolved effective settings (multicast, my_ip, bin_socket, query_time, master_stickiness, manage_shtags, shtag_mode, member_count). The password is never exposed. Useful for confirming a per-cluster override took effect.
• cc_shtag_force cluster_id=N node_id=M — pin the active sharing tag to a chosen node, overriding automatic master-driven allocation (master-only). The override is carried in MEMBER_LIST, survives master fail-over, and auto-clears if the forced node departs. shtag_mode then reports override:M.
• cc_shtag_auto cluster_id=N — clear the override and resume automatic, master-driven sharing-tag allocation (master-only).

Security

• Each packet's payload is sealed with an AEAD, behind a 2-byte cleartext magic (a key-tier selector, readable before decryption), a 2-byte cluster_id, and the nonce — everything meaningful (type, seq, IPs, pubkeys, salts, member lists) is inside the sealed payload. The magic and cluster_id are bound into the AEAD tag as AAD, so a captured packet can't be re-stamped onto another cluster_id on a shared multicast+password group. The AEAD is AES-256-GCM (WolfSSL) by default, or XChaCha20-Poly1305 when built against libsodium (its 192-bit nonce removes any random-nonce collision concern) — a build-time choice; all nodes in a cluster must match, and the active suite is logged at startup.
• A bootstrap key protects only the admission handshake and the split-brain beacon; it is derived from the shared password with a memory-hard KDF — scrypt (N=2¹⁶) or Argon2id in the libsodium build, with a per-cluster salt, derived once at startup — so a password captured from a bootstrap packet can't be brute-forced cheaply offline. The session key used for all other traffic is derived (HKDF-SHA256) from a random master salt agreed via per-worker X25519 ECDH — the salt is never sent in the clear, and the join nonce binds each wrap to a single exchange. A startup warning fires if the password is the default or has low estimated entropy.
• Replay protection via per-source 32-bit monotonic sequence numbers, with no dependency on clock synchronisation.
• Rate limiting (256 source slots, 20 pkt/s) applied before any decryption attempt to blunt CPU-exhaustion floods on the multicast group.
• Join authentication: the master sends an encrypted, unforgeable JOIN_REJECT after repeated bootstrap-decrypt failures. A wrong-password node can't read that reject, so it also self-detects — while joining it counts undecryptable packets from other peers and, at the join deadline, shuts down ("cannot authenticate — wrong password?") instead of self-promoting into a lone split-brain master. An active member ignores JOIN_REJECT, so a correct-password peer can never be evicted. A full peer table rejects new spoofed-IP joins.
• Rogue-traffic isolation: a node requests a re-key only for an undecryptable session packet from its current master (a legitimate rotation). Undecryptable session packets from any other source (e.g. a wrong-password or malicious node on the group) are ignored, so they can't force the cluster into a re-JOIN churn.
• Config-consistency enforcement: all nodes of a cluster must use identical manage_shtags, master_stickiness and query_time — a per-node mismatch would otherwise cause silent, inconsistent failover/sharing-tag behaviour (e.g. a master with manage_shtags=0 leaves no node holding the active tag). Each node advertises these effective settings in its ALIVE heartbeat and JOIN_REQ; the on_config_mismatch modparam picks the policy when a node tries to join an established cluster with different settings: reject (default) — the master returns a JOIN_REJECT and the misconfigured node shuts down with a clear message; warn — the node joins but a single deduplicated CONFIG MISMATCH warning is logged; adopt — the joining node adopts the running cluster's settings at runtime (and cc_list_config then reports the adopted values).

---

Changes to the clusterer module

New controller API (clusterer_ctrl)

A new clusterer_ctrl.c/.h pair exposes a load-time API (load_clusterer_ctrl_binds) through which an external module can drive the clusterer topology at runtime:

• set_my_identity(cluster_id, node_id, bin_url) — register this node's identity post-fork
• add_node / remove_node — add/remove peers at runtime
• update_identity — correct node_id after master assignment
• sync_current_id — sync process-local current_id after fork
• activate_backup_shtags / force_backup_shtags — sharing tag control
• set_shtag_managed — lock a cluster's tags against manual changes

use_controller=1 mode

A new use_controller modparam tells clusterer to create stub clusters at startup without requiring DB or static neighbor config. Identity and peer list are filled in post-fork by the controller. When use_controller=0 (the default), the clusterer is completely unaffected — all existing behaviour, DB loading, static neighbors, sharing tag event routes, and MI commands work exactly as before.

Bug fixes

• NULL dereference in cl_set_state and handle_remove_node: NULL guard was placed after lock_get(current_node->lock), making it dead code. Both moved before the lock call.
• Spurious "Sync request aborted!" on first-node startup: in controller mode, sync_req_time was left at zero (epoch) before identity was assigned post-fork, causing the seed fallback timer to fire immediately. Fixed by always calling gettimeofday() when queuing a sync request.
• Double _current_id_shm allocation in mod_init: second allocation inside the use_controller block removed.
• O(n²) shtag_activate_all_backup: replaced with a collect-then-activate pattern.
• NULL current_node dereference in bin_rcv_cl_packets: when the topology is built dynamically via clusterer_ctrl, a cluster can exist and receive BIN packets before this node's identity is established. An inbound packet in that window dereferenced a NULL cl->current_node and crashed the receiver; such packets are now dropped until the local identity is set.

---

Testing

Five HA scenarios were tested on a 3-node cluster (10.22.23.191–193) with dialog replication and sharing tag failover:

• Sequential cold start — first node becomes master; with master_stickiness=1 it stays master as higher-IP nodes join, each newcomer taking the backup role and the previous backup dropping to member. Tags activated on the master, others in backup ✅
• Master hard stop (kill -9) — the backup (highest-IP survivor) is promoted to master in ~3s using the preserved key (no re-keying, no election loop), a new backup is designated, and its backup tags are activated ✅
• Master restart (rejoin) — node rejoins as member/backup, master stays put, tags remain on the current master ✅
• Member hard stop — master detects departure and activates its own backup tags for that cluster ✅
• Member restart (rejoin) — node rejoins cleanly, no tag disruption ✅
• master_stickiness=0 — verified a higher-IP joiner takes over as master immediately (the non-sticky path) ✅
• Wrong-password node — a node configured with a different password shuts down at the join deadline ("cannot authenticate … wrong password?") instead of forming a lone split-brain master, and the live cluster is completely undisturbed (zero session-mismatch/demote events); a corrected node then rejoins normally ✅
• Rogue-joiner tool (test/cc_join_reject_test.py) — a standalone script (no node config changes) run from a non-member host on the multicast segment: it sends unauthenticated JOIN_REQs and confirms the master emits a JOIN_REJECT, then floods fake MASTER_ALIVE packets and confirms the cluster ignores them (no re-JOIN churn) ✅
• Sharing-tag override — cc_shtag_force pins the active tag onto a non-master node (all nodes report override:N), the override survives a master kill -9 (the new master keeps honouring it), auto-clears when the forced node departs, and cc_shtag_auto restores automatic allocation; error paths (non-master, unknown node/cluster) rejected ✅
• Split-brain prevention & merge — a simultaneous all-node cold start now converges to a single cluster: lower-IP nodes log "deferring self-promotion" and join the highest-IP starter (decrypt-failure warnings dropped from ~700/node to a handful), and an induced divergent-key split is merged via MASTER_BEACON ("superior master … found via beacon") ✅
• libsodium suite — the same HA scenarios (join, failover, rejoin) verified end-to-end with the XChaCha20-Poly1305 + Argon2id build ✅

Full test details are documented in modules/clusterer_controller/doc/clusterer_controller_tests.xml.

---

Limitations (alpha)

• IPv4 only (multicast address must be in 224.0.0.0/4)
• Rough clock agreement between nodes is recommended — the election window is quantized to query_time — but strict NTP is not required for correctness; replay protection is sequence-number based, not time-window based
• Multicast routing must be available on the network (LAN via IGMPv2/v3, or routed via PIM-SM or similar multicast routing protocol for multi-site deployments)
• The bootstrap key uses a memory-hard KDF (scrypt, or Argon2id in the libsodium build) over a fixed per-cluster salt (the password is a shared group secret, so the salt can't be per-node); a strong, high-entropy password is still recommended — the KDF raises the cost of an offline guess but doesn't rescue a weak password
• The crypto suite is a build-time choice (WolfSSL AES-256-GCM/scrypt vs. libsodium XChaCha20-Poly1305/Argon2id); the two wire formats are not interoperable, so a cluster can't mix nodes built with different suites (the active suite is logged at startup to make mismatches visible)

---

Roadmap (not in this PR)

Planned follow-ups (also documented under Planned Features in the admin guide):

• Node maintenance mode — an MI command to place a member into maintenance: it becomes fully inert (never master, no elections, holds no shtags — any it holds are released and re-allocated, all cluster activity suspended) while staying visible with a maintenance state. A new evict_on_maintenance modparam (default 1) controls the data plane: 1 evicts the node from clusterer (clean drain, re-sync on exit); 0 keeps it in the clusterer topology so it keeps replicating usrloc/dialog/etc. and stays warm. When it lands, cc_shtag_force will additionally reject pinning a tag onto a maintenance node. It will also expose the state to the config script and event system: cc_in_maintenance() / cc_node_in_maintenance(cluster_id, node_id) script functions; cc_is_evicted() / cc_node_is_evicted(cluster_id, node_id) (distinguishing a node drained out of clusterer from one in maintenance but still warm and replicating); and an E_MAINTENANCE_MODE_CHANGE event (cluster_id, node_id, ip, state, is_local, evicted) usable from an event_route or any event subscriber. Two channels, two purposes: the event (actionable "react now" signal) is distributed over the clusterer BIN protocol, so only non-evicted members raise it and react — a node evicted from clusterer (evict_on_maintenance=1) is outside the BIN mesh and won't receive it (it's inert anyway). The queryable per-peer state (maintenance + evicted flags) is instead propagated over the controller's own multicast, which a maintenance node stays on even when evicted — so cc_(node_)in_maintenance() / cc_(node_)is_evicted() keep working on any node, including one evicted from clusterer, for both the local node and other peers. In short: an evicted node can't react to the event, but it can still query other nodes' status.

• Observability and script integration — surface cluster state through OpenSIPS's standard interfaces rather than logs alone: statistics (role, member count, master changes, joins/leaves, JOIN_REJECTs, decrypt failures, config mismatches, split-brain merges) via get_statistics/monitoring; events raised on state transitions (became master, demoted, node joined/left, split-brain merged, config mismatch, auth reject) for event_route/subscribers; pseudo-variables ($cc_role, $cc_is_master, $cc_master_ip, $cc_member_count, with a cluster_id index) for script branching; and script functions (cc_is_master(), alongside the maintenance accessors).

(The previously-listed sharing-tag override, resume-auto-tagging and shtag_mode items are now implemented in this PR — see the MI commands section above.)

[razvancrainea]

Thank you for your contribution! However, our policy is to push features only in master - therefore, as this is a new feature, will not get to be backported to 3.6, so the PR will never be merged. So I am closing it. Please do continue this work in the master branch, in PR #4074.

Thank you again!
Răzvan