clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast

[Lt-Flash]

OpenSIPS devel / 4.1 target. This PR is self-contained — the full feature
description is inline below.

Devel-specific changes

Everything below this section describes the feature as a whole. This section
collects the parts that are specific to the devel/4.1 branch — the core-API
adaptations, plus a few robustness/log fixes made while validating it on
hardware.

Port adaptations (devel core API)

• socket_info refactor. cc_discover_bin_sockets() iterates
  protos[PROTO_BIN].listeners as struct socket_info_full (with next/prev)
  and reads the embedded socket_info — the socket-list change that landed on
  devel.
• Re-seated clusterer edits. The core edits are merged onto devel's
  clusterer.c / clusterer_mod.c / node_info.h. They combine cleanly with
  devel's new inter-cluster bridges feature and the
  clusterer_enable_rerouting guard — the receive-path "is this for me?"
  check is now dest_id != cluster_self_id(cl) and honours the rerouting
  guard. Per-cluster identity (cluster_self_id()) is used everywhere a node
  must know its own id in a specific cluster.
• No unrelated core changes. This PR touches only clusterer_controller and
  the clusterer module — no OpenSIPS core files are modified.
• Crypto suite. With libsodium on the build host the module is built
  sodium-only (XChaCha20-Poly1305 + Argon2id, and X25519 / HKDF-SHA256 /
  RNG from libsodium too — WolfSSL is not linked, keeping the module binary a
  few hundred KB instead of several MB); without it, WolfSSL provides the
  fallback (AES-256-GCM + scrypt). All nodes in a cluster must be built the
  same way.

The clusterer module is a strict no-op unless at least one cluster is
registered as controller-managed via a cluster_options line with
use_controller=1; the controller is opt-in.

Startup stabilisation & log hygiene (devel-only fixes)

A near-simultaneous cold start always converged, but was noisy. Four fixes make
it clean and deterministic:

1. clusterer_controller: bootstrap vs session decrypt failures.
   cc_decrypt_pkt now distinguishes a bootstrap-key failure (a real
   JOIN_REQ → genuinely wrong password / foreign cluster / tampering → stays
   WARN) from a transient session-key mismatch during a (re)key or split-brain
   heal (→ DBG). A healthy start no longer logs alarming "wrong password or
   tampered" lines.
2. clusterer_controller: no premature self-promotion. The split-brain
   defer budget resets on a fresh higher-IP JOIN_REQ, so a lower-IP node keeps
   waiting for a demonstrably-live higher-IP peer to take master instead of
   self-promoting into a divergent-key split brain. A new absolute cap
   (CC_JOIN_DEFER_HARDMAX) still guarantees termination if a peer is stuck
   joining forever.
3. clusterer_controller: JOIN_REQ rate-limit (CC_JOIN_REQ_MIN_US) so a
   key-mismatch burst cannot flood the multicast group.
4. clusterer: seed sync-fallback is not an error. On a fresh start a seed
   node queues a capability sync (e.g. dialog-dlg-repl), finds no peer with
   data, and after seed_fb_interval falls back to synced. sync_check_timer
   logged this at ERROR ("Sync request aborted! (no donor found in due
   time)") — but that is the seed fallback working as designed, so it is
   downgraded to DBG. The genuine partial-sync failure path (the
   SYNC_IN_PROGRESS timeout) is unchanged.

Net effect on a simultaneous 3-node start: "wrong password" warnings several →
0; nodes wrongly self-promoting → only the highest-IP node; JOIN_REQ
seen ~30+ → ~5; sync ERRORs several → 0. Wrong-password rejection is
unchanged (the mismatched node still defers then shuts down, and the healthy
cluster is undisturbed).

Hybrid clusters (native + controller in one instance)

A single instance can run DB-native / statically-provisioned clusters and
controller-managed clusters side by side (e.g. a fixed DB replication
cluster alongside a zero-config controller HA cluster). A cluster_id is
exclusively one kind or the other; declaring the same id both ways is
rejected at startup. See Changes to the clusterer module → hybrid below for
the rules and config examples.

Validated on devel:

• Coexistence — native cluster 10 (db_mode=0, static my_node_info) +
  controller cluster 1 both initialise; the controller pre-creates its
  cluster-1 stub and never touches the DB path of cluster 10.
• Overlap guard — declaring native cluster 1 and controller cluster 1
  aborts module init with a clear error, so the ambiguity can never reach
  runtime.

---

Overview

This PR introduces clusterer_controller, a new OpenSIPS module that drives the existing clusterer module's topology automatically at runtime. Rather than replacing the clusterer, it acts as a control plane on top of it — dynamically adding and removing nodes, electing a master, and managing sharing tag failover, without requiring a database, static node lists, or manual configuration.

The clusterer module itself is unchanged in its normal mode of operation. For any cluster without a cluster_options use_controller=1 line, the clusterer behaves exactly as before — DB-backed or static neighbor config, standard sharing tag event routes and MI commands, no multicast involvement whatsoever. The controller is strictly opt-in, per cluster.

Peer discovery is done via authenticated, encrypted UDP multicast (AES-256-GCM by default; XChaCha20-Poly1305 + Argon2id when the module is built against libsodium). Nodes announce themselves on a shared multicast group and elect a master deterministically (master / backup / member roles, see below), and the master assigns node IDs and maintains the member list. All of this is fed into the clusterer module at runtime through a new clusterer_ctrl API, so every existing clusterer consumer (dialog, usrloc, dispatcher, etc.) works without any changes to how they use the clusterer.

By default, multicast operates within a single LAN segment. For multi-site deployments, discovery can be extended across routable networks using multicast routing protocols such as PIM-SM (Protocol Independent Multicast — Sparse Mode) or similar services, provided the network infrastructure supports it.

The result is a fully identical config across all nodes — the only difference between machines is the BIN socket IP address.

---

How it works

Each node joins a multicast group and broadcasts a JOIN_REQ. The master responds with a KEY_GRANT (handing the joiner the cluster session key) and NODE_ASSIGN packets that allocate a node_id to every member and feed it into the clusterer module via the clusterer_ctrl API. The master periodically broadcasts MEMBER_LIST so late joiners and re-joiners converge quickly, and a 1-second MASTER_ALIVE keepalive lets peers detect master failure within ~3 seconds.

Each cluster has three roles: master (the active coordinator), backup (the standby that takes over on master failure — always the highest-IP non-master) and member. Election is deterministic across nodes and needs no negotiation round-trips. The master_stickiness parameter (default 1) keeps a live master in place: a higher-IP node that joins becomes the backup rather than preempting the master, so joins cause no handover (the previous backup simply drops to member if the newcomer has a higher IP). With master_stickiness=0 the highest-IP node always becomes master.

On master failure — graceful GOODBYE or keepalive timeout — the backup is promoted immediately. Because the session key is generated once at cluster bootstrap and preserved across every master change, the new master reuses the key every member already holds: no re-keying and no re-join storm.

Split-brain is prevented and healed by three cooperating mechanisms:

1. Prevention at join time — simultaneously-starting nodes see each other's JOIN_REQs, so at the join deadline a node that has seen a higher-IP starter defers self-promotion (bounded) and joins it instead. Only the highest-IP starter becomes master, so independent-key lone masters never form in the first place.
2. Same-key yield — two masters that share a session key (e.g. after a partition heals) read each other's MASTER_ALIVE; the lower-IP one yields.
3. Divergent-key merge — masters bootstrapped independently hold different session keys and can't read each other's MASTER_ALIVE, so each emits a bootstrap-key MASTER_BEACON (readable by any correctly-configured node). On hearing a beacon from a superior partition (larger member count, ties broken by higher IP), a node re-joins that master and adopts its key, converging the whole cluster onto one master and key.

Because node identity and peer lists are injected into clusterer at runtime, no DB reads or static neighbor modparams are needed. All existing clusterer consumers continue to work transparently.

Node state machine

Each cluster's worker runs its own copy of this state machine (state lives in shm — cl->peers->node_state plus the is_master/is_backup role flags):

stateDiagram-v2
    [*] --> Joining : worker start
    Joining --> Joining : defer and retry (higher-IP peer joining / transient auth failures)
    Joining --> Member : KEY_GRANT / MEMBER_LIST
    Joining --> Master : join deadline + highest IP (bootstrap, generates session key)
    Joining --> Terminated : JOIN_REJECT targeted at us, or auth failures after defer budget
    Terminated --> [*] : exit(-1)

    state "Active (CC_NODE_ACTIVE)" as Active {
        Member --> Backup : elected backup (highest-IP non-master)
        Backup --> Member : displaced by higher-IP joiner
        Backup --> Master : master dead ~3 s / GOODBYE / KEY_HANDOFF
        Member --> Master : election win (no backup, or stickiness=0 preemption)
        Master --> Member : superior master seen / beacon merge (re-JOIN, adopt winner's key)
    }
    Active --> [*] : GOODBYE (graceful shutdown)

Loading

Joining (CC_NODE_NEW) — entered at worker start. The node multicasts JOIN_REQ (rate-limited) and is receive-only: it never sends ALIVE or MASTER_ALIVE. Three ways out:

• → Member: a KEY_GRANT (session key handed over; node_id arrives via NODE_ASSIGN) or a MEMBER_LIST from the current master.
• → Master: the join deadline (query_time) expires with no master answering and this node is the highest-IP joiner — it self-promotes and generates the cluster session key (bootstrap). The defer and retry self-loop is what prevents split-brain here: if a higher-IP peer is also still joining, or auth failures might be transient, the node re-arms the deadline (bounded by the defer budget) instead of promoting.
• → Terminated: a JOIN_REJECT targeted at this node (wrong password or on_config_mismatch=reject), or the deadline passing with the undecryptable-packet threshold hit after the full defer budget — the "wrong password?" shutdown.

Active (CC_NODE_ACTIVE) — the three roles are the same state, differing only in duties (member/backup send ALIVE every query_time; the master also sends the 1-second MASTER_ALIVE, MEMBER_LIST, NODE_ASSIGN, KEY_GRANT and the periodic bootstrap-key MASTER_BEACON):

• Member ⇄ Backup: every election deterministically marks the highest-IP non-master as backup; a higher-IP joiner displaces the current backup back to member.
• Backup → Master: promotion on master death (3 missed MASTER_ALIVE, ~3 s), a graceful GOODBYE, or KEY_HANDOFF. The session key is preserved across the handover, so no re-key storm.
• Member → Master (election win): direct promotion when no backup exists, or preemption by a higher-IP node when master_stickiness=0. With stickiness on (the default), a live master is never displaced by a joiner.
• Master → Member (superior master / beacon merge): the split-brain cure — a MASTER_BEACON or MASTER_ALIVE from a superior master (larger member count, tie broken by higher IP) demotes this one, which re-joins to adopt the winner's key while staying active.

Not drawn to keep the diagram readable: an active node that sees session-key mismatches from its own master sends a re-key JOIN_REQ without leaving the active state.

---

New module: clusterer_controller

Key parameters

• cluster (repeatable, required) — defines a cluster to participate in. Value is a comma-separated key=value string:

  • id=N — positive integer cluster identifier, must match the cluster_id used by clusterer consumers (dialog, usrloc, etc.)
  • multicast=A.B.C.D:PORT — IPv4 multicast group and UDP port. Address must be in the 224.0.0.0/4 range. Two clusters on the same LAN can share the same address with different ports, or use entirely different addresses — both isolate traffic correctly.
  • password=STRING — per-cluster shared secret; falls back to the global password modparam if not set
  • bin_socket=bin:IP:PORT — BIN socket to advertise to clusterer peers for this cluster; required when multiple clusters are defined
  • manage_shtags=0|1 — per-cluster override for automatic sharing tag failover; falls back to the global manage_shtags modparam if not set, regardless of modparam order in the config file

• password (default: 3eCrEt*5629) — the shared cluster secret. A per-cluster bootstrap key is derived from it with a memory-hard KDF (scrypt, or Argon2id in the libsodium build); the session key is agreed via X25519 ECDH (see Security). All nodes in a cluster must use the same password. Change this in production.

• manage_shtags (default: 1) — global default for automatic sharing tag management. Can be overridden per cluster via the manage_shtags= key in the cluster string.

• master_stickiness (default: 1) — controls whether a live master is kept when a higher-IP node joins. 1 (sticky) = the master stays and the higher-IP joiner becomes the backup, minimising handovers; 0 = the highest-IP node always becomes master. Global default, overridable per cluster via the master_stickiness= key in the cluster string.

• query_time (default: 5) — heartbeat interval in seconds. Controls the election window (3 × query_time) and the peer purge window (6 × query_time). Valid range: 1–60.

• on_config_mismatch (default: reject) — policy when a node's consistency-critical settings (manage_shtags, master_stickiness, query_time) differ from the running cluster: reject (refuse the join, node shuts down), warn (join but log CONFIG MISMATCH), or adopt (adopt the cluster's settings at runtime). See Security.

• my_ip (default: auto-detected) — pin the controller identity to a specific local IPv4 address. Useful on multi-homed hosts.

• interface (default: auto-detected) — pin the controller to a named network interface (e.g. eth0). Ignored if my_ip is set.

Automatic sharing tag management (manage_shtags)

When manage_shtags=1 (the default), the controller master is the single authority for sharing tag state in the clusterer module. No script-level event routes or MI commands are needed to handle failover:

• Startup: all local sharing tags are forced to backup state regardless of the =active value in the clusterer config. This prevents a restarting node from stealing the active tag from a running cluster.
• Bootstrap: the first node to start alone activates its backup tags exactly once, after waiting query_time × 3 seconds with no master response.
• Failover: when any node departs, the master immediately promotes its own backup tags to active for that cluster.
• Rejoin: a returning node always starts in backup state and never reclaims the active tag from the current holder.
• Blocking: while manage_shtags=1, the clusterer_set_tag_active MI command and $shtag() script variable setter are blocked for controller-managed clusters, preventing accidental conflicts with automatic failover.

manage_shtags can be set globally and overridden per cluster, regardless of the order modparams appear in the config file:

# Global on, one cluster opts out
modparam("clusterer_controller", "manage_shtags", 1)
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444,manage_shtags=0")

# Global off, one cluster opts in
modparam("clusterer_controller", "manage_shtags", 0)
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,manage_shtags=1")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444")

Minimal config (identical on all nodes, only BIN socket IP differs)

socket=bin:10.22.23.191:3857

loadmodule "proto_bin.so"

loadmodule "clusterer.so"
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
modparam("clusterer", "sharing_tag",   "vip1/1=active")
modparam("clusterer", "ping_interval", 4)
modparam("clusterer", "ping_timeout",  1500)

loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster",  "id=1,multicast=239.0.90.1:3333")
modparam("clusterer_controller", "password", "MyStr0ngPassw0rd!")

Multiple clusters

Clusters are isolated by their multicast address and/or port, so you can use the same multicast address with different ports, or entirely different addresses — both work:

# Same multicast address, different ports (same LAN segment)
socket=bin:10.0.1.10:5566
socket=bin:10.0.2.10:5567

# both clusters registered controller-managed on the clusterer side
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
modparam("clusterer", "cluster_options", "cluster_id=2, use_controller=1")

modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.1.10:5566")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.1:4444,bin_socket=bin:10.0.2.10:5567")

# Or different addresses entirely
modparam("clusterer_controller", "cluster",
    "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.1.10:5566")
modparam("clusterer_controller", "cluster",
    "id=2,multicast=239.0.90.2:3333,bin_socket=bin:10.0.2.10:5567")

Each cluster runs an independent controller worker process with its own election, membership, and sharing tag state. Packets from different clusters never mix: every packet carries a 2-byte cluster_id in its cleartext header, and a node silently drops any packet whose cluster_id differs from its own before decryption — so multiple clusters can even share the same multicast address and port without interfering, and foreign traffic never trips the wrong-password machinery. Two clusters merge only if they are configured identically (same address, port, cluster_id and password).

MI commands

• cl_ctr_list_members — list the current members of each cluster with their node_id, status (master / backup / member) and BIN sockets.
• cl_ctr_node_info node_id=N — full info for a specific node across all clusters.
• cl_ctr_list_config — list every configured cluster and its resolved effective settings (multicast, my_ip, bin_socket, query_time, master_stickiness, manage_shtags, shtag_mode, member_count). The password is never exposed. Useful for confirming a per-cluster override took effect.
• cl_ctr_shtag_force cluster_id=N node_id=M — pin the active sharing tag to a chosen node, overriding automatic master-driven allocation (master-only). The override is carried in MEMBER_LIST, survives master fail-over, and auto-clears if the forced node departs. shtag_mode then reports override:M.
• cl_ctr_shtag_auto cluster_id=N — clear the override and resume automatic, master-driven sharing-tag allocation (master-only).

Script variables

Read-only pseudo-variables expose live cluster state to the routing script (read from shared memory, so available in every process — SIP workers included), so a check like only the master runs this job needs no MI round-trip. Each takes an optional cluster id: the bare form ($cl_ctr_role) resolves the sole configured cluster, while an explicit id ($cl_ctr_role(2)) reads any specific cluster — so a node that participates in several clusters can query each one independently. With more than one cluster defined the bare form returns NULL and logs a one-time warning, so the cluster must be named.

• $cl_ctr_role — master / backup / member / joining
• $cl_ctr_is_master — 1 / 0
• $cl_ctr_master_ip, $cl_ctr_backup_ip — current master / backup IP (NULL if none)
• $cl_ctr_node_id — this node's node_id in that cluster (a node may hold different ids in different clusters)
• $cl_ctr_my_ip, $cl_ctr_members — controller identity IP; live member count
• $cl_ctr_shtag_mode (auto/forced), $cl_ctr_forced_node — sharing-tag allocation mode and pinned node id

# single-cluster node: the bare form
if ($cl_ctr_is_master)
    route(master_only_job);

# multi-cluster node: address each cluster by id
if ($cl_ctr_is_master(1))                 # controller cluster 1
    route(dialog_master_duty);
if ($cl_ctr_role(2) == "backup")          # a different cluster
    route(warm_standby_prep);
xlog("cl2: role=$cl_ctr_role(2) master=$cl_ctr_master_ip(2) members=$cl_ctr_members(2) myid=$cl_ctr_node_id(2)\n");

The convention is queries are nouns (variables), actions are verbs (MI commands): this node's own state is read through the variables above and changed through MI (cl_ctr_shtag_force / cl_ctr_shtag_auto). Script action functions (verbs — e.g. cl_ctr_set_maintenance() for a node to change its own state from the routing logic) are planned together with node maintenance mode; see the Roadmap.

Input validation. The optional cluster-id key is parsed at config load: a non-numeric, negative, zero, or out-of-range value (capped at 9 digits so it can never overflow into a valid-looking id) is rejected with a descriptive error and fails the config check rather than binding to the wrong cluster. The per-peer functions below are equally defensive at runtime — an out-of-range, negative, or non-numeric argument passed via $var resolves to not-found rather than faulting.

Script functions

Per-peer lookups take two arguments (cluster_id, node_id) and so are functions (a comma inside a pvar's parens is ambiguous once the pvar is used as a function argument). Booleans return true/false for if(); value lookups write an output variable:

• cl_ctr_node_is_master(cluster_id, node_id) — true if that node is master
• cl_ctr_node_present(cluster_id, node_id) — true if node_id is a live member
• cl_ctr_get_node_role(cluster_id, node_id, $out) — writes master/backup/member
• cl_ctr_get_node_ip(cluster_id, node_id, $out) — writes that node's IP

if (cl_ctr_node_present(1, 3) && cl_ctr_node_is_master(1, 3)) {
    cl_ctr_get_node_ip(1, 3, $var(ip));
    xlog("node 3 ($var(ip)) leads cluster 1\n");
}

Security

• Each packet's payload is sealed with an AEAD, behind a 2-byte cleartext magic (a key-tier selector, readable before decryption), a 2-byte cluster_id, and the nonce — everything meaningful (type, seq, IPs, pubkeys, salts, member lists) is inside the sealed payload. The magic and cluster_id are bound into the AEAD tag as AAD, so a captured packet can't be re-stamped onto another cluster_id on a shared multicast+password group. The AEAD is AES-256-GCM (WolfSSL) by default, or XChaCha20-Poly1305 when built against libsodium (its 192-bit nonce removes any random-nonce collision concern) — a build-time choice; all nodes in a cluster must match, and the active suite is logged at startup.
• A bootstrap key protects only the admission handshake and the split-brain beacon; it is derived from the shared password with a memory-hard KDF — scrypt (N=2¹⁶) or Argon2id in the libsodium build, with a per-cluster salt, derived once at startup — so a password captured from a bootstrap packet can't be brute-forced cheaply offline. The session key used for all other traffic is derived (HKDF-SHA256) from a random master salt agreed via per-worker X25519 ECDH — the salt is never sent in the clear, and the join nonce binds each wrap to a single exchange. A startup warning fires if the password is the default or has low estimated entropy.
• Replay protection via per-source 32-bit monotonic sequence numbers, with no dependency on clock synchronisation.
• Rate limiting (256 source slots, 20 pkt/s) applied before any decryption attempt to blunt CPU-exhaustion floods on the multicast group.
• Join authentication: the master sends an encrypted, unforgeable JOIN_REJECT after repeated bootstrap-decrypt failures. A wrong-password node can't read that reject, so it also self-detects — while joining it counts undecryptable packets from other peers and, at the join deadline, shuts down ("cannot authenticate — wrong password?") instead of self-promoting into a lone split-brain master. An active member ignores JOIN_REJECT, so a correct-password peer can never be evicted. A full peer table rejects new spoofed-IP joins.
• Rogue-traffic isolation: a node requests a re-key only for an undecryptable session packet from its current master (a legitimate rotation). Undecryptable session packets from any other source (e.g. a wrong-password or malicious node on the group) are ignored, so they can't force the cluster into a re-JOIN churn.
• Config-consistency enforcement: all nodes of a cluster must use identical manage_shtags, master_stickiness and query_time — a per-node mismatch would otherwise cause silent, inconsistent failover/sharing-tag behaviour (e.g. a master with manage_shtags=0 leaves no node holding the active tag). Each node advertises these effective settings in its ALIVE heartbeat and JOIN_REQ; the on_config_mismatch modparam picks the policy when a node tries to join an established cluster with different settings: reject (default) — the master returns a JOIN_REJECT and the misconfigured node shuts down with a clear message; warn — the node joins but a single deduplicated CONFIG MISMATCH warning is logged; adopt — the joining node adopts the running cluster's settings at runtime (and cl_ctr_list_config then reports the adopted values).
• node_id allocation & identity: ids are allocated only by the current master, serialised under the peer-table lock — a joining node never picks its own. The master hands out the lowest unused id from the live peer table, so a node that has failed but not yet timed out still holds its slot and its id is never reused for a different joiner — new nodes get a distinct id even during the failure-detection window. A node that restarts and rejoins from the same address reuses its previous id (and its replay counter is reset), so ids stay stable across restarts. Peers are keyed by source IP, so each node must present a stable, unique source address: two nodes behind one address (e.g. NAT) would share a slot/node_id.
• Trust model — shared secret, not per-node identity: the cluster is a single shared-secret trust domain. Authentication proves only that a peer holds the password; it does not bind a cryptographic identity to a node, and there is no per-node authorisation or revocation. So any holder of the password is a fully trusted member and can legitimately win the highest-IP election and take master — there is no "may be member" vs "may be master" distinction. An attacker without the password cannot influence the election at all (forged/replayed MASTER_ALIVE and beacons fail the AEAD tag or the per-source sequence check and are dropped before any election logic runs); the residual exposure is a malicious insider that already holds the key. Per-node keypairs with enrolment/revocation — distrust one node without re-keying the cluster — are on the Roadmap.

---

Changes to the clusterer module

Opt-in build — zero change to stock clusterer

clusterer_controller is excluded from the default build (added to exclude_modules in Makefile.conf.template), like the other modules with external-library dependencies. You enable it by adding it to include_modules:

include_modules= clusterer_controller

All of the clusterer-side integration below (the clusterer_ctrl API, the cluster_options modparam, and every Phase-0/Phase-1/controller hook) is compiled only when clusterer_controller is part of the build. The top-level Makefile detects that and passes -DCLUSTERER_CTRL_SUPPORT to the clusterer module; a small self-contained wiring in clusterer/Makefile acts on it.

If clusterer_controller is not built, the clusterer module is the stock upstream module — same behaviour, same exported parameters and commands. Verified with unifdef -UCLUSTERER_CTRL_SUPPORT against the base: the stock-preprocessed source has no semantic difference from upstream (the only textual residue is call sites using accessor macros that expand to the upstream globals, e.g. cluster_self_id(cl) → current_id). Concretely, in a stock build clusterer.so contains no cluster_options string and rejects the parameter as unknown; a native my_node_info/DB config loads unchanged. Enabling the controller rebuilds clusterer with the hooks — the two are always a matched pair.

New controller API (clusterer_ctrl)

A new clusterer_ctrl.c/.h pair exposes a load-time API (load_clusterer_ctrl_binds) through which an external module can drive the clusterer topology at runtime:

• set_my_identity(cluster_id, node_id, bin_url) — register this node's identity post-fork
• add_node / remove_node — add/remove peers at runtime
• update_identity — correct node_id after master assignment
• sync_current_id — sync process-local current_id after fork
• activate_backup_shtags / force_backup_shtags — sharing tag control
• set_shtag_managed — lock a cluster's tags against manual changes

cluster_options / use_controller mode and hybrid topologies

Each modparam("clusterer","cluster_options","cluster_id=N, use_controller=1") registers cluster N as a controller-driven stub at startup; its identity and peer list are filled in post-fork by the controller — no DB or static neighbor config for those. Any cluster without such a line is completely unaffected: DB loading, static neighbors, sharing-tag event routes and MI commands all work exactly as before.

use_controller is a per-cluster option carried in cluster_options (default 0). So one instance can run native clusters (DB-backed or static my_node_info/neighbor_node_info) and controller-managed clusters side by side. Each cluster is flagged controller_managed (a cluster_options line set use_controller=1) or native; a controller cluster always behaves as db_mode=0 even when native clusters are DB-backed (cl_db_mode(cl)), the two kinds share one cluster_id namespace, and declaring the same id both ways is rejected. my_node_id/db_url are required only when native clusters exist. This lets a fixed DB-provisioned replication cluster coexist with a zero-config controller HA cluster in one process — see Hybrid Topologies in the admin guide.

Config (devel): controller-managed clusters are declared per-cluster with modparam("clusterer","cluster_options","cluster_id=N, use_controller=1") — use_controller is a 0/1 field (default 0), so native clusters need no line. The controller-managed ids and the clusterer_controller cluster entries must match exactly, or startup aborts naming the offending id. (The interim use_controller/cluster_id int modparams were never released and now fail with a migration hint.)

Config examples:

Controller-only — two controller clusters sharing one BIN socket:

socket=bin:10.0.0.10:5566
loadmodule "clusterer.so"
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
modparam("clusterer", "cluster_options", "cluster_id=2, use_controller=1")
loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster", "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.0.10:5566")
modparam("clusterer_controller", "cluster", "id=2,multicast=239.0.90.1:3334,bin_socket=bin:10.0.0.10:5566")

Hybrid — a DB-native cluster 10 next to a controller cluster 1:

socket=bin:10.0.0.10:5566
loadmodule "db_mysql.so"
loadmodule "clusterer.so"
# native side: cluster 10 is defined by rows in the clusterer DB table (one per
# node, incl. a row with node_id = my_node_id for this node) - no cluster_id modparam
modparam("clusterer", "db_mode", 1)
modparam("clusterer", "db_url", "mysql://opensips:pass@localhost/opensips")
modparam("clusterer", "my_node_id", 5)          # this node's id in its native/DB clusters (global)
# controller side: cluster 1 is dynamic
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster", "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.0.10:5566")

modparam("dialog", "dialog_replication_cluster", 1)   # controller-managed (cluster 1)
modparam("dispatcher", "cluster_id", 10)              # DB-native (cluster 10)

Hybrid, no DB — a static native cluster 7 next to a controller cluster 1:

loadmodule "clusterer.so"
# native side: cluster 7 provisioned statically - db_mode=0 is REQUIRED,
# else my_node_info/neighbor_node_info are silently ignored
modparam("clusterer", "db_mode", 0)
modparam("clusterer", "my_node_id", 5)                     # this node's id in native cluster 7
modparam("clusterer", "my_node_info", "cluster_id=7, url=bin:10.0.0.10:5566")
modparam("clusterer", "neighbor_node_info", "cluster_id=7, node_id=6, url=bin:10.0.0.11:5566")
# controller side: cluster 1 is dynamic
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
loadmodule "clusterer_controller.so"
modparam("clusterer_controller", "cluster", "id=1,multicast=239.0.90.1:3333,bin_socket=bin:10.0.0.10:5566")

Here my_node_id=5 is this node's id in the native cluster 7; in cluster 1 the controller assigns an id at runtime, which may differ.

Per-cluster node identity (node_id / cluster_id)

Every clusterer packet already carries a cluster_id, and each cluster keeps its own membership. The change here is how a node's own identity is represented.

Before. A node's identity was a single global current_id (the my_node_id modparam). Classic clusterer works because an operator gives a node the same node_id in every cluster it belongs to — add_node_info() finds "me" in each cluster by matching node_id == current_id, and every runtime check (the "this message is from myself" source check, outgoing id stamps, topology and capability comparisons, cl_get_my_index) reads that one global.

The problem the controller introduces. Each controller cluster's master allocates node_ids independently (lowest free id in that cluster). So the same node can legitimately be node_id=1 in one cluster and node_id=2 in another — the ids diverge. A single global current_id cannot represent that: clusterer_ctrl_sync_current_id() collapsed it to the first cluster's value, so for every other cluster the node mis-identified itself. Concretely, a peer whose id equalled the stale global was treated as "me" (and vice-versa), and outgoing packets were stamped with the wrong source id — silently breaking replication for the divergent cluster. This also affected two controller clusters with different membership, not just hybrids.

The change. Identity is now per-cluster. The authoritative source is cl->current_node->node_id, which already lives in shared memory, exposed through a small accessor cluster_self_id(cl) (returns -1 — matches no node — before this cluster's identity is established). Specifically:

• add_node_info() takes an explicit self_id for the self/peer decision instead of reading the global; native load/provision pass my_node_id (unchanged behaviour), the controller and runtime-topology callers pass the per-cluster id.
• Every runtime read — the source/dest checks, id stamps, topology and capability comparisons, cl_get_my_index — uses cluster_self_id(cl).
• The three BIN receive paths resolve the cluster before the "bad source" self-check, so it compares against this node's id in that cluster.
• msg_add_trailer() stamps the source id per cluster.

The legacy global current_id / GET_CURRENT_ID remains only on the native DB/static provisioning path (where a node uses one id across all its clusters, by the operator's own configuration) and for the get_my_id() API. Result: within a hybrid or multi-controller-cluster instance, cluster_id selects the cluster and node_id is meaningful within that cluster, independently — validated live with a node holding node_id=3 in one cluster and node_id=2 in another, replicating cleanly in both.

---

Testing

Built against OpenSIPS 4.1.0-dev (clean make all; crypto suite
XChaCha20-Poly1305 + Argon2id via libsodium). Modules load against the fresh
core with no version-control mismatch and initialise cleanly. All scenarios
below were validated end-to-end on a live 3-node cluster (10.22.23.191–193)
with dialog replication and sharing-tag failover:

Area	Test	Result
Build & load	Clean make all (core + all non-excluded modules); modules load with no version-control mismatch; BIN socket discovered via socket_info_full; crypto initialised, no leaks/crashes	✅
Formation & roles	Cluster forms; master / backup (highest-IP non-master) / member consistent on all nodes; identical config bar the BIN socket IP	✅
Master failover (kill -9)	Backup promoted in ~3s using the preserved session key (no re-keying, no election loop); new backup designated; its shtags activated	✅
Rejoin	Master or member restart rejoins as member/backup; with master_stickiness=1 the current master stays put; no tag disruption	✅
master_stickiness=0	A higher-IP joiner takes over as master immediately (non-sticky path)	✅
Master-election stability	Staggered and simultaneous starts converge to a single stable master with only that master transmitting — no oscillation (a demoted node no longer keeps sending MASTER_ALIVE)	✅
Split-brain prevention & merge	Simultaneous cold start converges to one cluster (lower-IP nodes defer to the highest-IP starter); an induced divergent-key split is merged via MASTER_BEACON; ~5 JOIN_REQ total, 0 spurious "wrong password" warnings	✅
Native mode	no cluster_options use_controller=1 line, static my_node_info: clusterer mesh up, static shtag, no auto-promote on node loss (native semantics)	✅
Hybrid mode	Native cluster 10 + controller cluster 1 coexist; divergent per-cluster node_ids (191 nat1/ctrl2, 192 nat2/ctrl3, 193 nat3/ctrl1); independent shtag planes; same-id overlap rejected at startup	✅
Multiple clusters / shared BIN socket	Two controller clusters (id=1, id=2) on distinct multicast endpoints share one bin:IP:3857: both form independently, both BIN links Up, 0 decrypt/cross-talk errors (each BIN packet's cluster_id demuxes); two clusters on the same multicast are rejected at startup	✅
Wrong-password rejection	Mismatched node defers then shuts down (cannot authenticate … wrong password?) instead of forming a lone split-brain master; live cluster undisturbed (0 session-mismatch/demote); a corrected node rejoins	✅
Rogue-joiner tool (test/cc_join_reject_test.py)	From a non-member host: unauthenticated JOIN_REQs draw a JOIN_REJECT; a fake-MASTER_ALIVE flood is ignored (no re-JOIN churn)	✅
Sharing-tag override	cl_ctr_shtag_force pins the active tag to a non-master (all nodes report override:N), survives a master kill -9, auto-clears when the forced node departs; cl_ctr_shtag_auto restores; error paths rejected	✅
MI outputs	cl_ctr_list_members / node_info / list_config / shtag_force / shtag_auto correct; error paths non-master 409, bad node 404, bad cluster 404	✅
Buffer overrun/underrun	MI args huge/negative/non-numeric/empty/512-char → graceful 400/404/-32602; multicast parser fuzzed (115 cases: truncated < header, ciphertext < tag, oversized to 65 KB) + BIN-port fuzz → 0 crashes/cores	✅
Config-mismatch	on_config_mismatch=reject: divergent manage_shtags node rejected and self-shuts-down; reverting lets it rejoin	✅
libsodium suite	The join / failover / rejoin scenarios verified end-to-end with the XChaCha20-Poly1305 + Argon2id build	✅

Full failover walk-throughs are in modules/clusterer_controller/doc/clusterer_controller_tests.xml.

Note on MI access under 4.1

opensips-cli 0.3.3 cannot drive the 4.1 MI interface (it misparses the command
listing and reports no command '<x>' in module 'mi' even for core commands
like uptime). Until the CLI is updated, use the mi_http transport, e.g.:

curl -s http://<node-ip>:8888/os_mi_http \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"cl_ctr_list_config","params":[],"id":1}'

(the httpd listener binds the node IP, not loopback). This is a
tooling-version issue, not a module or core defect.

Limitations (alpha)

• IPv4 only (multicast address must be in 224.0.0.0/4)
• Rough clock agreement between nodes is recommended — the election window is quantized to query_time — but strict NTP is not required for correctness; replay protection is sequence-number based, not time-window based
• Multicast routing must be available on the network (LAN via IGMPv2/v3, or routed via PIM-SM or similar multicast routing protocol for multi-site deployments)
• The bootstrap key uses a memory-hard KDF (scrypt, or Argon2id in the libsodium build) over a fixed per-cluster salt (the password is a shared group secret, so the salt can't be per-node); a strong, high-entropy password is still recommended — the KDF raises the cost of an offline guess but doesn't rescue a weak password
• The crypto suite is a build-time choice: with libsodium the module is built sodium-only (XChaCha20-Poly1305/Argon2id, plus X25519/HKDF/RNG from libsodium — no WolfSSL linked), otherwise it falls back to WolfSSL (AES-256-GCM/scrypt). The two wire formats are not interoperable, so a cluster can't mix nodes built with different suites (the active suite is logged at startup to make mismatches visible)

---

Roadmap (not in this PR)

Planned follow-ups (also documented under Planned Features in the admin guide):

• Node maintenance mode — take a node out of duty for a rolling upgrade while it stays in the cluster (keeps replicating, answers pings, stays in cl_ctr_list_members) but is excluded from election and sheds its sharing tags (a cl_ctr_shtag_force pin auto-clears). Two levels: evicted (out of election/tags; the script refuses new work while dialogs finish) and full (additionally marks the node down for clusterer consumers). State is cluster-wide (carried in ALIVE/MEMBER_LIST, survives failover) and runtime-only. Interfaces: MI cl_ctr_maintenance (any node, master propagates), script function cl_ctr_set_maintenance(), read-only vars $cl_ctr_maintenance / $cl_ctr_node_maint(cid,nid) ($cl_ctr_role gains a maintenance value), and event E_CL_CTR_MAINTENANCE.

• Statistics — role, member count, master changes, joins/leaves, JOIN_REJECTs, decrypt failures, config mismatches, split-brain merges via get_statistics/monitoring.

• Events — E_CL_CTR_* raised on state transitions (became master, demoted, node joined/left, split-brain merged, config mismatch, auth reject) for event_route/subscribers.

• Per-node identity & stronger admission crypto — move beyond the single shared secret to per-node keypairs with enrolment and revocation, so one node can be distrusted without re-keying the whole cluster (closing the insider-trust limitation noted under Security). Candidate directions: a standard handshake (Noise) for admission, and a PAKE (CPace/SPAKE2) to remove offline password-guessing on captured bootstrap packets.

• IPv6 multicast — the control plane currently uses IPv4 multicast (groups in 224.0.0.0/4). Add IPv6 multicast support (ff00::/8 groups, AF_INET6 sockets and membership) so the controller can run on IPv6-only or dual-stack deployments.

---

[razvancrainea]

Thank you very much for the contribution, I really like the idea behind it. I do see though that it is marked as a Draft - is it still work in progress, or has it reached to its final state? Let us know when it is ready to review.

[Lt-Flash]

Hi,
Thanks a lot, I'm very glad you like the idea! I'm just finishing the latest touches in regards to variables and testing and then today I am planning to convert it to a proper PR!

[Lt-Flash]

Now it's ready for review, thanks!

[Lt-Flash]

Follow-up commit 4348580f07 — kept as a separate commit on purpose.

Since this PR is already open for review, I added this as a distinct follow-up commit rather than squashing it into the main one, so the incremental change is easy to review and the existing review isn't disrupted by a force-push of the main commit.

It enforces consistency between clusterer's global use_controller switch and whether clusterer_controller is loaded (admin-guide Dependencies section updated to match):

• clusterer_controller now refuses to start (mod_init fails) if the clusterer module has use_controller=0. That switch is what pre-creates the controller-managed cluster stubs, marks them controller_managed (so they never touch the DB), and arms the guard that stops the controller from hijacking a native cluster of the same id — with it off, the controller would run with those safety mechanisms disabled.
• The mirror case — use_controller=1 but clusterer_controller not loaded — logs an ERROR (the controller-managed stubs would otherwise never obtain an identity), but clusterer does not abort, since its native/hybrid clusters still work.

Hybrid environments are unaffected. use_controller is a single global switch — in a hybrid instance (native + controller-managed clusters side by side) it is always 1; only the per-cluster kind differs (native via DB/static vs. controller via the cluster_id list). So neither check ever trips a hybrid or pure-controller deployment; they only fire on a genuine module/config mismatch. Verified on all permutations: pure controller, hybrid, native-only, and both mismatches.

[Lt-Flash]

Config API for controller-managed clusters is now the per-cluster cluster_options modparam on the clusterer side:

modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
modparam("clusterer", "cluster_options", "cluster_id=2, use_controller=1")

Same key=value idiom as my_node_info. cluster_id is required; use_controller is a 0/1 field defaulting to 0 (native), and only use_controller=1 registers the controller-managed stub. Native clusters need no line, and every other clusterer setting (db_mode, ping_*, my_node_id, sharing_tag, …) stays a global modparam.

The controller-managed ids and the clusterer_controller cluster entries must match exactly — clusterer_controller aborts at startup, naming the offending id, if either side references a cluster the other doesn't (a managed id with no cluster config has no BIN socket or crypto params; a cluster config for an unmanaged id has nothing to drive).

Verified locally (the build links wolfSSL, so the controller runs without a node): the new syntax loads, a managed id with no controller config aborts, a controller config for an unmanaged id aborts, and matching config starts. Docs (admin guide, tests appendix, README) and the PR description are updated to this form.

[Lt-Flash]

Follow-up 18c199ef68: clusterer_controller is now opt-in, and the stock clusterer module is unchanged unless you build it.

Previously the clusterer-side integration (the clusterer_ctrl API, the cluster_options modparam, and the Phase-0/Phase-1 hooks) was always compiled into the clusterer module. That is now fully decoupled:

• clusterer_controller is excluded from the default build (added to exclude_modules in Makefile.conf.template), like the other modules with external-library dependencies. Enable it with include_modules= clusterer_controller.
• The top-level Makefile exports CLUSTERER_CTRL_SUPPORT=1 only when clusterer_controller is part of the build, and clusterer/Makefile turns that into -DCLUSTERER_CTRL_SUPPORT. Every clusterer-side controller hook is behind that flag.

So clusterer_controller can be completely omitted — a build without it produces the stock upstream clusterer module: no cluster_options parameter (it's rejected as unknown), no behavioural change, no added exports. The per-cluster identity / hybrid-db_mode accessors get #else fallbacks to the upstream globals (cluster_self_id(cl) → current_id, cl_db_mode(cl) → db_mode, etc.), so call sites compile to the exact upstream object code.

Verified with unifdef -UCLUSTERER_CTRL_SUPPORT diffed against the base branch: no semantic difference in the clusterer module. Built and checked both ways — stock (clusterer.so has zero cluster_options strings, native config loads, cluster_options rejected) and with the controller (cluster_options parses, the exact-match guard fires). Enabling the controller automatically rebuilds clusterer with the hooks; the two are a matched pair. New "Building the Module" section added to the admin guide/README.