feat: llama.cpp RPC — --rpc client and an in-JVM RpcServer

[bernardladenthin]

Summary

• RPC client. A model can offload its layers to llama.cpp RPC servers on other machines.
  • Java: ModelParameters.setRpcServers(RpcEndpoint...).
  • CLI: --rpc host:port[,…], on both HTTP servers.
  • value.RpcEndpoint validates endpoints: IPv4 or host name. The transport has no IPv6, and upstream's parser would cut an IPv6 literal at its first colon.
• RPC server. RpcServer serves this machine's devices (every GPU found, else the CPU) and is the in-JVM counterpart of rpc-server.
  • Binds to loopback only; startOnNetwork is the explicit opt-in and logs a warning.
  • One server per process.
  • Also runnable as java -cp <jar> net.ladenthin.llama.RpcServer.
• Still one library, no new runtime dependency.
  • ggml-rpc is a static backend linked into jllama on every platform and in every classifier.
  • The transport is plain TCP. jllama.dll already imported WS2_32.dll through cpp-httplib.
  • GGML_RPC_RDMA is forced off. Upstream enables it whenever the build host has libibverbs/librdma, which would make the library unloadable on any machine without rdma-core.
• patches/0015 makes ggml-rpc safe to embed. Upstream GGML_ABORTs on every client-side connection problem, which in a JVM kills the application.
  • Registration fails softly instead: an unreachable, malformed or non-RPC endpoint becomes a LlamaException naming it.
  • A cached registration is re-checked.
  • A gone server's device reports 0/0 memory instead of aborting.
  • The server can be stopped (ggml_backend_rpc_stop_server) and reports whether it is listening (ggml_backend_rpc_server_listening).
  • Transport fixes: file-descriptor leaks and SIGPIPE.
• rpc_support.hpp runs before every parse (LlamaModel and NativeServer).
  • It registers the --rpc servers up front.
  • ggml never forgets a registered RPC device and puts RPC devices first by default. So when an earlier load in the same JVM registered one, a load that did not ask for it gets an explicit --device list without it. That list is chosen by llama.cpp's own default rules.
  • Without this, a load after a stopped server aborts the JVM. That was confirmed by a falsification run with the filter disabled.
• .github/verify-native-deps.py (in the package job) checks every shipped native library against the dependencies it is allowed to have. It reads ELF, PE (including Windows arm64) and Mach-O directly.
  • It found a pre-existing defect on its first run: the macOS dylib links Homebrew's openssl@3, so it does not load on a Mac without that formula. This is true for published 5.1.0 and for the current snapshot.
  • The two paths are allowlisted as a known defect so the check only flags new dependencies. The fix is in TODO.md because it needs a macOS CI run.
• Side effect. llama_supports_gpu_offload() is now true on CPU-only builds: the -ngl "no usable GPU" warning disappears and the load log shows "offloading" lines. Upstream's own release binaries (all built with RPC) behave the same, and nothing else reads it.

Test plan

• C++: test_rpc.cpp, 22 tests. ctest 581/581 on a fresh build directory, where the applier applied all nine patches, 0015 included. Covered:
  • the selection rules;
  • a real loopback server/client (mul_mat over RPC equals the local CPU result);
  • stop, including a still-connected client;
  • the unreachable, malformed and stale-server paths;
  • memory 0/0 for a gone server.
• Java unit tests: RpcEndpointTest, RpcServerOptionsTest, plus additions to ModelParametersExtendedTest, OpenAiServerCliTest and WireNameRegistryTest. PIT on value.*: 388/388.
• Java native, model-free: RpcServerTest (8 tests): lifecycle, restart on the same port, single instance, port taken, unreachable --rpc load. The JVM survives.
• Java model-backed: RpcIntegrationTest. It checks that the layers are on the RPC server (the load log shows model buffer size for the endpoint). It then loads a model without --rpc after the server stopped, at -lv 4, which queries every registered device's memory.
  • Run locally against a tiny generated GGUF, because HuggingFace is blocked in this sandbox.
  • With the device filter disabled, the second load aborts the JVM.
• Two-JVM fat-jar smoke .github/smoke-rpc-fatjar.sh, run locally on the default fat jar. It is wired into smoke-fatjar-linux and requires:
  • a chat completion over RPC;
  • an RPC model buffer in the log;
  • the server accepting a client;
  • a clean non-SIGABRT exit naming the endpoint for a server nobody runs.
• Full llama Java suite: 1817 tests, 0 failures. spotbugs:check, spotless, javadoc, clang-format 23.1.1 and REUSE all clean.
• CI is green on this branch. This is the first run on Windows (x86, x64, arm64), macOS, Android, s390x and the GPU classifiers.
• Docs / CHANGELOG updated where applicable: README "Distributed inference over RPC", CLAUDE.md (RPC section, patch row, test table, JNI entry-point count), CHANGELOG, TODO.

Related issues / PRs

Follows #460. The deferred follow-ups are in TODO.md "RPC backend — follow-ups":

• a server lost mid-inference still aborts the process (an upstream limitation);
• filing 0015 upstream;
• Android on a device;
• TLS;
• RDMA.

Checklist

• I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
• My commits follow Conventional Commits
• No security-sensitive changes. This adds a network server. It is loopback-only by default, remote binding is an explicit opt-in with a warning, and the README documents that the protocol has no authentication or encryption.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QSXAwbXqx9u5xsMArMvgT8

---

Generated by Claude Code

[bernardladenthin]

Two checks are red that are not caused by this PR:

• Verify GPG signing key (no secrets printed)
• Verify GPG signing key — Gradle/BouncyCastle path (no secrets printed)

Both jobs run in the maven-central environment. Its secrets are not available on a PR branch, so the jobs end within seconds without writing a log. They fail the same way on every PR run of this branch, including #459 and #460.

This PR changes neither job. No change on this branch can fix them, and a re-run would fail the same way. The checks that matter here are the build, test and smoke jobs, which run after the start gate.

---

Generated by Claude Code

[bernardladenthin]

claude-review failed too, but not because of this diff. The job ended 245 ms after the model started, with total_cost_usd: 0 and an empty modelUsage, and reported Claude result reported subtype success with is_error:true. The review never read any code, so the error came from the review action or its API access before any review began.

Nothing on this branch can fix it. A re-run of the job, or a later push, will start a fresh review.

---

Generated by Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
31 New issues
0 Accepted issues

Measures
0 Security Hotspots
89.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud