in-cluster SSH deployment #545

Activity

1. dciangot

   added

   enhancementNew feature or requestNew feature or request

   on Jul 29, 2026
2. dciangot

   added the

   Feature

   issue type on Jul 29, 2026
3. dciangot

   changed the issue type from

   Feature

   to

   Task

   on Jul 29, 2026
4. dciangot

   added sub-issues on Jul 29, 2026

   • Update Helm chart to support in-cluster deployment via ssh #546
   • rework the cookbook in the doc to support only edge or in-cluster #547
   • Explore the ssh-only deployment with mesh #548
5. matbun commented on Jul 29, 2026

   

   matbun

   on Jul 29, 2026

   Collaborator

   More actions

   Opened a first pass at this. Nothing has been tested end-to-end yet — I wanted feedback on the shape before spending time on hardware. Happy to rework any of it.

   Four PRs, in merge order:

   	PR	Closes
   1	#550		VK: shadow rename, node propagation, SSH shadow
   2	interlink-hq/interlink-slurm-plugin#148		plugin reports the allocated node
   3	interlink-hq/interlink-helm-chart#56	#546	chart values
   4	#551	#547	docs

   PR 1 has to merge and be released before PR 2 — it uses a new field on PodStatus, so PR 2 does not compile until then and needs a go.mod bump afterwards (its CI will be red in the meantime). PR 3 renders standalone but is inert without PR 1's VK image. PR 4 is independent.

   The idea

   Instead of the offloaded pod dialing out to a public ingress (wstunnel), the shadow pod dials in to the site's login node and runs ssh -N -L <port>:<compute node>:<port>, one -L per exposed port:

   browser -> Ingress/Service -> shadow pod (ssh -L) -> login node -> compute node
   

   The only requirement is outbound SSH from the cluster to the login node. The offloaded pod runs nothing on its side — no wstunnel client, no WireGuard, no pre-exec injection at all.

   Selected explicitly, defaulting to today's behaviour:

   Network:
     EnableTunnel: true
     ShadowMode: ssh
     SSH:
       LoginHost: login.hpc.example.org
       User: alice
       KeySecret: hpc-ssh-key       # or Auth: kerberos + KeytabSecret + Principal

   Both auth methods sites actually use are supported: SSH public key (BSC) and Kerberos keytab (CERN, with kinit in an init container and a sidecar refreshing the ticket for the life of the tunnel).

   This replaces wstunnel, not mesh

   	direction	trigger	workload must run
   wstunnel	cluster -> pod	EnableTunnel + exposed ports	wstunnel client, dials out
   ssh	cluster -> pod	EnableTunnel + exposed ports	nothing
   mesh	bidirectional	FullMesh: true, every pod	mesh.sh

   Same trigger, same Deployment/Service/pod-IP contract, inverted dialing direction. It does not give the offloaded pod access back into the cluster, so object storage / message bus callbacks (what @antoinetran raised for LISA) are still mesh's job. ShadowMode: ssh + FullMesh: true is rejected at startup with a message pointing at #548 rather than half-working.

   #548 is deliberately out of scope here. I'd rather land the narrow, useful case first and treat mesh-over-SSH as its own design problem once we've confirmed it's even possible.

   How the shadow finds the compute node

   The shadow is rendered at CreatePod time, long before SLURM has scheduled anything, so nothing in the pod spec can carry the target.

   Rather than have the shadow read a file off the shared filesystem over SSH (what our deployments do today, and it races between tenants), plugins report it through the status API:

   PodStatus.NodeName string `json:"nodeName,omitempty"`

   The VK creates a per-shadow ConfigMap, empty, and patches the value in once a plugin reports one. Mounted rather than injected as env, because kubelet refreshes mounted ConfigMaps in place — changing the pod spec would restart the shadow and change its pod IP, which has already been reported as the offloaded pod's IP.

   This is the piece I'd most like a second opinion on. It's plugin-agnostic, which seemed worth the extra field: any plugin that can name the node it allocated gets tunneling, not just ones with a shared FS. But it does put a new field on a cross-repo contract.

   Waiting is the normal path, not an error: an init container polls the ConfigMap, so a pod whose job is queued sits in Init:0/1 rather than crash-looping a tunnel, bounded by a generous timeout (2h default).

   Two decisions worth arguing about

   wstunnel -> shadow rename (PR 1, its own commit). The per-pod Deployment/Service is a shadow pod; wstunnel is one thing that can run in it, and the SSH shadow runs none. So wstunnel-<pod>-<ns> becomes shadow-<pod>-<ns> and namespace <ns>-wstunnel becomes <ns>-shadow. Pod annotations and the Wstunnel* config keys are left alone on purpose — renaming those breaks existing manifests for no user-visible gain. It's still a soft break: anything hard-coding a shadow Service name (Ingress, VirtualService, admission policies) needs updating. Easy to drop if you'd rather not take it.

   Credential replication defaults to on. Shadows follow the offloaded pod, which with interlink.eu/shadow-same-ns means per-user Kubeflow namespaces where the SSH credential doesn't exist. The VK copies it there unless you set ReplicateCredentials: false. That makes the credential readable by anyone who can read Secrets in those namespaces, and the chart grants cluster-wide Secret write to match. Multi-tenant notebooks are unusable without it, but I can see an argument for defaulting it off and making people opt in.

   Docs (#547)

   PR 4 takes the cookbook ask literally: two deployment scenarios, edge and in-cluster. "Tunneled" was never a third place to deploy interLink — the API server is in the cluster either way, only the plugin moves — so it becomes a variant of in-cluster, alongside the plugin-in-the-pod setup. 3-tunneled.mdx is removed and intro.mdx updated to match. Most of that page was systemd boilerplate already duplicated verbatim in the systemd guide; what was genuinely tunnel-specific moved there.

   That page is also where the SSH shadow appears, as one of three ways to reach a service inside an offloaded pod, plus a new guide for the details.

   Heads up that the in-cluster page now says "SSH" about two unrelated things: the ssh-tunnel binary carrying the control plane, and the SSH shadow carrying traffic into a running pod. I've flagged the distinction inline, but if that naming is going to confuse people we could rename one of them before this lands.

   Testing status

   Unit tests only — config validation and every rejection path, the SSH template rendered through the same decoder the VK uses (asserting the decoded Deployment, both auth variants, UDP skipping, host-key pinning), node ConfigMap lifecycle, and credential replication including opt-out. Chart output for the existing test values is byte-identical to main, and the docs build passes with onBrokenLinks: throw.

   End-to-end against CERN HPC and MareNostrum5 needs the published ssh-tunnel sidecar image, so that comes after PR 1 merges and tags. I'll report back here with results.

   ---

   Unrelated to this issue, but found while adapting the above: interlink-hq/interlink-slurm-plugin#150 fixes a silent mesh bug where the workload runs outside the mesh network namespace. Independent of everything here.

   😄React with 😄1Reacted by James Bhattarai
6. dciangot

   closed this as completedin #550on Oct 1, 2026