You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: let offloaded pods reach cluster endpoints over the ssh shadow - #577
With ShadowMode: ssh an offloaded pod can open connections back into the cluster, without WireGuard and without a network namespace. The site enables it with Network.SSH.Reverse.Enabled: true; the pod lists what it needs:
The shadow's ssh-reverse container opens a session to the compute node's own sshd through the login node, the same hop as the forward of #580, and asks it for one ssh -R per endpoint, each ending in a unix socket in the pod's socket directory. A relay started by the job holds a loopback address derived from the pod UID for the whole job and passes on only connections from the job's own user, and the job's /etc/hosts maps the Service names to it, so the workload uses the names and ports unchanged. If a site's node sshd refuses forwarding, Reverse.Sshd: userspace starts an sshd inside the job instead, and Reverse.NodeExec: srun carries the session through Slurm where ssh to the node is not possible, at lower throughput.
interlink.eu/reverse-socks-port adds a SOCKS5 proxy for undeclared endpoints only where the virtual kubelet sets Reverse.AllowSocks. It listens on 127.0.0.1 of the node, open to every user there, so a pod that asks for it gets a Warning event, the virtual kubelet and the shadow log a warning, and without AllowSocks the pod is refused. The virtual kubelet's Events now reach the API server.
Tested on three Slurm sites, with the node's sshd and, on one, with the user-space fallback: S3 and Redis by Service name, the Kubernetes API by FQDN with TLS verified, sockets srw------- in a drwx------ directory, SOCKS refused by default and working with its warnings when allowed. Unit tests cover the annotations, the rendered scripts in every mode (sh -n, bash -n) and the relay.
…terlink-hq#577
The branch carried early copies of interlink-hq#580, interlink-hq#583, interlink-hq#584 and interlink-hq#585 folded into
one commit. It now holds main as of interlink-hq#571 (interlink-hq#580, interlink-hq#581, interlink-hq#582, interlink-hq#583, interlink-hq#585 as
merged), interlink-hq#584 as reviewed and the rebased reverse tunnel, with the demo's
own commits re-applied: TenantNamespace, RefuseForeignPods, kept init
containers on confined nodes and their logs.
Fixes found while porting:
- interlink-hq#571 refreshed the pod IP through computeShadowResourceIdentity, which
ignores TenantNamespace; on a confined node it looked for the shadow in
<ns>-shadow and never moved the IP. It uses the provider's identity now.
- The completed kept-init wait saves the init containers' logs before it
takes the wait over, which cancels its context.
- RefuseForeignPods handles the errors of SetWatchErrorHandler and
AddEventHandler (errcheck).
- The node ConfigMap patch is a struct, so interlink-hq#584 and interlink-hq#577 together do not
trip goconst on "metadata".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
With ShadowMode ssh a pod can now reach cluster endpoints from the job,
without WireGuard and without a network namespace. The site enables it
with Network.SSH.Reverse.Enabled; the pod lists its endpoints in the
interlink.eu/reverse-forwards annotation.
The shadow gets an ssh-reverse container. Once the compute node is known
it starts a user-space sshd -i there through the login node (ssh <node>,
or srun --jobid --overlap with Reverse.NodeExec srun), as the job's user
and with the command's stdio as transport, and binds the forwards on it
with ssh -R on a loopback address derived from the pod UID. A snippet
prepended to slurm-job.vk.io/pre-exec maps the names in the container's
/etc/hosts and waits for the tunnel, so the workload uses the Service
name and port unchanged. Each forward ends in a unix socket only the
job's user can open, with a relay checking the connecting uid, unless
interlink.eu/reverse-owner-only is false. interlink.eu/reverse-socks-port
adds a SOCKS5 proxy for endpoints not declared, off unless asked for.
The annotations are parsed and validated in Go, and a pod asking for the
tunnel where it is not enabled is refused at creation rather than run
without it. shouldCreateShadow is also true for a pod with forwards and
no exposed port; the template then renders neither ssh-forward nor a
Service. Under Kerberos the login hop delegates the ticket for pods with
a reverse tunnel, because ssh or srun on the login node needs it.
Part of interlink-hq#548
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
The pod controller's event broadcaster had no sink, so every Event
recorded through it was dropped. It now records to the API server, and
the provider gets the same recorder, so the provider can also put
Events on the pods it runs. The helm chart's RBAC already lets the
virtual kubelet create Events.
Part of interlink-hq#548
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
The first version of the reverse tunnel left gaps a site review would
not accept. The shadow reached the node with an unchecked
`ssh <node>` from the login node, so the cluster endpoints went to
whatever host answered to that name. The owner-only relay ran as the
tunnel's session command, so on every reconnect its address was free
for another user of the node to take. Owner-only and the carrier could
be switched per pod, and any pod could open a SOCKS proxy that every
user of the node can use.
The forwards are now made by the compute node's own sshd
(Network.SSH.Reverse.Sshd "node", the default), reached through the
same checked node hop as ForwardTarget "node", so NodeHostKeys applies
to the reverse tunnel too. Each one is `-R` to rev-<port>.sock in the
pod's socket directory. The session holds only while that directory
exists, and it writes to its channel so that it also ends on the node
when the connection drops. The user-space sshd stays as Sshd
"userspace" for sites whose node sshd refuses forwarding: it keeps its
files in the socket directory, drops GatewayPorts and ForceCommand,
and is started through the same node hop or srun.
The relay moves out of the shadow and into the job. The pre-exec
writes it to the socket directory and starts it with python3 -I before
the workload; it binds every port first and fails the job if one is
taken, signals readiness with rev-relay.ready, and exits when the
job's shell or the directory goes. The job therefore holds its address
for its whole life and a tunnel outage refuses connections instead of
handing them to someone else. Since it now lives as long as the job,
an accept or thread error no longer ends a listener, and each kind of
failure is logged once and then every 1000th time: the log is the
job's output, which any user of the node could otherwise grow by
connecting in a loop. The /etc/hosts copy is a mktemp file in
node-local temporary space, because only the batch node has the relay.
Owner-only is no longer optional, and the carrier is the site's
decision: interlink.eu/reverse-owner-only and
interlink.eu/reverse-node-exec are gone (never released; pods setting
them are not refused). A SOCKS proxy needs Reverse.AllowSocks; a pod
asking for one without it is refused at creation. With it the proxy
binds 127.0.0.1 on the node, in a session of its own so that a port
another user takes while the tunnel is down cannot take the listed
forwards down too; the job waits until the listener is its own
user's, and the pod gets a VK warning, a Warning Event
ReverseSocksProxy and a WARNING line in the shadow log.
Kerberos delegation on the login hop is on only with Reverse.NodeExec
srun, where srun on the login node needs the ticket. The reverse
tunnel needs a plugin that sets SocketDirRoot; without it the job
fails at start with a message saying so.
Part of interlink-hq#548
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Adds opt-in SSH reverse tunnels so offloaded pods can reach cluster endpoints without WireGuard or a network namespace, extending the existing SSH shadow networking.
Changes:
Adds reverse-forward annotations, configuration validation, and workload setup.
Introduces an owner-only relay, node/userspace SSH forwarding, and Slurm carrier support.
Adds gated SOCKS access with warnings, Kubernetes Event delivery, and reverse-tunnel tests.
File
Description
pkg/virtualkubelet/virtualkubelet.go
Integrates reverse tunnels into shadow creation and pod validation.
- A pod with reverse annotations and interlink.eu/pod-vpn is refused at
creation: pod-vpn disables the shadow, so the job would start without
its forwards.
- The hosts file bind no longer drops the job's own binds under
Apptainer: APPTAINER_BIND starts from SINGULARITY_BIND when only the
legacy variable is set, since Apptainer reads it only while
APPTAINER_BIND is unset.
- An ssh shadow is not created for a pod that exposes only UDP ports
and asks for no reverse tunnel. ssh carries TCP only, and since the
forward container is skipped without a TCP port, the Deployment had
no container left with public-key auth.
- The relay also looks up the client in /proc/net/tcp6: a client on an
IPv6 socket, as a JVM opens by default, reaches the IPv4 listener
through a v4-mapped address and was refused.
- The srun carrier steps run on the batch host (squeue %B, srun -w),
where the relay and the socket directory are, rather than on any node
of a multi-node allocation.
- The session holding the forwards also ends when one of its sockets is
gone, as after a requeue on the same node recreated the socket
directory under the same path; the SOCKS session still holds on the
directory alone.
Part of interlink-hq#548
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
errcheck on the echo server's io.Copy, goconst on the forward and the
pod-vpn annotation (now test constants), and gosec G204 on the two
commands the tests build themselves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The readiness check accepts wildcard SOCKS listeners under GatewayPorts yes, exposing an unauthenticated proxy to other machines instead of enforcing node-local access.
Qualify bare Service destinations for the pod namespace
pkg/virtualkubelet/shadow_ssh_reverse.go:169
A bare Service name such as minio:9000 is accepted, but SSH resolves it in the shadow's namespace, which differs from the original pod's namespace by default. Connections can therefore fail or reach a different Service even though the tunnel reports ready. Qualify same-namespace Service destinations while preserving their workload aliases, or reject ambiguous bare names with an error requiring a namespace-qualified endpoint.
Check reserved IPv4 address after normalization
pkg/virtualkubelet/shadow_ssh_reverse.go:188
The reserved-address check compares the original string rather than the parsed IPv4 address. interlink.eu/reverse-bind: "::ffff:127.0.0.1" passes this check, but ip.String() then sets the relay bind address to the explicitly forbidden 127.0.0.1, reintroducing collisions with other jobs and node-local services. Check the normalized address instead.
- The SOCKS readiness check no longer accepts the job user's listener on
a wildcard address. A node sshd with GatewayPorts yes binds the -R
there whatever address it asks for, which would open the proxy to
other machines; the job now fails at start with a message naming
GatewayPorts. tcp6 is read too.
- A bare Service name in interlink.eu/reverse-forwards is dialled as
<name>.<pod namespace>. The shadow usually runs in <namespace>-shadow,
where the bare name resolved to another Service or to nothing. The
job keeps the name it asked for in /etc/hosts; localhost is left as is.
- interlink.eu/reverse-bind is checked after parsing, so
::ffff:127.0.0.1 is refused like 127.0.0.1 instead of becoming it.
Part of interlink-hq#548
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
The two other points from the last review are fixed too: a bare Service name in reverse-forwards is now dialled as name., and reverse-bind is checked after parsing, so ::ffff:127.0.0.1 is refused
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #548.
With
ShadowMode: sshan offloaded pod can open connections back into the cluster, without WireGuard and without a network namespace. The site enables it withNetwork.SSH.Reverse.Enabled: true; the pod lists what it needs:The shadow's
ssh-reversecontainer opens a session to the compute node's own sshd through the login node, the same hop as the forward of #580, and asks it for onessh -Rper endpoint, each ending in a unix socket in the pod's socket directory. A relay started by the job holds a loopback address derived from the pod UID for the whole job and passes on only connections from the job's own user, and the job's/etc/hostsmaps the Service names to it, so the workload uses the names and ports unchanged. If a site's node sshd refuses forwarding,Reverse.Sshd: userspacestarts an sshd inside the job instead, andReverse.NodeExec: sruncarries the session through Slurm where ssh to the node is not possible, at lower throughput.interlink.eu/reverse-socks-portadds a SOCKS5 proxy for undeclared endpoints only where the virtual kubelet setsReverse.AllowSocks. It listens on127.0.0.1of the node, open to every user there, so a pod that asks for it gets a Warning event, the virtual kubelet and the shadow log a warning, and withoutAllowSocksthe pod is refused. The virtual kubelet's Events now reach the API server.Tested on three Slurm sites, with the node's sshd and, on one, with the user-space fallback: S3 and Redis by Service name, the Kubernetes API by FQDN with TLS verified, sockets
srw-------in adrwx------directory, SOCKS refused by default and working with its warnings when allowed. Unit tests cover the annotations, the rendered scripts in every mode (sh -n,bash -n) and the relay.Related issue : #548