Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# IDE
.idea/
.vscode/
.codex/
*.swp

# Build output
Expand Down
5 changes: 5 additions & 0 deletions api/config/crd/bases/rlinf.io_jobs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,11 @@ spec:
properties:
kind:
type: string
pvcSizeGbMap:
additionalProperties:
format: int32
type: integer
type: object
pvcStorageMap:
additionalProperties:
type: string
Expand Down
7 changes: 7 additions & 0 deletions api/config/crd/bases/rlinf.io_nodes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,11 @@ spec:
等 Warning 事件及镜像拉取/调度相关事件)。控制面 Task reconciler 在 Task
处于 Pending 期间聚合各节点事件到 Task.status.events,供前端展示。
items:
description: |-
NodeEvent represents a Kubernetes Event observed on a node that is relevant
for surfacing to operators (e.g. DiskPressure warnings, FailedScheduling,
image pull failures). The node-agent collects Warning events plus a small
set of Normal scheduling/pulling events and writes them to Node.status.events.
properties:
count:
format: int32
Expand Down Expand Up @@ -127,6 +132,8 @@ spec:
type: string
pullProgress:
items:
description: PullProgress captures the progress of an in-flight
image pull on a node.
properties:
downloaded:
format: int64
Expand Down
5 changes: 5 additions & 0 deletions api/config/crd/bases/rlinf.io_tasks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,11 @@ spec:
properties:
kind:
type: string
pvcSizeGbMap:
additionalProperties:
format: int32
type: integer
type: object
pvcStorageMap:
additionalProperties:
type: string
Expand Down
5 changes: 5 additions & 0 deletions api/config/crd/bases/rlinf.io_workflows.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,11 @@ spec:
properties:
kind:
type: string
pvcSizeGbMap:
additionalProperties:
format: int32
type: integer
type: object
pvcStorageMap:
additionalProperties:
type: string
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions api/rlark.io/v1alpha1/task_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ type KubernetesWorkloadSpec struct {
Replicas *int32 `json:"replicas,omitempty"`
Template corev1.PodTemplateSpec `json:"template,omitempty"`
PvcStorageMap map[string]string `json:"pvcStorageMap,omitempty"`
PvcSizeGbMap map[string]int32 `json:"pvcSizeGbMap,omitempty"`
}

type DockerTaskSpec struct {
Expand Down
7 changes: 7 additions & 0 deletions api/rlark.io/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions apps/embodied-runtime/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,11 +240,11 @@ Where host device passthrough mounts existing `/dev/*` nodes, **host macvlan** c
```yaml
initContainers:
- name: devinit
image: rlinf/embodied-runtime:v0.1.0
command: ["devinit", "setup"] # reads RLINF_EMBODIED_DEVINIT_SOCKET_PATH
image: busybox:latest
command: ["/opt/rlinf/bin/devinit", "setup"] # reads RLINF_EMBODIED_DEVINIT_SOCKET_PATH
resources:
requests:
rlinf.io/device: 1 # triggers Allocate → RunDir mount + env vars
rlinf.io/device: 1 # triggers Allocate → RunDir mount + BinDir mount + env vars
limits: # required: LimitRanger/ResourceQuota rejects init containers without limits
rlinf.io/device: 1 # extended-resource limits must equal requests
```
Expand All @@ -271,7 +271,7 @@ The webhook has **automatic CA management**:
2. It reads the `MutatingWebhookConfiguration` (named via `--webhook-mutating-config`); when a webhook's `caBundle` is empty it patches in the CA certificate.
3. It signs a serving certificate with the CA and starts the HTTPS server.

When the `caBundle` is non-empty the webhook leaves it alone (assumed managed); a mismatch logs a warning. The init image defaults to the auto-discovered device-plugin image (downward API), which ships `devinit` — so it usually needs no configuration.
When the `caBundle` is non-empty the webhook leaves it alone (assumed managed); a mismatch logs a warning. The init image defaults to the auto-discovered device-plugin image (downward API), falling back to `busybox:latest`. The devinit binary is mounted from the host via BinDir, so the image does not need to contain it.

Device-plugin CLI flags:

Expand All @@ -283,7 +283,7 @@ Device-plugin CLI flags:
| `--webhook-mutating-config` | `MutatingWebhookConfiguration` name whose `caBundle` is auto-managed. |
| `--webhook-service-name` / `--webhook-service-namespace` | Service fronting the webhook (forms the serving cert DNS SAN). |
| `--webhook-ca-secret-name` / `--webhook-ca-secret-namespace` | Secret persisting the CA (empty = in-memory). |
| `--webhook-devinit-image` | Injected init container image (default: auto-discovered device-plugin image). |
| `--webhook-devinit-image` | Injected init container image (default: auto-discovered device-plugin image, then `busybox:latest`). The binary is mounted from the host, so the image does not need to contain devinit. |

See the [Helm chart](./charts/embodied-runtime) `webhook:` values for a turnkey deployment: it renders the webhook Service, the `MutatingWebhookConfiguration` (empty `caBundle`), the required RBAC (cluster-scoped `mutatingwebhookconfigurations` get/patch + namespaced `secrets`), and wires all the flags above into the device-plugin DaemonSet. Enable it with:

Expand All @@ -295,7 +295,7 @@ webhook:
caSecret: # persist the CA across restarts (recommended)
name: devinit-ca
namespace: rlark-system # defaults to the release namespace
# devinitImage: "" # defaults to .Values.devicePlugin.image
# devinitImage: "" # defaults to .Values.devicePlugin.image, then busybox:latest (binary is mounted from host)
```

The webhook only renders when **both** `webhook.enabled` and `config.hostMacvlans` are set; enabling it without macvlans is a no-op (the handler injects nothing).
Expand Down
12 changes: 6 additions & 6 deletions apps/embodied-runtime/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,11 +238,11 @@ host_devices:
```yaml
initContainers:
- name: devinit
image: rlinf/embodied-runtime:v0.1.0
command: ["devinit", "setup"] # 读取 RLINF_EMBODIED_DEVINIT_SOCKET_PATH
image: busybox:latest
command: ["/opt/rlinf/bin/devinit", "setup"] # 读取 RLINF_EMBODIED_DEVINIT_SOCKET_PATH
resources:
requests:
rlinf.io/device: 1 # 触发 Allocate → RunDir 挂载 + 环境变量
rlinf.io/device: 1 # 触发 Allocate → RunDir 挂载 + BinDir 挂载 + 环境变量
limits: # 必填:LimitRanger/ResourceQuota 会拒绝未设置 limits 的 init 容器
rlinf.io/device: 1 # 扩展资源的 limits 必须等于 requests
```
Expand All @@ -269,7 +269,7 @@ host_macvlans:
2. 读取 `MutatingWebhookConfiguration`(由 `--webhook-mutating-config` 指定);当某 webhook 的 `caBundle` 为空时,把 CA 证书 patch 进去。
3. 用该 CA 签发服务证书并启动 HTTPS 服务。

`caBundle` 非空时 webhook 不动它(视为已托管);不匹配时打印告警。init 镜像默认取自动发现的 device plugin 镜像(downward API),其中已包含 `devinit`,通常无需配置。
`caBundle` 非空时 webhook 不动它(视为已托管);不匹配时打印告警。init 镜像默认取自动发现的 device plugin 镜像(downward API),回退到 `busybox:latest`。devinit 二进制从宿主通过 BinDir 挂载,镜像无需包含它。

device-plugin CLI 参数:

Expand All @@ -281,7 +281,7 @@ device-plugin CLI 参数:
| `--webhook-mutating-config` | 待自动管理 `caBundle` 的 `MutatingWebhookConfiguration` 名称。 |
| `--webhook-service-name` / `--webhook-service-namespace` | 前置 webhook 的 Service(构成服务证书 DNS SAN)。 |
| `--webhook-ca-secret-name` / `--webhook-ca-secret-namespace` | 持久化 CA 的 Secret(留空 = 内存中生成)。 |
| `--webhook-devinit-image` | 注入的 init 容器镜像(默认:自动发现的 device plugin 镜像)。 |
| `--webhook-devinit-image` | 注入的 init 容器镜像(默认:自动发现的 device plugin 镜像,回退到 `busybox:latest`)。二进制从宿主挂载,镜像无需包含 devinit。 |

参见 [Helm chart](./charts/embodied-runtime) 的 `webhook:` 值,提供一键式部署:渲染 webhook Service、`MutatingWebhookConfiguration`(`caBundle` 留空)、所需 RBAC(集群级 `mutatingwebhookconfigurations` 的 get/patch + 命名空间级 `secrets`),并把上述参数全部接进 device-plugin DaemonSet。启用方式:

Expand All @@ -293,7 +293,7 @@ webhook:
caSecret: # 持久化 CA,跨重启复用(推荐)
name: devinit-ca
namespace: rlark-system # 默认取发布命名空间
# devinitImage: "" # 默认取 .Values.devicePlugin.image
# devinitImage: "" # 默认取 .Values.devicePlugin.image,回退到 busybox:latest(二进制从宿主挂载)
```

webhook 仅在 `webhook.enabled` 与 `config.hostMacvlans` **同时**设置时渲染;未配置 macvlan 时启用它无效(handler 不注入任何内容)。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ spec:
mountPath: /var/lib/kubelet/device-plugins
- name: socket-dir
mountPath: /var/run/rlark
- name: bin-dir
mountPath: /opt/rlinf/bin

securityContext:
privileged: true
Expand All @@ -118,3 +120,7 @@ spec:
hostPath:
path: /var/run/rlark
type: DirectoryOrCreate
- name: bin-dir
hostPath:
path: /opt/rlinf/bin
type: DirectoryOrCreate
9 changes: 5 additions & 4 deletions apps/embodied-runtime/charts/embodied-runtime/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,11 @@ webhook:
caSecret:
name: ""
namespace: ""
# devinitImage overrides the injected init container image. It must contain
# the devinit binary at /usr/local/bin/devinit. When empty, the device
# plugin auto-discovers its own image (.Values.devicePlugin.image, via the
# downward API) which ships devinit — so this is rarely needed.
# devinitImage overrides the injected init container image. The devinit
# binary is mounted from the host (/opt/rlinf/bin/devinit) via Allocate, so
# the image does not need to contain it. When empty, the device plugin
# auto-discovers its own image (.Values.devicePlugin.image, via the
# downward API) or falls back to "busybox:latest".
devinitImage: ""

# device-plugin configuration file. Keys are snake_case to match the binary's
Expand Down
2 changes: 1 addition & 1 deletion apps/embodied-runtime/cmd/device-plugin/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ defaults are used. See examples/device-plugin-config.yaml for a template.`,
cmd.Flags().StringVar(&wh.CASecretNamespace, "webhook-ca-secret-namespace", "",
"Namespace of the CA Secret (required when --webhook-ca-secret-name is set)")
cmd.Flags().StringVar(&wh.DevinitImage, "webhook-devinit-image", "",
"Image for the injected init container; must contain devinit at /usr/local/bin/devinit (default: auto-discovered device-plugin image)")
"Image for the injected init container; the binary is mounted from the host, so the image does not need to contain devinit (default: auto-discovered device-plugin image, then busybox:latest)")

if err := cmd.Execute(); err != nil {
log.Fatalf("[device-plugin] fatal: %v", err)
Expand Down
8 changes: 4 additions & 4 deletions apps/embodied-runtime/docs/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,16 +361,16 @@ ip route
curl -k https://172.16.0.2/ # reach the robot directly on its subnet
```

If you prefer not to use the webhook, author the init container yourself (it just requests the resource and runs `devinit setup`):
If you prefer not to use the webhook, author the init container yourself (it just requests the resource and runs `devinit setup`). The devinit binary is mounted from the host via BinDir, so the image does not need to contain it:

```yaml
initContainers:
- name: devinit
image: rlinf/embodied-runtime:v0.1.0
command: ["devinit", "setup"] # reads RLINF_EMBODIED_DEVINIT_SOCKET_PATH
image: busybox:latest
command: ["/opt/rlinf/bin/devinit", "setup"] # reads RLINF_EMBODIED_DEVINIT_SOCKET_PATH
resources:
requests:
rlinf.io/device: 1 # triggers Allocate → RunDir mount + env vars
rlinf.io/device: 1 # triggers Allocate → RunDir mount + BinDir mount + env vars
limits: # required: LimitRanger/ResourceQuota rejects init containers without limits
rlinf.io/device: 1 # extended-resource limits must equal requests
```
Expand Down
8 changes: 4 additions & 4 deletions apps/embodied-runtime/docs/examples.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,16 +361,16 @@ ip route
curl -k https://172.16.0.2/ # 直接在机器人子网上访问机器人
```

若不想用 webhook,可自行编写 init 容器(只需申请资源并运行 `devinit setup`):
若不想用 webhook,可自行编写 init 容器(只需申请资源并运行 `devinit setup`)。devinit 二进制从宿主通过 BinDir 挂载,镜像无需包含它:

```yaml
initContainers:
- name: devinit
image: rlinf/embodied-runtime:v0.1.0
command: ["devinit", "setup"] # 读取 RLINF_EMBODIED_DEVINIT_SOCKET_PATH
image: busybox:latest
command: ["/opt/rlinf/bin/devinit", "setup"] # 读取 RLINF_EMBODIED_DEVINIT_SOCKET_PATH
resources:
requests:
rlinf.io/device: 1 # 触发 Allocate → RunDir 挂载 + 环境变量
rlinf.io/device: 1 # 触发 Allocate → RunDir 挂载 + BinDir 挂载 + 环境变量
limits: # 必填:LimitRanger/ResourceQuota 会拒绝未设置 limits 的 init 容器
rlinf.io/device: 1 # 扩展资源的 limits 必须等于 requests
```
Expand Down
34 changes: 34 additions & 0 deletions apps/embodied-runtime/pkg/deviceplugin/plugin.go
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,35 @@ func envOrDefault(key, defaultVal string) string {
return defaultVal
}

// ensureDevinitBinary copies the devinit CLI binary from the device-plugin
// image (/usr/local/bin/devinit) to the host-mounted BinDir so it can be
// mounted into workload pods via Allocate. The init container then runs the
// binary from the mounted path without needing the device-plugin image.
// Idempotent: if the destination already exists it is silently skipped.
func ensureDevinitBinary() {
src := "/usr/local/bin/devinit"
dst := filepath.Join(BinDir, "devinit")

if _, err := os.Stat(dst); err == nil {
return
}

data, err := os.ReadFile(src)
if err != nil {
log.Printf("[device-plugin] WARNING: devinit binary not found at %s — init container will be skipped: %v", src, err)
return
}
if err := os.MkdirAll(BinDir, 0755); err != nil {
log.Printf("[device-plugin] WARNING: create %s: %v — init container will be skipped", BinDir, err)
return
}
if err := os.WriteFile(dst, data, 0755); err != nil {
log.Printf("[device-plugin] WARNING: write %s: %v — init container will be skipped", dst, err)
return
}
log.Printf("[device-plugin] devinit binary copied to %s", dst)
}

// PluginSocketPath returns the full path to the plugin's gRPC socket.
func PluginSocketPath() string {
return pluginapi.DevicePluginPath + PluginSocketName
Expand Down Expand Up @@ -412,6 +441,11 @@ func NewPlugin(cfg PluginConfig, whcfg WebhookConfig) *Plugin {
// Detect devices (after managers are started).
p.devices = p.detectDevices()

// Copy the devinit binary to the host-mounted BinDir so the webhook can
// inject an init container that uses the mounted binary (no image
// dependency). Best-effort: a failure is logged but does not block startup.
ensureDevinitBinary()

return p
}

Expand Down
Loading
Loading