Skip to content

fix(configs): K8S required_resources.memory must be bytes, not "8Gi" - #1053

Open
elliot-barn wants to merge 1 commit into
add-azure-declarative-compute-configfrom
fix-k8s-required-resources-memory-bytes
Open

fix(configs): K8S required_resources.memory must be bytes, not "8Gi"#1053
elliot-barn wants to merge 1 commit into
add-azure-declarative-compute-configfrom
fix-k8s-required-resources-memory-bytes

Conversation

@elliot-barn

Copy link
Copy Markdown
Contributor

Stacked on #940 — targets add-azure-declarative-compute-config so the diff is just this fix.

Problem

The three K8S configs specify memory as Kubernetes quantity strings (8Gi, 16Gi, 32Gi). That form is accepted by anyscale compute-config create -f, because the SDK converts suffixes to bytes before building the API request (PhysicalResources.to_dict(for_api=True)_parse_memory_string). That's the path #940 was validated on, so it looked fine.

The publish path never touches the SDK. rayapp build copies required_resources verbatim into the bundle, and the console clone path (workspaces_service._make_template_compute_config) parses that bundle straight into the backend's PhysicalResources, whose memory is an Optional[int] of bytes with no quantity parsing.

Confirmed against the live Azure staging API by replaying the built bundle through the legacy create path:

422 Unprocessable Content from POST /cluster_computes/
  body.config.head_node_type.required_resources.memory: value is not a valid integer
  body.config.worker_node_types.0.required_resources.memory: value is not a valid integer

This is latent today only because the launch path force-selects AWS on Kubernetes clouds — it would surface the moment the K8S key is honored, i.e. exactly when #940 becomes useful.

Fix

Integer bytes in all three configs (8Gi = 8589934592, 16Gi = 17179869184, 32Gi = 34359738368). Accepted by both paths — the SDK takes Union[str, int].

Plus a validator rule, because nothing else in the pipeline catches this: the one path that converts suffixes is the path templates are tested on, so a string passes local testing and fails only at launch.

Testing

  • pre-commit run --all-files green; validator reports OK: validated 56 BUILD.yaml entries.
  • The new rule fires on a reverted value: configs/getting-started/k8s.yaml: worker_nodes[0].required_resources.memory must be an integer number of bytes, not '32Gi'.
  • All three bundles rebuilt; each carries integer memory.
  • The rebuilt pytorch-fsdp bundle — GPU plus required_labels — now creates successfully through the legacy path on Azure staging (cpt_6ji34wrn5c2iqf43v6h68rdvw2), so the backend's check_gpu_accelerator_consistency passes too.
  • End to end: a workspace launched on the byte-valued config on an AKS cloud (compute_stack: K8S, westus2) and workspace-intro ran its full notebook test there — 24 cells, exit 0.

One caveat on that last line: rayapp test can't yet drive this itself on Kubernetes clouds. It reports success while executing nothing (ray-project/rayci#496 fixes the detection), and two further k8s gaps — run_command starting in $HOME rather than the workspace working directory, and bash -c not sourcing the ~/.bashrc where k8s workspaces declare PATH and ANYSCALE_* — had to be worked around by hand to get the run above. None of them are caused by these configs.

🤖 Generated with Claude Code

The three K8S configs used Kubernetes quantity strings for memory. That
form is accepted by `anyscale compute-config create -f` because the SDK
converts suffixes to bytes before building the API request, which is why
it passed local validation. The publish path never touches the SDK:
`rayapp build` copies required_resources verbatim into the bundle, and
the console clone path parses that bundle straight into the backend's
PhysicalResources, whose `memory` is an int.

Confirmed against the Azure staging API by replaying the built bundle
through the legacy create path:

  422 Unprocessable Content from POST /cluster_computes/
    body.config.head_node_type.required_resources.memory:
      value is not a valid integer
    body.config.worker_node_types.0.required_resources.memory:
      value is not a valid integer

The same bundle with integer bytes is accepted, GPU shape included
(the accelerator-consistency check passes with required_labels intact).

Add a validator rule so the string form can't come back: nothing else in
the pipeline catches it, since the one path that does convert suffixes is
the path templates are tested on.

Signed-off-by: Elliot Barnwell <elliot.barnwell@anyscale.com>
Comment on lines +1 to +4
# memory is in bytes, not a "32Gi" quantity string: the published bundle is
# parsed straight into the backend's PhysicalResources (`memory: int`), and
# only the SDK path converts suffixes — a string passes local `compute-config
# create -f` and 422s on launch. 8Gi = 8589934592, 32Gi = 34359738368.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like the same issue we had for compute configs few months ago, how did we solve it then ?

Can we maybe install the sdk and use it to convert the config inside of rayapp ?

The goal is to have these .yaml be the actual compute config we document in our docs
https://docs.anyscale.com/configuration/compute/declarative#examples

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants