fix(configs): K8S required_resources.memory must be bytes, not "8Gi" - #1053
Open
elliot-barn wants to merge 1 commit into
Open
fix(configs): K8S required_resources.memory must be bytes, not "8Gi"#1053elliot-barn wants to merge 1 commit into
elliot-barn wants to merge 1 commit into
Conversation
The three K8S configs used Kubernetes quantity strings for memory. That
form is accepted by `anyscale compute-config create -f` because the SDK
converts suffixes to bytes before building the API request, which is why
it passed local validation. The publish path never touches the SDK:
`rayapp build` copies required_resources verbatim into the bundle, and
the console clone path parses that bundle straight into the backend's
PhysicalResources, whose `memory` is an int.
Confirmed against the Azure staging API by replaying the built bundle
through the legacy create path:
422 Unprocessable Content from POST /cluster_computes/
body.config.head_node_type.required_resources.memory:
value is not a valid integer
body.config.worker_node_types.0.required_resources.memory:
value is not a valid integer
The same bundle with integer bytes is accepted, GPU shape included
(the accelerator-consistency check passes with required_labels intact).
Add a validator rule so the string form can't come back: nothing else in
the pipeline catches it, since the one path that does convert suffixes is
the path templates are tested on.
Signed-off-by: Elliot Barnwell <elliot.barnwell@anyscale.com>
Aydin-ab
approved these changes
Sep 1, 2026
Comment on lines
+1
to
+4
| # memory is in bytes, not a "32Gi" quantity string: the published bundle is | ||
| # parsed straight into the backend's PhysicalResources (`memory: int`), and | ||
| # only the SDK path converts suffixes — a string passes local `compute-config | ||
| # create -f` and 422s on launch. 8Gi = 8589934592, 32Gi = 34359738368. |
Contributor
There was a problem hiding this comment.
This looks like the same issue we had for compute configs few months ago, how did we solve it then ?
Can we maybe install the sdk and use it to convert the config inside of rayapp ?
The goal is to have these .yaml be the actual compute config we document in our docs
https://docs.anyscale.com/configuration/compute/declarative#examples
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #940 — targets
add-azure-declarative-compute-configso the diff is just this fix.Problem
The three K8S configs specify memory as Kubernetes quantity strings (
8Gi,16Gi,32Gi). That form is accepted byanyscale compute-config create -f, because the SDK converts suffixes to bytes before building the API request (PhysicalResources.to_dict(for_api=True)→_parse_memory_string). That's the path #940 was validated on, so it looked fine.The publish path never touches the SDK.
rayapp buildcopiesrequired_resourcesverbatim into the bundle, and the console clone path (workspaces_service._make_template_compute_config) parses that bundle straight into the backend'sPhysicalResources, whosememoryis anOptional[int]of bytes with no quantity parsing.Confirmed against the live Azure staging API by replaying the built bundle through the legacy create path:
This is latent today only because the launch path force-selects AWS on Kubernetes clouds — it would surface the moment the
K8Skey is honored, i.e. exactly when #940 becomes useful.Fix
Integer bytes in all three configs (
8Gi= 8589934592,16Gi= 17179869184,32Gi= 34359738368). Accepted by both paths — the SDK takesUnion[str, int].Plus a validator rule, because nothing else in the pipeline catches this: the one path that converts suffixes is the path templates are tested on, so a string passes local testing and fails only at launch.
Testing
pre-commit run --all-filesgreen; validator reportsOK: validated 56 BUILD.yaml entries.configs/getting-started/k8s.yaml: worker_nodes[0].required_resources.memory must be an integer number of bytes, not '32Gi'.required_labels— now creates successfully through the legacy path on Azure staging (cpt_6ji34wrn5c2iqf43v6h68rdvw2), so the backend'scheck_gpu_accelerator_consistencypasses too.compute_stack: K8S, westus2) andworkspace-introran its full notebook test there — 24 cells, exit 0.One caveat on that last line:
rayapp testcan't yet drive this itself on Kubernetes clouds. It reports success while executing nothing (ray-project/rayci#496 fixes the detection), and two further k8s gaps —run_commandstarting in$HOMErather than the workspace working directory, andbash -cnot sourcing the~/.bashrcwhere k8s workspaces declare PATH andANYSCALE_*— had to be worked around by hand to get the run above. None of them are caused by these configs.🤖 Generated with Claude Code