Skip to content

Send the Kubernetes discovery scope with each upload - #851

Merged
stewartshea merged 7 commits into
mainfrom
resources-design
Oct 30, 2026
Merged

stewartshea merged 7 commits into
mainfrom
resources-design

Conversation

@answersamir

Copy link
Copy Markdown
Contributor

What

The workspace builder now sends a small workspaceScope summary with every upload. It lists the Kubernetes clusters and namespaces the run covered, and the kubeconfig credential reference each cluster's tasks use. The platform can then configure Kubernetes resource discovery for those clusters itself. Users change nothing: the same workspaceInfo.yaml and the same upload.

Shape (version 1)

{
  "version": 1,
  "generatedAt": "2026-09-25T10:00:00Z",
  "builderVersion": "0.x.y",
  "kubernetes": {
    "clusters": [{
      "name": "prod-eu",                 // kubeconfig clusters[].name, same value SLX cluster tags use
      "context": "prod-eu-admin",
      "clusterType": "kubernetes",       // kubernetes | gke | aks | eks | ...
      "namespaceSelection": "all",       // "explicit" | "all"
      "namespaces": ["orders", "payments"],
      "excludedNamespaces": ["kube-system"],
      "credentialRef": "k8s:file@secret/kubeconfig:kubeconfig",
      "inClusterAuth": false,
      "reachable": true
    }]
  }
}
  • No secret material. The payload holds names and the credential reference only, never kubeconfig bytes, tokens, certificates or keys. credentialRef is rendered through the same kubernetes-auth.yaml template the SLX runbooks use, so the two can't drift. A test guards against secret material in the payload.
  • Clusters the builder couldn't reach are still listed, with reachable: false. This lets the platform tell "gone" from "unreachable this time".
  • Namespace lists are capped at 2,000 entries each (namespacesTruncated: true when cut).

How it travels

  • kubeapi.index() records a per-cluster ClusterScanResult. The server resets it at the start of each run.
  • The /run/ response carries workspaceScope. run writes it to workspace-scope.json in the output root. That is outside the workspaces/ tree, so the file is never part of the uploaded tarball or a Git-backed workspace repo.
  • upload (and run --upload) attaches it to POST /api/v3/workspaces/{ws}/upload as an optional field, best-effort. A platform that doesn't know the field ignores it, and a missing or unreadable file never fails an upload. A run that produced no scope removes the previous run's file.
  • In-cluster kubeconfigs now carry the same workspace-builder extension the cloud generators add (cluster_type: kubernetes, in_cluster_auth: true). kubernetes matches no auth, tag or hierarchy template branch, so rendered output is unchanged.

Tests

  • src/test_workspace_scope.py: payload shape, credential-ref rendering per cluster type, truncation, and the no-secrets guard.
  • src/indexers/test_kubeapi_workspace_scope.py: scan results for explicit/all selection, exclusions and unreachable clusters.
  • src/test_run_upload_workspace_scope.py: the upload body carries the scope, the tarball doesn't, and a stale file is removed. These tests sandbox $HOME.

Heads-up for reviewers

Some existing tests (test_gke_multi_project_discovery.py, and anything that calls run.main()) read or write the real ~/.kube. clear_stale_kubeconfig_artifacts deletes ~/.kube/config when run.main() runs outside a container. Run the suite with a sandboxed HOME until those are fixed.

answersamir and others added 6 commits September 25, 2026 00:44
The workspace builder already knows which clusters and namespaces a run
covered and which kubeconfig credential each cluster's tasks use. It now
sends that summary to the platform as an optional `workspaceScope` field
on the upload request, so the platform can configure resource discovery
without any extra setup. Platforms that don't know the field ignore it.

- kubeapi records per-cluster scan results (context, cluster type,
  explicit-vs-all namespace selection, in-scope and excluded namespaces,
  reachability), including clusters it could not reach.
- workspace_scope.build_workspace_scope renders the credential reference
  through the same kubernetes-auth.yaml template the SLX runbooks use, so
  the two cannot drift. No secret material is included.
- The run response carries the scope; `run` writes it to
  workspace-scope.json in the output root (outside the uploaded tarball)
  and `upload` attaches it best-effort. A run without a scope removes a
  previous run's file, and the long-lived builder server resets scan
  results per run.
- In-cluster kubeconfigs carry the same workspace-builder extension the
  cloud generators use, marking in-cluster auth.
…me cluster

A kubeconfig can have several contexts (e.g. different users/auth) pointing
at the same cluster. index() was keying ClusterScanResult by cluster name
but overwriting the whole entry for each context, so the workspaceScope
payload only reported the last context's included/excluded namespaces.

Now the ClusterScanResult for a cluster is reused across its contexts:
included/excluded namespaces are unioned, namespace_selection stays
"explicit" only if every context is, context/cluster_type/in_cluster_auth
come from the first context seen, and reachable is re-affirmed on success
so a later-succeeding context isn't left stuck at a prior context's
failure.
With contexts merged per cluster, a later context that fails (another
user or auth for the same server) no longer marks a cluster that an
earlier context already scanned as unreachable.
Resolve the generated resource catalog conflicts by regenerating them
from the merged code with the scripts/*/dump_*_resource_catalog.py
dumpers.
@answersamir
answersamir marked this pull request as ready for review October 30, 2026 12:25
@stewartshea
stewartshea merged commit 856f343 into main Oct 30, 2026
6 of 7 checks passed
@stewartshea
stewartshea deleted the resources-design branch October 30, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants