reduce nvidiadriver reconciles on unrelated node label updates

[rahulait]

summary

This PR reduces unnecessary NVIDIADriver reconciles from unrelated GPU node label updates.

Previously, any label change on a GPU node passed the NVIDIADriver node watch predicate. That caused the controller to list all NVIDIADriver objects and enqueue all of them, even when the changed label could not affect driver ownership, daemonset rendering, scheduling, or status.

Now, the node watch predicate first checks whether the changed label is relevant to NVIDIADriver behavior. Unrelated GPU node label updates are ignored before listing NVIDIADrivers. Relevant label updates still list and enqueue all NVIDIADrivers so owner and non-owner driver status stays correct.

behavior change

GPU node label update	before	now
unrelated label update, for example team=platform	listed all NVIDIADrivers and enqueued all of them	ignored; no NVIDIADriver list and no NVIDIADriver reconcile
unrelated label update, for example environment=staging	listed all NVIDIADrivers and enqueued all of them	ignored; no NVIDIADriver list and no NVIDIADriver reconcile
unrelated label update, for example custom.company.com/maintenance=true	listed all NVIDIADrivers and enqueued all of them	ignored; no NVIDIADriver list and no NVIDIADriver reconcile
owner label update, nvidia.com/gpu-operator.driver.owner	listed all NVIDIADrivers and enqueued all of them	still lists all NVIDIADrivers and enqueues all of them
GPU presence label update, nvidia.com/gpu.present	listed all NVIDIADrivers and enqueued all of them	still lists all NVIDIADrivers and enqueues all of them
driver deploy label update, nvidia.com/gpu.deploy.driver	listed all NVIDIADrivers and enqueued all of them	still lists all NVIDIADrivers and enqueues all of them
vGPU host manager deploy label update, nvidia.com/gpu.deploy.vgpu-manager	listed all NVIDIADrivers and enqueued all of them	still lists all NVIDIADrivers and enqueues all of them
NFD OS/kernel label update	listed all NVIDIADrivers and enqueued all of them	still lists all NVIDIADrivers and enqueues all of them
label used by any NVIDIADriver.spec.nodeSelector, for example region=us-east-1	listed all NVIDIADrivers and enqueued all of them	still lists all NVIDIADrivers and enqueues all of them
selector conflict is created or resolved by a node label change	listed all NVIDIADrivers and enqueued all of them	still lists all NVIDIADrivers and enqueues all of them, so owner and non-owner statuses stay current

what counts as unrelated

A GPU node label update is unrelated when the changed label cannot affect NVIDIADriver ownership, daemonset rendering, scheduling, or status.

label change	unrelated when
team=platform	no NVIDIADriver uses team in spec.nodeSelector
environment=staging	no NVIDIADriver uses environment in spec.nodeSelector
topology.kubernetes.io/zone=us-east-1a	no NVIDIADriver uses topology.kubernetes.io/zone in spec.nodeSelector
custom.company.com/maintenance=true	no NVIDIADriver uses custom.company.com/maintenance in spec.nodeSelector
foo=bar	no NVIDIADriver uses foo in spec.nodeSelector

what counts as relevant

A GPU node label update is relevant when the changed label can affect NVIDIADriver ownership, daemonset rendering, scheduling, or status.

label change	why it is relevant
nvidia.com/gpu-operator.driver.owner	changes which NVIDIADriver owns the node
nvidia.com/gpu.present	affects whether the node is treated as a GPU node
nvidia.com/gpu.deploy.driver	affects normal driver daemonset scheduling
nvidia.com/gpu.deploy.vgpu-manager	affects vGPU host manager daemonset scheduling
NFD OS/kernel labels	affect rendered driver daemonset selectors
region=us-east-1	relevant if any NVIDIADriver uses region in spec.nodeSelector
team=platform	relevant if any NVIDIADriver uses team in spec.nodeSelector

why relevant updates still enqueue all NVIDIADrivers

Relevant label updates still enqueue all NVIDIADrivers intentionally.

A node label change can affect drivers that do not currently own the node. For example, driver-a may own a node, while driver-b has:

spec:
  nodeSelector:
    region: us-east-1

If the node gets region=us-east-1, driver-b must reconcile and report notReady because it now conflicts with driver-a. When that label is removed, driver-b must reconcile again and return to ready.

Because of that, this PR only suppresses unrelated label updates. It does not reduce fanout for relevant label updates.

how the selector-key cache helps

The controller needs to treat labels from NVIDIADriver.spec.nodeSelector as relevant, because changing one of those labels can change driver ownership or create/resolve selector conflicts.

Without a cache, the only way to know whether a changed node label is used by any NVIDIADriver selector is to list all NVIDIADriver objects on every GPU node label update.

This PR adds a small cache of node selector keys used by existing NVIDIADriver specs. The node watch predicate can check that cache before deciding whether the update is relevant.

node label update	without selector-key cache	with selector-key cache
team=platform, no NVIDIADriver selects team	list all NVIDIADrivers to find out whether team is relevant	check cache, see team is not used, ignore update
region=us-east-1, some NVIDIADriver selects region	list all NVIDIADrivers to find out whether region is relevant	check cache, see region is used, list/enqueue all NVIDIADrivers
built-in relevant label, like nvidia.com/gpu.deploy.driver	no selector lookup needed, but old code still listed/enqueued all NVIDIADrivers after the predicate passed	no selector lookup needed; label is statically relevant, so list/enqueue all NVIDIADrivers
unrelated label churn across many GPU nodes	each update listed all NVIDIADrivers and enqueued all of them	each update is rejected by the predicate without listing NVIDIADrivers

The cache does not reduce fanout for relevant label changes. Relevant updates still enqueue all NVIDIADrivers intentionally, because a node label change can affect drivers that do not currently own the node.

The cache helps by avoiding the expensive list/enqueue path for unrelated label changes.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing