From 73b020705d4cfec784b67f0d79cf079ff9da7d62 Mon Sep 17 00:00:00 2001 From: Pawan Date: Wed, 28 Oct 2026 16:12:46 +0530 Subject: [PATCH 1/2] docs: add DACS installation and operations troubleshooting guides Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 2 + docs/installation-troubleshooting.md | 87 ++++++++++++++++++++++++++++ docs/operations-troubleshooting.md | 49 ++++++++++++++++ docs/quickstart.md | 2 + docs/tsg/admission-webhook.md | 53 +++++++++++++++++ docs/tsg/cache-capacity.md | 65 +++++++++++++++++++++ docs/tsg/cache-connectivity.md | 66 +++++++++++++++++++++ docs/tsg/cache-creation.md | 59 +++++++++++++++++++ docs/tsg/cache-misses.md | 57 ++++++++++++++++++ docs/tsg/cache-scaling.md | 61 +++++++++++++++++++ docs/tsg/cacheserver-scheduling.md | 57 ++++++++++++++++++ docs/tsg/cacheserver-startup.md | 67 +++++++++++++++++++++ docs/tsg/checksum-failure.md | 60 +++++++++++++++++++ docs/tsg/client-writes.md | 56 ++++++++++++++++++ docs/tsg/cluster-access.md | 50 ++++++++++++++++ docs/tsg/controller-startup.md | 58 +++++++++++++++++++ docs/tsg/helm-installation.md | 62 ++++++++++++++++++++ docs/tsg/image-pull.md | 50 ++++++++++++++++ docs/tsg/missing-metrics.md | 72 +++++++++++++++++++++++ docs/tsg/node-recovery.md | 65 +++++++++++++++++++++ docs/tsg/nvme-setup.md | 74 +++++++++++++++++++++++ docs/tsg/slow-reads.md | 65 +++++++++++++++++++++ docs/tsg/storage-access.md | 59 +++++++++++++++++++ user-guide.md | 7 +++ 24 files changed, 1303 insertions(+) create mode 100644 docs/installation-troubleshooting.md create mode 100644 docs/operations-troubleshooting.md create mode 100644 docs/tsg/admission-webhook.md create mode 100644 docs/tsg/cache-capacity.md create mode 100644 docs/tsg/cache-connectivity.md create mode 100644 docs/tsg/cache-creation.md create mode 100644 docs/tsg/cache-misses.md create mode 100644 docs/tsg/cache-scaling.md create mode 100644 docs/tsg/cacheserver-scheduling.md create mode 100644 docs/tsg/cacheserver-startup.md create mode 100644 docs/tsg/checksum-failure.md create mode 100644 docs/tsg/client-writes.md create mode 100644 docs/tsg/cluster-access.md create mode 100644 docs/tsg/controller-startup.md create mode 100644 docs/tsg/helm-installation.md create mode 100644 docs/tsg/image-pull.md create mode 100644 docs/tsg/missing-metrics.md create mode 100644 docs/tsg/node-recovery.md create mode 100644 docs/tsg/nvme-setup.md create mode 100644 docs/tsg/slow-reads.md create mode 100644 docs/tsg/storage-access.md diff --git a/README.md b/README.md index 71a0ece4..098d46bd 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,8 @@ See [Architecture](docs/architecture.md) for the detailed request flow. | Understand the system architecture | [Architecture](docs/architecture.md) | | Understand the security model and trust boundary | [Security model](docs/security-model.md) | | Install and operate DACS | [Deployment user guide](user-guide.md) | +| Diagnose basic installation failures | [Installation troubleshooting guide](docs/installation-troubleshooting.md) | +| Diagnose common runtime issues | [Post-installation troubleshooting guides](docs/operations-troubleshooting.md) | | Build from a clean clone | [Build from source](docs/build-from-source.md) | | Understand Azure Pipelines build caching | [Azure Pipelines build and caching](docs/azure-pipeline-build-and-caching.md) | | Understand GitHub Actions build caching | [GitHub Actions build and caching](docs/github-actions-build-and-caching.md) | diff --git a/docs/installation-troubleshooting.md b/docs/installation-troubleshooting.md new file mode 100644 index 00000000..518c3145 --- /dev/null +++ b/docs/installation-troubleshooting.md @@ -0,0 +1,87 @@ +# DACS installation troubleshooting guides (TSGs) + +Use these guides when a DACS Helm installation fails or the first cache does not +become usable. Each issue has a separate guide with Symptoms, Description, +Steps to Mitigate, Verification Steps, and Contact Information. + +These guides cover the controller-based `dacs` chart used by the +[AKS quickstart](quickstart.md) and [deployment user guide](../user-guide.md). +The standalone `cache-server` chart has different values and resource names; +do not mix its `cacheServer.*` settings with the controller chart's `cache.*` +settings. + +For issues after installation, see the +[post-installation troubleshooting guides](operations-troubleshooting.md). + +## Installation issues + +| Symptom | Troubleshooting guide | +| --- | --- | +| Wrong cluster, `Unauthorized`, or `Forbidden` | [Cluster access or permissions failure](tsg/cluster-access.md) | +| Chart pull fails or Helm times out | [Helm installation failure](tsg/helm-installation.md) | +| `ErrImagePull` or `ImagePullBackOff` | [Container image pull failure](tsg/image-pull.md) | +| Controller is not Ready | [Controller startup failure](tsg/controller-startup.md) | +| Cache resource is missing or no servers appear | [Cache creation failure](tsg/cache-creation.md) | +| Cache creation fails with a webhook error | [Admission webhook failure](tsg/admission-webhook.md) | +| CacheServer pods remain `Pending` | [CacheServer scheduling failure](tsg/cacheserver-scheduling.md) | +| Prereq init container fails or storage is missing | [NVMe prerequisite setup failure](tsg/nvme-setup.md) | +| CacheServer restarts or stays unready | [CacheServer startup failure](tsg/cacheserver-startup.md) | +| Pods are Ready but the client cannot connect | [Cache discovery or client connectivity failure](tsg/cache-connectivity.md) | + +## Initial diagnostics + +Examples assume namespace `dacs-system`, Helm release `dacs`, Cache +`cache-sample`, and the default `namePrefix=dacs`. Substitute your actual values +and resource names. Replace angle-bracket placeholders before running commands. +Individual guides include their own variable setup so they can be used directly. + +```bash +export NAMESPACE="dacs-system" +export RELEASE="dacs" +export CACHE_NAME="cache-sample" + +kubectl config current-context +kubectl cluster-info +helm list -n "$NAMESPACE" --all +helm status "$RELEASE" -n "$NAMESPACE" +kubectl get deployments,daemonsets,statefulsets,pods,services -n "$NAMESPACE" -o wide +kubectl get cache -n "$NAMESPACE" +kubectl get events -n "$NAMESPACE" --sort-by=.metadata.creationTimestamp +``` + +If a resource does not exist yet, its `NotFound` error identifies the stage to +investigate. If `kubectl` cannot reach the cluster, fix cluster access first. + +## Safety precautions + +- Record events and logs before restarting pods or retrying Helm. +- Keep the original values file and chart version. Do not retry with defaults + that silently change storage mode, node selection, or sizing. +- Do not delete the namespace, Cache CRs, CRDs, or storage as a diagnostic step. + Replacing an ephemeral cache pod loses its cached data. +- NVMe setup can format unused disks. Confirm node and disk ownership before + installing or rerunning the prerequisite chart. Do not use `mkfs`, `wipefs`, + or force-create a RAID array as a troubleshooting shortcut. + +## Escalation checklist + +If the issue persists, collect: + +- Installation command, timestamp/time zone, chart version and `appVersion`, + and relevant values overrides. +- Cluster/Kubernetes version, node OS/architecture/VM size, namespace, release, + Cache name, and storage mode. +- Helm status/history, pod inventory, affected pod descriptions and events. +- Current and previous logs for the affected controller, CacheServer, or prereq + init container. +- Cache CR specification/status and, for storage failures, read-only host + diagnostics from the [NVMe setup guide](tsg/nvme-setup.md). +- Expected versus actual behavior and the first failing verification step. + +Review and redact diagnostics before sharing. Do not include credentials, +kubeconfig files, tokens, Kubernetes Secret contents, or TLS private keys. +Logs, values, CRs, and host output may also contain sensitive storage URLs and +private endpoints. + +Use the channels in [SUPPORT.md](../SUPPORT.md). Report suspected vulnerabilities +through [SECURITY.md](../SECURITY.md), not a public issue. diff --git a/docs/operations-troubleshooting.md b/docs/operations-troubleshooting.md new file mode 100644 index 00000000..a45e4d4e --- /dev/null +++ b/docs/operations-troubleshooting.md @@ -0,0 +1,49 @@ +# DACS post-installation troubleshooting guides (TSGs) + +Use these guides for basic issues after DACS has been installed. Each issue has +its own Symptoms, Description, Steps to Mitigate, Verification Steps, and +Contact Information sections. + +| Symptom | Troubleshooting guide | +| --- | --- | +| Reads become slow or throughput drops | [Slow reads or low throughput](tsg/slow-reads.md) | +| Repeated reads continue to miss the cache | [Repeated cache misses](tsg/cache-misses.md) | +| Cache writes fail or nodes report disk pressure | [Cache capacity or disk pressure](tsg/cache-capacity.md) | +| Replica count does not change as expected | [Unexpected cache scaling](tsg/cache-scaling.md) | +| Cache access degrades after a node restart or replacement | [Recovery after node disruption](tsg/node-recovery.md) | +| Remote Blob requests fail with authentication or access errors | [Remote storage access failure](tsg/storage-access.md) | +| Writes fail or data is absent from remote storage | [Client write failure or unexpected persistence](tsg/client-writes.md) | +| Dashboards or client metrics are missing | [Missing metrics](tsg/missing-metrics.md) | +| Reads report checksum mismatches | [Checksum validation failure](tsg/checksum-failure.md) | + +For initial setup failures, use the +[installation troubleshooting index](installation-troubleshooting.md). +Existing guides for [client connectivity](tsg/cache-connectivity.md), +[CacheServer startup](tsg/cacheserver-startup.md), and +[controller startup](tsg/controller-startup.md) also apply to runtime incidents. + +## Before making changes + +- Record the failure time, affected workloads, recent configuration changes, + node maintenance, and relevant logs/events before restarting anything. +- Examples use namespace `dacs-system`, Cache `cache-sample`, and + `namePrefix=dacs`. Substitute actual names and replace angle-bracket + placeholders. Each guide includes the variables it needs. +- Change Helm-managed settings through the original values file and pinned chart + version. Direct changes to generated StatefulSets may be reconciled away. +- Cache eviction, pod replacement, scaling, and node replacement can remove + cached data. Do not treat cache-only writes or pending asynchronous uploads as + durable Blob data. Confirm the authoritative copy before destructive actions. +- Do not delete Cache resources, CRDs, namespaces, or cache directories to + diagnose an issue. Do not disable checksum, TLS, or authorization checks. +- Redact credentials, tokens, SAS query strings, and sensitive paths from + diagnostics. Use the existing + [escalation checklist](installation-troubleshooting.md#escalation-checklist). + +## Verification + +Verify the affected application operation, not just pod status. Where +appropriate, also run the +[quickstart smoke test](quickstart.md#run-the-end-to-end-smoke-test). +It verifies cache upload/download and checksum validation; it does not verify +remote Blob authorization, application write durability, or production performance. diff --git a/docs/quickstart.md b/docs/quickstart.md index d6f9bdb5..19ec6e88 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -17,6 +17,8 @@ This configuration is for evaluation only. Cache data is lost when the CacheServer pod or release is removed. For NVMe/RAID storage, production sizing, scaling, authentication, troubleshooting, and uninstall details, see the [deployment user guide](../user-guide.md). +If installation or the smoke test fails, use the +[installation troubleshooting guide (TSG)](installation-troubleshooting.md). ## Set variables diff --git a/docs/tsg/admission-webhook.md b/docs/tsg/admission-webhook.md new file mode 100644 index 00000000..a299c0e4 --- /dev/null +++ b/docs/tsg/admission-webhook.md @@ -0,0 +1,53 @@ +# DACS Admission Webhook Failure + +## Symptoms + +Cache creation or updates fail with `failed calling webhook`, +`no endpoints available`, connection refused, a timeout, or an `x509` error. +Alternatively, the webhook rejects an invalid Cache field. + +## Description + +The DACS admission webhook defaults and validates Cache resources. The Kubernetes +API server must reach its Service and trust its serving certificate. By default, +the controller manages webhook certificates; cert-manager is not required for +the default installation. A field-validation rejection is not a connectivity +failure. + +## Steps to Mitigate + +1. Inspect the Service, endpoints, and controller logs. These names assume + `namePrefix=dacs`; substitute your actual names if different: + + ```bash + export NAMESPACE="dacs-system" + kubectl get service dacs-webhook-service -n "$NAMESPACE" + kubectl get endpointslices -n "$NAMESPACE" \ + -l kubernetes.io/service-name=dacs-webhook-service + kubectl logs deployment/dacs-manager -n "$NAMESPACE" -c manager --tail=200 + ``` + +2. For no ready endpoints, restore + [controller readiness](controller-startup.md). +3. For timeouts, verify API-server connectivity to the webhook Service, including + cluster firewall and network policies. The default Service exposes TCP `443` + and targets controller port `9443`. +4. For certificate errors, inspect certificate-management logs and confirm that + the webhook configuration references the intended Service and namespace. + Do not print TLS private keys or delete certificate Secrets blindly. +5. If optional integrations were enabled, verify their prerequisites separately. +6. For validation rejection, correct the Cache field named in the error. + Do not disable admission validation or change the failure policy merely to + bypass the failure. + +## Verification Steps + +Confirm the webhook Service has ready endpoints. Retry the original Cache +creation or update and confirm it is accepted without connection or certificate +errors. For invalid input, verify the corrected resource passes validation. + +## Contact Information + +Contact the cluster administrator for API-server connectivity or policy changes. +For DACS help, use [SUPPORT.md](../../SUPPORT.md) and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cache-capacity.md b/docs/tsg/cache-capacity.md new file mode 100644 index 00000000..e7e178c6 --- /dev/null +++ b/docs/tsg/cache-capacity.md @@ -0,0 +1,65 @@ +# DACS Cache Capacity or Disk Pressure + +## Symptoms + +Cache writes fail with `No space left on device`, nodes report `DiskPressure`, +pods are evicted, or frequent cache eviction causes repeated remote reads. + +## Description + +Configured cache capacity and physical filesystem free space are different +limits. Other node workloads, logs, or inode exhaustion can consume the backing +filesystem. DACS eviction controls cache retention; it does not clean unrelated +host data. Ephemeral storage also consumes node-local capacity. + +## Steps to Mitigate + +1. Inspect the Cache, affected pod, node, and eviction logs: + + ```bash + export NAMESPACE="dacs-system" + export CACHE_NAME="cache-sample" + kubectl describe cache "$CACHE_NAME" -n "$NAMESPACE" + kubectl describe pod -n "$NAMESPACE" + kubectl describe node + kubectl logs -n "$NAMESPACE" -c cacheserver --tail=300 + ``` + +2. With approved host access, check the backing filesystem and inodes using + read-only commands. Substitute the configured host path: + + ```bash + findmnt --mountpoint /var/lib/ssd/cacheserver + df -h /var/lib/ssd/cacheserver + df -i /var/lib/ssd/cacheserver + ``` + + For ephemeral mode, inspect the node filesystem backing the pod's `emptyDir` + through your cluster's normal node diagnostics. +3. If the expected NVMe mount is absent, use the [NVMe setup TSG](nvme-setup.md). + Do not assume an existing directory is backed by the data disk. +4. Review `cache.serverSizeInGB` and `cache.eviction` settings, including + `enableAccessTimeEviction`, `evictionMaintainFreeSpace`, `sizeThreshold`, + `sizeTarget`, and `passFrequency`. Keep the target below the threshold and + leave filesystem headroom. Inspect logs for invalid-setting warnings and + eviction progress. +5. If the working set exceeds safe capacity, add suitable capacity or revise + retention/sizing through the original values file. Have the administrator + address unrelated host disk use; do not delete arbitrary host/cache files. +6. Before reducing `cache.serverSizeInGB`, assess data impact: the startup + space-check script can clear the cache directory when existing contents exceed + the configured limit. Confirm authoritative remote copies and pending writes + before any size change, eviction-policy change, or restart. + +## Verification Steps + +Confirm filesystem space and inodes have headroom, node disk pressure clears, +and affected pods remain stable. Run a bounded application write/read and +observe capacity and eviction through at least the configured eviction interval. +Verify pressure does not recur immediately under the representative workload. + +## Contact Information + +Contact the cluster/storage administrator for host capacity. For DACS help, +use [SUPPORT.md](../../SUPPORT.md) and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cache-connectivity.md b/docs/tsg/cache-connectivity.md new file mode 100644 index 00000000..d56a5003 --- /dev/null +++ b/docs/tsg/cache-connectivity.md @@ -0,0 +1,66 @@ +# DACS Cache Discovery or Client Connectivity Failure + +## Symptoms + +CacheServer pods are Ready, but a client cannot discover servers or connect. +The first request may fail with DNS resolution errors, connection errors, +timeouts, or a failed quickstart smoke-test Job. + +## Description + +Clients use the cache discovery Service to locate CacheServers and then reach +cache endpoints. Successful discovery alone does not prove endpoint +connectivity. Service DNS normally resolves inside the cluster, not directly +from a workstation. + +## Steps to Mitigate + +1. Inspect pods, the discovery Service, endpoints, and network policies: + + ```bash + export NAMESPACE="dacs-system" + export CACHE_NAME="cache-sample" + kubectl get pods -n "$NAMESPACE" \ + -l "dacs.azure.com/cache-name=$CACHE_NAME" -o wide + kubectl get service "${CACHE_NAME}-discovery" -n "$NAMESPACE" + kubectl get endpointslices -n "$NAMESPACE" \ + -l "kubernetes.io/service-name=${CACHE_NAME}-discovery" + kubectl get networkpolicy -n "$NAMESPACE" + ``` + +2. For a missing Service, inspect + [Cache creation and reconciliation](cache-creation.md). +3. For no ready endpoints, restore + [CacheServer readiness](cacheserver-startup.md) and verify Service selectors. +4. Use the correct namespace in the client endpoint, for example + `cache-sample-discovery.dacs-system.svc.cluster.local`. Substitute the cluster + DNS suffix if it differs from `cluster.local`. +5. Test DNS from the affected workload using its available tools. Review policies + in both client and cache namespaces, DNS egress, and access to discovered + cache endpoints on the configured data port (default TCP `9065`). +6. For failed quickstart Jobs, inspect events and logs: + + ```bash + kubectl describe job dacs-smoke-upload -n "$NAMESPACE" + kubectl logs job/dacs-smoke-upload -n "$NAMESPACE" + kubectl describe job dacs-smoke-download -n "$NAMESPACE" + kubectl logs job/dacs-smoke-download -n "$NAMESPACE" + ``` + + These names apply to the quickstart; use your actual Job names otherwise. + Inspect the affected Job pod for startup failures. Use matching client images + and the same key prefix for upload and download. + +## Verification Steps + +Run the [quickstart smoke test](../quickstart.md#run-the-end-to-end-smoke-test). +Both Jobs must report Complete, their logs must contain +`All iterations succeeded: true`, and the checksum-validated download must +succeed. A deployed Helm release or a Running pod alone does not establish +end-to-end success. + +## Contact Information + +Contact the cluster/network administrator for DNS and policy issues. +For DACS help, use [SUPPORT.md](../../SUPPORT.md) and include redacted diagnostics +from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cache-creation.md b/docs/tsg/cache-creation.md new file mode 100644 index 00000000..95380cb3 --- /dev/null +++ b/docs/tsg/cache-creation.md @@ -0,0 +1,59 @@ +# DACS Cache Creation Failure + +## Symptoms + +The Cache resource is missing, `no matches for kind "Cache"` is reported, or +the Cache exists but no CacheServer StatefulSet or pods appear. + +## Description + +The `dacs` chart installs the Cache CRD and controller. A Cache custom resource +must then request cache infrastructure. The chart's `cache.enabled` value +defaults to `false`; installing the controller alone does not create a cache. + +## Steps to Mitigate + +1. Inspect the CRD, chart values, Cache status, and controller logs: + + ```bash + export NAMESPACE="dacs-system" + export RELEASE="dacs" + export CACHE_NAME="cache-sample" + kubectl get crd caches.storage.azure.com + kubectl get cache -n "$NAMESPACE" + helm get values "$RELEASE" -n "$NAMESPACE" --all + kubectl describe cache "$CACHE_NAME" -n "$NAMESPACE" + kubectl logs deployment/dacs-manager -n "$NAMESPACE" -c manager --tail=200 + ``` + + Substitute your actual names; `dacs-manager` assumes `namePrefix=dacs`. + A missing Cache or CRD may produce `NotFound` during these checks. +2. For a missing CRD, verify the `dacs` chart installed successfully with the + required cluster-scoped permissions. Do not delete/recreate an existing CRD + as a troubleshooting shortcut. +3. For a missing Cache, enable `cache.enabled` through the original Helm values + file or create a Cache CR using the [deployment guide](../../user-guide.md). +4. Check `cache.name` and namespace overrides. The controller watches the Helm + release namespace; keep the Cache CR in that namespace. +5. For an existing Cache without pods, inspect reconciliation errors and scaling + configuration. Confirm the desired count is nonzero and the node selector + matches eligible nodes. Restore + [controller readiness](controller-startup.md) if needed. +6. If admission rejects Cache creation, use the + [admission webhook TSG](admission-webhook.md). + +## Verification Steps + +```bash +kubectl get cache "$CACHE_NAME" -n "$NAMESPACE" +kubectl get statefulsets,pods -n "$NAMESPACE" \ + -l "dacs.azure.com/cache-name=$CACHE_NAME" +``` + +Confirm the Cache exists and the expected StatefulSet and pods appear. If pods +remain Pending, continue with the [scheduling TSG](cacheserver-scheduling.md). + +## Contact Information + +Use the channels in [SUPPORT.md](../../SUPPORT.md) and include redacted +diagnostics from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cache-misses.md b/docs/tsg/cache-misses.md new file mode 100644 index 00000000..85c05049 --- /dev/null +++ b/docs/tsg/cache-misses.md @@ -0,0 +1,57 @@ +# DACS Repeated Cache Misses + +## Symptoms + +Repeated reads of the same data continue fetching from remote storage, the +client reports a low hit rate, or a previously warm workload becomes cold. + +## Description + +Cache contents can change after eviction, expiry, server-set changes, pod/node +replacement, or source-version changes. Client settings can also disable caching +or direct requests to a different cache. A successful read may be served remotely +and does not prove a cache hit. + +## Steps to Mitigate + +1. Identify a small, unchanged object and capture client hit/miss metrics before + and after reading it twice. Allow pending asynchronous cache population to + complete before interpreting the second read. +2. Inspect cache placement, restarts, and configuration: + + ```bash + export NAMESPACE="dacs-system" + export CACHE_NAME="cache-sample" + kubectl get pods -n "$NAMESPACE" \ + -l "dacs.azure.com/cache-name=$CACHE_NAME" -o wide + kubectl describe cache "$CACHE_NAME" -n "$NAMESPACE" + kubectl logs -n "$NAMESPACE" -c cacheserver --tail=200 + ``` + +3. Review the affected client's configuration locally without printing secrets. + For StorageIntercept, verify `cacheEnable`, the intended server/discovery + endpoint, and the actual `cacheServerPort`. Do not assume the client's default + port matches the chart's default TCP `9065`. +4. Check for cache-upload errors and unexpected expiry/group settings such as + `cacheKeyExpirySeconds` and `cacheKeyGroupId`. Review + [StorageIntercept configuration](../../src/subsystems/storage/storage_intercept/README.md). +5. Compare the incident time with scaling, rolling updates, node replacement, + and eviction logs. For rapid churn or a working set larger than capacity, + use the [capacity TSG](cache-capacity.md). +6. Verify that object paths and versions are stable. Versioned keys intentionally + distinguish source versions; do not disable versioning or consistency checks + merely to improve apparent hit rate. +7. If discovery is stale or points to another cache, follow the + [connectivity TSG](cache-connectivity.md). + +## Verification Steps + +Read the same unchanged object repeatedly with the same client configuration +and a stable server set. Confirm successful cache population followed by cache +hits in client metrics, and verify returned data. Do not require a 100% hit rate +for an entire workload that includes new data, metadata, or version changes. + +## Contact Information + +Use [SUPPORT.md](../../SUPPORT.md) with redacted hit/miss observations and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cache-scaling.md b/docs/tsg/cache-scaling.md new file mode 100644 index 00000000..ee1098d7 --- /dev/null +++ b/docs/tsg/cache-scaling.md @@ -0,0 +1,61 @@ +# DACS Unexpected Cache Scaling + +## Symptoms + +The cache does not scale after node changes, replica counts differ from +expectations, new replicas stay Pending, or a dynamically scaled cache drops to +zero replicas. + +## Description + +DACS supports static counts and percentage-based dynamic scaling. Dynamic +scaling uses schedulable matching nodes, server bounds, tolerance, and +stabilization. Its `minServers` is an all-or-nothing minimum: a calculated count +below the minimum can become zero rather than being raised to the minimum. +This is not CPU-based autoscaling. + +## Steps to Mitigate + +1. Inspect desired configuration, scaling status, eligible nodes, and logs: + + ```bash + export NAMESPACE="dacs-system" + export CACHE_NAME="cache-sample" + kubectl get cache "$CACHE_NAME" -n "$NAMESPACE" -o yaml + kubectl get statefulset "$CACHE_NAME" -n "$NAMESPACE" + kubectl get nodes -L dacs.azure.com/processing-unit + kubectl logs deployment/dacs-manager -n "$NAMESPACE" -c manager --tail=200 + ``` + + Use your actual Deployment name if `namePrefix` differs from `dacs`. + Review CR output before sharing it. +2. Confirm the intended mode. Static mode uses `cache.scaling.static.numServers`. + If dynamic configuration is supplied through the chart, it takes precedence + over the chart's default static configuration. +3. For dynamic mode, count schedulable nodes matching the configured selector + and calculate `floor(nodeCount * percentageOfNodesToUse / 100)`. Apply + `maxServers` and the documented all-or-nothing `minServers` behavior. +4. Check controller tolerance and stabilization settings. Defaults are 10% + tolerance and 15-minute windows; node churn can keep resetting the window. + Small count changes may be intentionally suppressed. +5. Correct the intended mode, bounds, percentage, or selector through the + original Helm values. Do not repeatedly restart the controller or disable + stabilization just to force a change. +6. If desired replicas are correct but pods remain Pending, use the + [scheduling TSG](cacheserver-scheduling.md). Host-storage pods require separate + eligible nodes because of pod anti-affinity. + +See the [scaling guide](../../src/subsystems/cache-controller/docs/controller/scaling.md) +for the exact calculation and stabilization behavior. + +## Verification Steps + +Compare Cache scaling status and StatefulSet desired/ready replicas after the +applicable stabilization window with stable node eligibility. Confirm the count +matches the documented calculation and all requested pods become Ready. +Check client discovery and reads after a server-set change. + +## Contact Information + +Use [SUPPORT.md](../../SUPPORT.md) with scaling configuration, node-count history, +and the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cacheserver-scheduling.md b/docs/tsg/cacheserver-scheduling.md new file mode 100644 index 00000000..22f5b1ec --- /dev/null +++ b/docs/tsg/cacheserver-scheduling.md @@ -0,0 +1,57 @@ +# DACS CacheServer Scheduling Failure + +## Symptoms + +CacheServer pods remain `Pending` and are not assigned to suitable nodes. +Events or controller logs may report selector mismatches, insufficient +resources, taints, or placement failures. + +## Description + +CacheServer placement depends on node eligibility, available resources, and +storage configuration. NVMe deployments default to nodes labeled +`dacs.azure.com/processing-unit=cpu`; the ephemeral quickstart selects +`kubernetes.io/os=linux`. Controller-managed placement may not produce +default-scheduler diagnostics. + +## Steps to Mitigate + +1. Inspect the affected pod, candidate node, assigned scheduler, and controller + logs. Replace placeholders with actual names. The Deployment name assumes + `namePrefix=dacs`: + + ```bash + export NAMESPACE="dacs-system" + kubectl describe pod -n "$NAMESPACE" + kubectl get nodes -L dacs.azure.com/processing-unit,kubernetes.io/os,kubernetes.io/arch + kubectl get pod -n "$NAMESPACE" \ + -o jsonpath='{.spec.schedulerName}{"\n"}{.spec.nodeSelector}{"\n"}' + kubectl describe node + kubectl logs deployment/dacs-manager -n "$NAMESPACE" -c manager --tail=200 + ``` + +2. For selector mismatches, label only verified cache-capable worker nodes or + correct `cache.nodeSelectorKey` / `cache.nodeSelectorValue` through the + deployment configuration. +3. For insufficient CPU/memory, node pressure, or taints, add suitable capacity + or correct the deployment configuration. Do not remove protective taints + blindly. +4. Inspect controller logs as well as pod events for placement failures. +5. For host storage, confirm the prerequisite DaemonSet ran on the same eligible + nodes and the expected mount exists. Follow the [NVMe setup TSG](nvme-setup.md). + +## Verification Steps + +```bash +kubectl get pods -n "$NAMESPACE" -l dacs.azure.com/cache-name=cache-sample -o wide +``` + +Substitute your actual Cache name. Confirm every requested cache pod is assigned +to an eligible node and progresses to Running. If a scheduled pod fails startup, +continue with the [CacheServer startup TSG](cacheserver-startup.md). + +## Contact Information + +Contact the cluster administrator for node capacity or policy changes. +For DACS help, use [SUPPORT.md](../../SUPPORT.md) and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cacheserver-startup.md b/docs/tsg/cacheserver-startup.md new file mode 100644 index 00000000..faa9c936 --- /dev/null +++ b/docs/tsg/cacheserver-startup.md @@ -0,0 +1,67 @@ +# DACS CacheServer Startup Failure + +## Symptoms + +A scheduled CacheServer pod restarts, reports `CrashLoopBackOff` or `OOMKilled`, +or remains unready. Logs may show mount/permission errors or +`No space left on device`. + +## Description + +CacheServers need valid configuration, accessible cache storage, and sufficient +resources. A Running pod is not necessarily Ready. `cache.serverSizeInGB` +configures cache capacity; it is not the container memory limit. + +## Steps to Mitigate + +1. Capture events and logs before replacing the pod. Replace `` with + the affected pod: + + ```bash + export NAMESPACE="dacs-system" + export CACHE_NAME="cache-sample" + kubectl get pods -n "$NAMESPACE" \ + -l "dacs.azure.com/cache-name=$CACHE_NAME" -o wide + kubectl describe pod -n "$NAMESPACE" + kubectl logs -n "$NAMESPACE" --all-containers --tail=200 + ``` + +2. For restarted containers, also capture previous logs: + + ```bash + kubectl logs -n "$NAMESPACE" --all-containers --previous --tail=200 + ``` + + If no previous container exists, that error is not an additional failure. +3. For `OOMKilled`, review `cache.memoryLimitMB`, thread count, and workload needs. + Update the original deployment values if sizing changes are required. +4. For mount/permission errors, verify the configured host path, mount, and + expected ownership. Follow the [NVMe setup TSG](nvme-setup.md) for host + storage; do not apply broad permissions changes blindly. +5. For `No space left on device` or `DiskPressure`, inspect the backing filesystem + and node ephemeral storage. Reduce cache sizing or add capacity; do not delete + unrelated host data. +6. For probe failures or process exits, use the actual startup error to check + ports, configuration, and chart/image compatibility before restarting. + +Replacing an ephemeral cache pod loses its cached data. Do not delete the Cache, +namespace, or storage as a diagnostic step. + +## Verification Steps + +```bash +kubectl wait -n "$NAMESPACE" --for=condition=Ready pod \ + --selector="dacs.azure.com/cache-name=$CACHE_NAME" --timeout=5m +kubectl get pods -n "$NAMESPACE" \ + -l "dacs.azure.com/cache-name=$CACHE_NAME" +``` + +Confirm all expected CacheServer pods are present and Ready and restart counts +stop increasing. Run the +[quickstart smoke test](../quickstart.md#run-the-end-to-end-smoke-test) to verify +upload and checksum-validated download. + +## Contact Information + +Use the channels in [SUPPORT.md](../../SUPPORT.md) and include redacted +diagnostics from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/checksum-failure.md b/docs/tsg/checksum-failure.md new file mode 100644 index 00000000..7cdf6ea3 --- /dev/null +++ b/docs/tsg/checksum-failure.md @@ -0,0 +1,60 @@ +# DACS Checksum Validation Failure + +## Symptoms + +Reads fail with checksum errors or `CHECKSUM_MISMATCH`, or the checksum-enabled +smoke-test download fails. + +## Description + +Checksum validation can detect mismatched data or missing checksum metadata +under a strict policy. Cache checksums and optional remote Blob transfer +checksums are separate mechanisms. A mismatch must be investigated; turning +validation off does not establish that the returned data is correct. + +## Steps to Mitigate + +1. Stop using suspect output for downstream processing. Record the affected + object/version or cache key, request range, client/server versions, and time. + Preserve logs and avoid broad cache deletion. +2. Collect affected client and server logs with sensitive paths redacted: + + ```bash + export NAMESPACE="dacs-system" + export CLIENT_NAMESPACE="application-namespace" + kubectl logs -n "$NAMESPACE" -c cacheserver --tail=200 + kubectl logs -n "$CLIENT_NAMESPACE" -c --tail=200 + ``` + +3. Review the actual client configuration. StorageIntercept settings include + `cacheCalculateChecksumEnabled`, `cacheValidateChecksumEnabled`, + `cacheChecksumAlgorithm`, and `cacheChecksumMetadataPolicy`. A `required` + metadata policy reports missing metadata as a mismatch; distinguish that from + a checksum value mismatch. Align writers/readers with the intended policy, + rather than weakening checks to read older entries. +4. Check for source changes or concurrent writes. Reproduce using an unchanged + test object and a unique key prefix so a different source version is not + mistaken for the same expected content. +5. Independently validate the authoritative remote object's contents through an + approved storage client. If the source is correct and a cached entry is + suspect, escalate for scoped remediation/repopulation; do not erase an entire + cache that may contain cache-only or pending writes. +6. For remote SDK downloads, review `remoteBlobChecksumEnabled` and + `remoteBlobChecksumRetryMaxAttempts`. Persistent errors after bounded retries + need investigation, not unlimited retries or disabled validation. See + [checksum configuration](../../src/subsystems/storage/storage_intercept/README.md). + +## Verification Steps + +Repeat the affected read with validation enabled and compare it with the +authoritative expected content. Run the +[checksum-enabled smoke test](../quickstart.md#run-the-end-to-end-smoke-test) +with a new prefix; both Jobs must complete with +`All iterations succeeded: true`. A passing isolated smoke test does not by +itself resolve a mismatch for the original object. + +## Contact Information + +Use [SUPPORT.md](../../SUPPORT.md) with sanitized mismatch details and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). +For suspected vulnerabilities, follow [SECURITY.md](../../SECURITY.md) privately. diff --git a/docs/tsg/client-writes.md b/docs/tsg/client-writes.md new file mode 100644 index 00000000..db81773f --- /dev/null +++ b/docs/tsg/client-writes.md @@ -0,0 +1,56 @@ +# DACS Client Write Failure or Unexpected Persistence + +## Symptoms + +Writes fail as read-only/unsupported operations, existing-object writes fail, +or an apparently successful write is not yet visible in remote Blob storage. + +## Description + +Write semantics depend on the API and selected mode. Core `DACSCacheClient` +disables writes when no mode is supplied; adapters can have different defaults. +`WRITE_CACHE_ONLY` does not persist data to Blob, and +`WRITE_CACHE_BLOB_ASYNC` uploads remotely in the background. API support for +append/overwrite also varies. Cache success alone is not proof of remote +durability. + +## Steps to Mitigate + +1. Identify the API, client version, selected write mode, object path, and exact + exception. Inspect affected workload logs without exposing credentials: + + ```bash + export CLIENT_NAMESPACE="application-namespace" + kubectl logs -n "$CLIENT_NAMESPACE" -c --tail=200 + ``` + +2. Compare the operation with the + [client API and write-mode documentation](../../src/subsystems/storage/dacs_client/README.md). + Do not assume every adapter supports append or overwriting existing objects. + Reproduce with a uniquely named object rather than deleting existing data. +3. For durable remote writes, explicitly choose a mode appropriate to the + application's contract. `WRITE_SYNC` writes cache and Blob synchronously; + `WRITE_BLOB_ONLY` bypasses cache. Do not switch production semantics without + assessing the application impact. +4. For async mode, inspect background upload errors and use documented + close/shutdown handling. Do not terminate the client or remove cache data + while uploads may be pending. Independently confirm remote persistence. +5. For StorageIntercept configuration, review `enableWrite`, `enableWriteRemote`, + and `enableWriteRemoteAsync`. Consistent mode with writes requires versioned + keys, remote writes enabled, and asynchronous remote writes disabled; see its + [configuration guide](../../src/subsystems/storage/storage_intercept/README.md). +6. Follow the [storage access TSG](storage-access.md) for remote authorization + errors and [capacity TSG](cache-capacity.md) for cache-space failures. + +## Verification Steps + +Write a small uniquely named object using the intended API/mode, then close it +through the documented lifecycle. Verify its length and contents from the +authoritative remote store using an approved client, not only a cached read. +For intentional cache-only mode, verify cache access and explicitly document +that remote persistence is not expected. + +## Contact Information + +Use [SUPPORT.md](../../SUPPORT.md) with the API, write mode, sanitized exception, +and [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/cluster-access.md b/docs/tsg/cluster-access.md new file mode 100644 index 00000000..51002a5f --- /dev/null +++ b/docs/tsg/cluster-access.md @@ -0,0 +1,50 @@ +# DACS Cluster Access or Permissions Failure + +## Symptoms + +Installation commands target the wrong cluster or fail with connection timeouts, +`Unauthorized`, or `Forbidden`. + +## Description + +Helm and `kubectl` require connectivity and credentials for the intended +Kubernetes cluster. The DACS chart also installs cluster-scoped CRDs, RBAC, and +webhook configurations, so namespace-only permissions may not be sufficient. + +## Steps to Mitigate + +1. Set the intended namespace and inspect the active context: + + ```bash + export NAMESPACE="dacs-system" + kubectl config current-context + kubectl cluster-info + kubectl get nodes + ``` + +2. Select the intended kubeconfig context. For AKS, refresh credentials for the + intended subscription, resource group, and cluster through your normal Azure + CLI access workflow. +3. For connection failures, check VPN/private-cluster connectivity and Kubernetes + API DNS resolution. +4. For authorization failures, inspect the denied operation and ask the cluster + administrator to grant the required permissions. These checks are examples, + not an exhaustive authorization check: + + ```bash + kubectl auth can-i create deployments -n "$NAMESPACE" + kubectl auth can-i create customresourcedefinitions.apiextensions.k8s.io + kubectl auth can-i create clusterroles.rbac.authorization.k8s.io + ``` + +## Verification Steps + +Confirm that the active context is the intended cluster, cluster queries +succeed, and the installer is authorized for the previously denied operation. +Retry the original installation command with its original version and values. + +## Contact Information + +Contact the cluster administrator for connectivity or permission changes. +For DACS help, use the channels in [SUPPORT.md](../../SUPPORT.md) and include +redacted diagnostics from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/controller-startup.md b/docs/tsg/controller-startup.md new file mode 100644 index 00000000..962d362f --- /dev/null +++ b/docs/tsg/controller-startup.md @@ -0,0 +1,58 @@ +# DACS Controller Startup Failure + +## Symptoms + +The controller Deployment does not become Available. Its pod remains Pending, +restarts, reports `CrashLoopBackOff` or `OOMKilled`, or fails readiness probes. + +## Description + +The controller reconciles Cache resources and manages cache infrastructure. +Scheduling, resource limits, RBAC, invalid configuration, or certificate startup +errors can prevent it from becoming Ready. + +## Steps to Mitigate + +1. Inspect the Deployment, pod events, and logs. These examples assume + `namePrefix=dacs`; substitute your actual Deployment name if different. + Replace `` with the pod returned by the query: + + ```bash + export NAMESPACE="dacs-system" + kubectl describe deployment dacs-manager -n "$NAMESPACE" + kubectl get pods -n "$NAMESPACE" -l control-plane=controller-manager + kubectl logs deployment/dacs-manager -n "$NAMESPACE" -c manager --tail=200 + kubectl describe pod -n "$NAMESPACE" + ``` + +2. If the controller container restarted, capture its previous logs before + replacing the pod: + + ```bash + kubectl logs -n "$NAMESPACE" -c manager --previous --tail=200 + ``` + +3. For Pending pods, inspect affinity, selectors, taints, and available resources. + Defaults target Linux `amd64`/`arm64` nodes. +4. For `OOMKilled`, review memory usage and the manager limit; adjust + `manager.resources` through your Helm values if needed. +5. For `Forbidden`, review the denied operation and chart-installed + ServiceAccount, RoleBindings, and ClusterRoleBindings with the administrator. +6. For certificate or webhook errors, follow the + [admission webhook TSG](admission-webhook.md). For invalid flags, correct + image/version mismatches or custom `manager.extraArgs` / environment overrides. + +## Verification Steps + +```bash +kubectl rollout status deployment/dacs-manager -n "$NAMESPACE" --timeout=5m +kubectl get pods -n "$NAMESPACE" -l control-plane=controller-manager +``` + +Confirm the Deployment is Available, its expected replicas are Ready, and restart +counts do not continue increasing. + +## Contact Information + +Use the channels in [SUPPORT.md](../../SUPPORT.md) and include redacted +diagnostics from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/helm-installation.md b/docs/tsg/helm-installation.md new file mode 100644 index 00000000..a6d19109 --- /dev/null +++ b/docs/tsg/helm-installation.md @@ -0,0 +1,62 @@ +# DACS Helm Installation Failure + +## Symptoms + +Helm reports OCI download errors, chart version not found, release ownership +conflicts, `context deadline exceeded`, or `timed out waiting for the condition`. + +## Description + +Helm must retrieve the requested chart and create its Kubernetes resources. +Download, authorization, ownership, or workload-readiness failures can prevent +installation. Chart `version` and image `appVersion` are different identifiers. +Helm readiness does not establish readiness of controller-created CacheServers. + +## Steps to Mitigate + +1. Inspect the chart and release. Replace `` with the pinned + version used for your installation: + + ```bash + export NAMESPACE="dacs-system" + export RELEASE="dacs" + export CHART_REF="oci://dacstest.azurecr.io/dacs/helmchart/dacs" + helm version --short + helm show chart "$CHART_REF" --version + helm list -n "$NAMESPACE" --all + helm history "$RELEASE" -n "$NAMESPACE" + helm get values "$RELEASE" -n "$NAMESPACE" + kubectl get events -n "$NAMESPACE" --sort-by=.metadata.creationTimestamp + ``` + + Release queries may return `NotFound` if installation never created a release. +2. For download errors, check DNS, proxy, firewall, and HTTPS access to the + registry. Published artifacts at `dacstest.azurecr.io` support anonymous + pulls; private mirrors may require authentication. +3. Confirm the exact chart version exists. Use its `appVersion` for matching + image tags, rather than using the chart version as an image tag. +4. For an existing release, use `helm upgrade --install` with its original values + file and pinned version. For ownership conflicts, inspect the resource's + owning release; do not delete resources or strip ownership metadata blindly. +5. For timeouts, identify the failing pod using events and the + [installation TSG index](../installation-troubleshooting.md). Increase the + timeout only when the deployment is making progress. +6. With `--atomic`, a failed install is removed and a failed upgrade is rolled + back. Capture diagnostics while the operation is pending where possible; + resources may already be gone afterward. + +## Verification Steps + +```bash +helm status "$RELEASE" -n "$NAMESPACE" +kubectl rollout status deployment/dacs-manager -n "$NAMESPACE" --timeout=5m +``` + +Use your actual Deployment name if `namePrefix` is not `dacs`. Confirm release +status is `deployed`, then verify CacheServer readiness and run the +[quickstart smoke test](../quickstart.md#run-the-end-to-end-smoke-test). + +## Contact Information + +Use the channels in [SUPPORT.md](../../SUPPORT.md) and include redacted +diagnostics from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/image-pull.md b/docs/tsg/image-pull.md new file mode 100644 index 00000000..a32c283e --- /dev/null +++ b/docs/tsg/image-pull.md @@ -0,0 +1,50 @@ +# DACS Container Image Pull Failure + +## Symptoms + +A controller, CacheServer, prerequisite, or client pod reports `ErrImagePull` +or `ImagePullBackOff`. + +## Description + +The node cannot retrieve a required container image. Typical causes include an +incorrect repository/tag, registry connectivity or authentication failures, and +an image that does not support the node architecture. + +## Steps to Mitigate + +1. Set the namespace, identify the affected pod, and inspect its image and events. + Replace `` with the affected pod: + + ```bash + export NAMESPACE="dacs-system" + kubectl get pods -n "$NAMESPACE" -o wide + kubectl describe pod -n "$NAMESPACE" + kubectl get pod -n "$NAMESPACE" \ + -o jsonpath='{range .spec.initContainers[*]}{.image}{"\n"}{end}{range .spec.containers[*]}{.image}{"\n"}{end}' + ``` + +2. For `manifest unknown` or `not found`, correct the repository and image tag. + Remove unintended overrides or align them with the installed chart's + `appVersion`. Use the image path specified by the relevant installation + procedure; do not assume every client uses the same repository path. +3. For DNS, connection, or TLS errors, restore node-side registry access, + including any required proxy or trusted certificates. A workstation pull + does not establish node connectivity. +4. For `unauthorized`, verify the registry and repository. Published artifacts + at `dacstest.azurecr.io` allow anonymous pulls. For private mirrors, configure + image-pull authentication using the supported deployment mechanism. +5. For platform mismatches, use a supported Linux node architecture (`amd64` or + `arm64`) and an image that includes that platform. + +## Verification Steps + +Inspect the affected pod again. Confirm the image is pulled, the container +starts, and no new image-pull errors appear. For a client Job, confirm it +completes successfully rather than expecting its pod to remain Running. + +## Contact Information + +Contact the cluster or registry administrator for node connectivity and private +registry access. For DACS help, use [SUPPORT.md](../../SUPPORT.md) and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/missing-metrics.md b/docs/tsg/missing-metrics.md new file mode 100644 index 00000000..aa532a29 --- /dev/null +++ b/docs/tsg/missing-metrics.md @@ -0,0 +1,72 @@ +# DACS Missing Metrics + +## Symptoms + +Dashboards show no DACS data, Prometheus targets are absent/down, or expected +client hit/miss and latency metrics do not appear in logs or an OTLP collector. + +## Description + +Controller, CacheServer, and client metrics are separate surfaces. Controller +metrics use a protected HTTPS endpoint (default port `8443`); CacheServers have +a separate metrics port (default `9096`). Client logging and OTLP export have +their own settings and intervals. Enabling one surface does not configure all +the others. + +## Steps to Mitigate + +1. Identify which surface is missing and verify its workload is healthy: + + ```bash + export NAMESPACE="dacs-system" + export RELEASE="dacs" + helm get values "$RELEASE" -n "$NAMESPACE" --all + kubectl get pods,services -n "$NAMESPACE" + ``` + + Review values locally and redact secrets before sharing. +2. For controller metrics, verify `metrics.enable`, its configured port, and the + Service's ready endpoints (names assume `namePrefix=dacs`): + + ```bash + kubectl get service dacs-manager-metrics-service -n "$NAMESPACE" + kubectl get endpointslices -n "$NAMESPACE" \ + -l kubernetes.io/service-name=dacs-manager-metrics-service + ``` + + Inspect collector errors for TLS or RBAC failures. Use the approved trust + and scrape identity; do not disable authentication or certificate validation. +3. If using the chart's ServiceMonitor, verify the Prometheus Operator CRDs + exist and `prometheus.enable` is enabled: + + ```bash + kubectl get crd servicemonitors.monitoring.coreos.com + kubectl get servicemonitors -n "$NAMESPACE" + ``` + + A missing CRD means the optional integration is unavailable. Also check the + Prometheus instance's namespace/label selectors and target discovery. +4. For CacheServer metrics, check `cache.metricsPort` and the actual scrape + target/port rather than scraping the controller endpoint for cache data. +5. For client metrics, review `STORAGE_METRIC_LOGGING_ENABLED` and + `STORAGE_METRIC_OTEL_ENABLED` before process startup. Logging defaults to + enabled at 15-minute intervals; OTLP defaults to disabled with a 5-minute + export interval when enabled. Verify the OTLP HTTP endpoint and collector + reachability from the client. +6. Use the [client metrics configuration](../../src/subsystems/storage/storage_intercept/README.md#configure-client-metrics) + and generate a small known workload. Allow the configured interval to elapse; + an empty dashboard immediately after startup is not sufficient evidence of + export failure. + +## Verification Steps + +Confirm the intended scrape target is healthy or client exports reach the +collector. Run known reads/writes and verify relevant counters or latency data +update within the configured interval. Check that dashboards select the right +namespace, service, process, and time range. + +## Contact Information + +Contact the monitoring administrator for collector, trust, and scrape-identity +configuration. For DACS help, use [SUPPORT.md](../../SUPPORT.md) and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/node-recovery.md b/docs/tsg/node-recovery.md new file mode 100644 index 00000000..36d0f9e7 --- /dev/null +++ b/docs/tsg/node-recovery.md @@ -0,0 +1,65 @@ +# DACS Recovery After Node Disruption + +## Symptoms + +Cache access degrades after a reboot, drain, node upgrade, or node replacement. +Pods may remain Pending, clients may time out, or previously cached data may be +missing. + +## Description + +DACS uses node-local storage. An ephemeral cache is lost when its pod is removed. +NVMe data may survive a reboot if the disks survive, but node/disk replacement +does not preserve it. Storage preparation, placement, scaling, and client +discovery must all be checked after a disruption. Do not assume cache-only +writes are recoverable from remote storage. + +## Steps to Mitigate + +1. Record the maintenance event and identify affected nodes and pods: + + ```bash + export NAMESPACE="dacs-system" + export CACHE_NAME="cache-sample" + kubectl get nodes -o wide + kubectl get pods -n "$NAMESPACE" \ + -l "dacs.azure.com/cache-name=$CACHE_NAME" -o wide + kubectl get events -n "$NAMESPACE" --sort-by=.metadata.creationTimestamp + kubectl describe node + ``` + +2. Have the cluster administrator resolve node health or capacity problems. + Do not force-delete pods or uncordon nodes still undergoing maintenance. +3. For NVMe deployments, inspect prerequisite pods on affected/replacement nodes: + + ```bash + kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/name=dacs-prereq -o wide + kubectl logs -n "$NAMESPACE" -c nvme-raid-setup --tail=200 + ``` + + Follow the [NVMe setup TSG](nvme-setup.md) to verify the actual host mount. + Do not format disks to bypass an assembly or mount failure. +4. Verify replacement-node labels and eligibility. Use the + [scheduling TSG](cacheserver-scheduling.md) for Pending pods and the + [scaling TSG](cache-scaling.md) if node-count changes affect replicas. +5. Check client discovery refresh and discovered endpoint reachability using + the [connectivity TSG](cache-connectivity.md). StorageIntercept discovery + settings include `cacheServerDiscoveryEnabled`, + `cacheServerDiscoveryEndpoint`, and `cacheServerDiscoveryIntervalInSeconds`. +6. Rewarm lost cache contents from an authoritative source at controlled + concurrency. If cache-only or pending asynchronous writes were affected, + preserve remaining evidence and escalate before deleting data or claiming + recovery. + +## Verification Steps + +Confirm intended nodes are healthy, NVMe mounts exist where required, and the +expected cache pods become Ready. Verify discovery no longer directs traffic to +unavailable endpoints, then read representative application data and validate +its contents. Track cold-cache load separately from persistent failures. + +## Contact Information + +Contact the cluster administrator for node recovery. For DACS help, use +[SUPPORT.md](../../SUPPORT.md) and the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/nvme-setup.md b/docs/tsg/nvme-setup.md new file mode 100644 index 00000000..17708811 --- /dev/null +++ b/docs/tsg/nvme-setup.md @@ -0,0 +1,74 @@ +# DACS NVMe Prerequisite Setup Failure + +## Symptoms + +The prerequisite DaemonSet is missing from expected nodes, its init container +fails or restarts, or the expected cache mount is absent. Logs may report +package-installation failures or `No unused NVMe devices found.`. + +## Description + +The `dacs-prereq` chart prepares local NVMe storage on selected nodes. Single-disk +installations use ext4 directly; multiple disks use RAID0. The setup script can +exit successfully when no unused NVMe disks are found, so a Ready prerequisite +pod alone does not prove that the cache mount exists. + +This guide does not apply to `cache.storage.ephemeral=true`; the ephemeral +quickstart does not require the prerequisite chart. + +## Steps to Mitigate + +1. Inspect the DaemonSet and affected pod. Replace `` with its actual + name; these commands assume the default prerequisite name `dacs-prereq`: + + ```bash + export NAMESPACE="dacs-system" + kubectl get daemonset dacs-prereq -n "$NAMESPACE" + kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/name=dacs-prereq -o wide + kubectl describe pod -n "$NAMESPACE" + kubectl logs -n "$NAMESPACE" -c nvme-raid-setup --tail=200 + ``` + + If the init container restarted, also collect its `--previous` logs. +2. For zero desired pods or missing nodes, check prerequisite `nodeSelector` and + `tolerations` against the intended cache nodes. +3. For admission denial of privileged containers or host PID access, ask the + cluster administrator to review required privileges. Do not weaken + cluster-wide policy as a workaround. +4. For package failures, verify host package-repository access. Supported host + package managers are Azure Linux `tdnf` and Ubuntu `apt-get`. +5. For `OOMKilled`, inspect the deployed init-container limit. Current defaults + request `512Mi` and allow `2Gi`; older or overridden limits may be too small + for package metadata refreshes. +6. For no unused disks or a missing mount, confirm the VM has local NVMe data + disks. With approved access, run these **read-only commands on the host**: + + ```bash + lsblk -o NAME,TYPE,SIZE,FSTYPE,LABEL,MOUNTPOINT + findmnt --mountpoint /var/lib/ssd/cacheserver + cat /proc/mdstat + df -h /var/lib/ssd/cacheserver + ``` + + Substitute your configured mount point. Inspect disk signatures and ownership; + do not erase them to make setup proceed. A missing RAID array is not itself + a failure for a single-disk installation. +7. Ensure prerequisite `mountPoint` matches + `cache.storage.hostPath.nodeCacheFolderPath`. + +NVMe setup can format unused disks. Confirm node and disk ownership before +installing or rerunning it. Do not run `mkfs`, `wipefs`, or force-create a RAID +array as a troubleshooting shortcut. + +## Verification Steps + +Confirm setup completes on every intended cache node. Use `findmnt` and `lsblk` +on each host to verify the configured path is a mount backed by the intended +data disk/filesystem, not merely a directory on the OS disk. Verify CacheServers +using that storage can start. + +## Contact Information + +Contact the cluster/storage administrator before disk or host changes. +For DACS help, use [SUPPORT.md](../../SUPPORT.md) and include redacted host +diagnostics from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/slow-reads.md b/docs/tsg/slow-reads.md new file mode 100644 index 00000000..154345c6 --- /dev/null +++ b/docs/tsg/slow-reads.md @@ -0,0 +1,65 @@ +# DACS Slow Reads or Low Throughput + +## Symptoms + +Application reads take longer than usual, throughput drops, or request timeouts +increase even though CacheServer pods are Running. + +## Description + +Reads may involve cache discovery, cache transfer, or a remote storage fetch +on a cache miss. Cold caches, frequent eviction, client concurrency, CPU/memory +pressure, network limits, and remote storage latency can affect performance. +A healthy controller does not establish data-path performance. + +## Steps to Mitigate + +1. Record a representative request size, concurrency, dataset, duration, and + previous baseline. Distinguish first reads from repeated reads of the same + unchanged data. +2. Inspect affected pods and recent events: + + ```bash + export NAMESPACE="dacs-system" + export CACHE_NAME="cache-sample" + kubectl get pods -n "$NAMESPACE" \ + -l "dacs.azure.com/cache-name=$CACHE_NAME" -o wide + kubectl describe pod -n "$NAMESPACE" + kubectl logs -n "$NAMESPACE" -c cacheserver --tail=200 + kubectl get events -n "$NAMESPACE" --sort-by=.metadata.creationTimestamp + ``` + +3. If Metrics Server is installed, inspect resource use: + + ```bash + kubectl top pods -n "$NAMESPACE" + kubectl top nodes + ``` + + If the metrics API is unavailable, use your existing monitoring system. + Review CPU throttling, memory pressure, and node/network utilization over the + incident window rather than relying only on a point-in-time sample. +4. Compare client hit/miss and latency metrics. For continued misses, follow the + [cache misses TSG](cache-misses.md). For timeouts or DNS errors, follow the + [connectivity TSG](cache-connectivity.md). +5. Review `cache.cpuLimit`, `cache.memoryLimitMB`, `cache.serverThreads`, and any + `cache.egressBandwidthLimitGbps` cap against the workload and node capacity. + Change one setting at a time through the original Helm values. +6. For StorageIntercept, review `chunkOperationMaxConcurrency`, + `maxConnsPerBlobClient`, and prefetch settings using its + [configuration guide](../../src/subsystems/storage/storage_intercept/README.md). + Increasing threads, buffers, or retries indiscriminately can worsen pressure. +7. If cache hits are fast but remote reads are slow, investigate storage/network + latency or throttling rather than restarting healthy cache pods. + +## Verification Steps + +Repeat the same bounded workload under comparable conditions, recording cold +and warm read latency/throughput separately. Confirm the application's latency +or throughput target is met and timeout/resource-pressure signals do not grow. +Do not use the quickstart's elapsed time as a production performance threshold. + +## Contact Information + +Use [SUPPORT.md](../../SUPPORT.md) with a redacted workload baseline, observed +performance, and the [escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/docs/tsg/storage-access.md b/docs/tsg/storage-access.md new file mode 100644 index 00000000..f2a55baa --- /dev/null +++ b/docs/tsg/storage-access.md @@ -0,0 +1,59 @@ +# DACS Remote Storage Access Failure + +## Symptoms + +Remote reads, metadata calls, or writes fail with HTTP `401`/`403`, +authentication errors, token-acquisition failures, or storage endpoint +connection errors. Warm cached reads may succeed while new data fails. + +## Description + +Cache connectivity and remote Blob access are separate paths. Client workloads +need valid storage credentials/identity, data-plane permissions, and network +access to storage and any token endpoint. A cache-only smoke test does not +validate Blob authorization. + +## Steps to Mitigate + +1. Capture the failing operation, sanitized endpoint, HTTP/error code, and + timestamp from the affected workload. Do not log tokens or SAS query strings. + Inspect its pod events and application logs: + + ```bash + export CLIENT_NAMESPACE="application-namespace" + kubectl describe pod -n "$CLIENT_NAMESPACE" + kubectl logs -n "$CLIENT_NAMESPACE" -c --tail=200 + ``` + +2. Identify the authentication mode configured in the affected process. Check + for stale overrides after credential rotation. Review the + [client authentication guide](../../src/subsystems/storage/dacs_client/README.md#authentication) + or [StorageIntercept guide](../../src/subsystems/storage/storage_intercept/README.md) + for the actual client in use. +3. For SAS or supplied access tokens, check expiry, audience/scope, and required + operation permissions locally without displaying the token. Rotate/update + credentials through the normal secret-management workflow. +4. For managed/workload identity, have the identity administrator verify the + intended identity, federation/service-account mapping, projected-token + configuration, and storage data-plane roles. Do not grant broad subscription + permissions as a shortcut. +5. For `403`, check both authorization and storage firewall/private-endpoint + rules. For DNS/TLS/timeouts, check connectivity from the affected workload + to storage and the applicable identity endpoint. Keep TLS validation enabled. +6. After correction, test a small permitted operation from the same workload + identity/network context. A successful request from an administrator's + workstation does not establish application access. + +## Verification Steps + +Verify remote metadata/read access for an object not already cached. If writes +are required, create a uniquely named test object through the intended write +mode and confirm it exists remotely. Check that authorization failures do not +recur across normal token refresh. Remove test data only through the approved +application/storage workflow. + +## Contact Information + +Contact the storage/identity administrator for access changes. For DACS help, +use [SUPPORT.md](../../SUPPORT.md) with redacted diagnostics from the +[escalation checklist](../installation-troubleshooting.md#escalation-checklist). diff --git a/user-guide.md b/user-guide.md index bef9d0b9..e8225ec7 100644 --- a/user-guide.md +++ b/user-guide.md @@ -316,6 +316,13 @@ kubectl describe cache -n dacs-system ## Troubleshooting +For symptom-based installation checks, fixes, readiness checkpoints, and an +escalation checklist, see the +[installation troubleshooting guide (TSG)](docs/installation-troubleshooting.md). +For common runtime issues such as slow reads, eviction, scaling, node recovery, +storage access, and client writes, see the +[post-installation troubleshooting guides](docs/operations-troubleshooting.md). + ### Controller Not Starting Check controller logs: From d49ce13680ecde2ddef4da3712be7e0513997b95 Mon Sep 17 00:00:00 2001 From: Pawan Date: Wed, 28 Oct 2026 22:45:25 +0530 Subject: [PATCH 2/2] docs: correct NVMe diagnostics and warn before cache downsizing Address review feedback on PR #145. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/tsg/cacheserver-startup.md | 11 +++++++++-- docs/tsg/nvme-setup.md | 25 ++++++++++++++++++++----- 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/docs/tsg/cacheserver-startup.md b/docs/tsg/cacheserver-startup.md index faa9c936..eec9a3b0 100644 --- a/docs/tsg/cacheserver-startup.md +++ b/docs/tsg/cacheserver-startup.md @@ -39,8 +39,15 @@ configures cache capacity; it is not the container memory limit. expected ownership. Follow the [NVMe setup TSG](nvme-setup.md) for host storage; do not apply broad permissions changes blindly. 5. For `No space left on device` or `DiskPressure`, inspect the backing filesystem - and node ephemeral storage. Reduce cache sizing or add capacity; do not delete - unrelated host data. + and node ephemeral storage. Add suitable capacity or investigate non-cache + disk use with the administrator; do not delete unrelated host data. + **Before considering a reduction of `cache.serverSizeInGB`, confirm + authoritative remote copies and resolve pending asynchronous writes.** + On subsequent startup, `check_space_overuse.sh` deletes every entry in the + cache directory if its existing contents exceed the configured limit; it does + not merely trim the excess. Cache-only or not-yet-persisted data can be lost. + Follow the [capacity and downsizing safeguards](cache-capacity.md#steps-to-mitigate) + before changing sizing or restarting. 6. For probe failures or process exits, use the actual startup error to check ports, configuration, and chart/image compatibility before restarting. diff --git a/docs/tsg/nvme-setup.md b/docs/tsg/nvme-setup.md index 17708811..c76e5aca 100644 --- a/docs/tsg/nvme-setup.md +++ b/docs/tsg/nvme-setup.md @@ -4,7 +4,8 @@ The prerequisite DaemonSet is missing from expected nodes, its init container fails or restarts, or the expected cache mount is absent. Logs may report -package-installation failures or `No unused NVMe devices found.`. +missing-tool errors or `No unused NVMe devices found.`. Older deployments may +also report runtime package-installation failures. ## Description @@ -35,11 +36,25 @@ quickstart does not require the prerequisite chart. 3. For admission denial of privileged containers or host PID access, ask the cluster administrator to review required privileges. Do not weaken cluster-wide policy as a workaround. -4. For package failures, verify host package-repository access. Supported host - package managers are Azure Linux `tdnf` and Ubuntu `apt-get`. +4. Inspect the deployed init-container image and tag: + + ```bash + kubectl get pod -n "$NAMESPACE" \ + -o jsonpath='{range .spec.initContainers[?(@.name=="nvme-raid-setup")]}{.image}{"\n"}{end}' + ``` + + The current cache-setup image bundles `e2fsprogs`, `mdadm`, and `util-linux`; + setup does not install these packages on the host at runtime. For + `Required tool ... is missing from the cache-setup image`, verify the deployed + repository/tag and restore the matching chart/image rather than installing + host packages. Only for older deployments whose logs show runtime package + installation should you check package-repository access for Azure Linux + `tdnf` or Ubuntu `apt-get`. 5. For `OOMKilled`, inspect the deployed init-container limit. Current defaults - request `512Mi` and allow `2Gi`; older or overridden limits may be too small - for package metadata refreshes. + request `512Mi` and allow `2Gi`. Use current/previous logs and resource + observations to identify the failing operation before adjusting the limit. + Package metadata refreshes are relevant only to older deployments that + install packages at runtime, not the current bundled-tools setup. 6. For no unused disks or a missing mount, confirm the VM has local NVMe data disks. With approved access, run these **read-only commands on the host**: