docs: add DACS installation and operations troubleshooting guides

README.md

@@ -86,6 +86,8 @@ See [Architecture](docs/architecture.md) for the detailed request flow.
 | Understand the system architecture | [Architecture](docs/architecture.md) |
 | Understand the security model and trust boundary | [Security model](docs/security-model.md) |
 | Install and operate DACS | [Deployment user guide](user-guide.md) |
+| Diagnose basic installation failures | [Installation troubleshooting guide](docs/installation-troubleshooting.md) |
+| Diagnose common runtime issues | [Post-installation troubleshooting guides](docs/operations-troubleshooting.md) |
 | Build from a clean clone | [Build from source](docs/build-from-source.md) |
 | Understand Azure Pipelines build caching | [Azure Pipelines build and caching](docs/azure-pipeline-build-and-caching.md) |
 | Understand GitHub Actions build caching | [GitHub Actions build and caching](docs/github-actions-build-and-caching.md) |

docs/installation-troubleshooting.md

@@ -0,0 +1,87 @@
+# DACS installation troubleshooting guides (TSGs)
+
+Use these guides when a DACS Helm installation fails or the first cache does not
+become usable. Each issue has a separate guide with Symptoms, Description,
+Steps to Mitigate, Verification Steps, and Contact Information.
+
+These guides cover the controller-based `dacs` chart used by the
+[AKS quickstart](quickstart.md) and [deployment user guide](../user-guide.md).
+The standalone `cache-server` chart has different values and resource names;
+do not mix its `cacheServer.*` settings with the controller chart's `cache.*`
+settings.
+
+For issues after installation, see the
+[post-installation troubleshooting guides](operations-troubleshooting.md).
+
+## Installation issues
+
+| Symptom | Troubleshooting guide |
+| --- | --- |
+| Wrong cluster, `Unauthorized`, or `Forbidden` | [Cluster access or permissions failure](tsg/cluster-access.md) |
+| Chart pull fails or Helm times out | [Helm installation failure](tsg/helm-installation.md) |
+| `ErrImagePull` or `ImagePullBackOff` | [Container image pull failure](tsg/image-pull.md) |
+| Controller is not Ready | [Controller startup failure](tsg/controller-startup.md) |
+| Cache resource is missing or no servers appear | [Cache creation failure](tsg/cache-creation.md) |
+| Cache creation fails with a webhook error | [Admission webhook failure](tsg/admission-webhook.md) |
+| CacheServer pods remain `Pending` | [CacheServer scheduling failure](tsg/cacheserver-scheduling.md) |
+| Prereq init container fails or storage is missing | [NVMe prerequisite setup failure](tsg/nvme-setup.md) |
+| CacheServer restarts or stays unready | [CacheServer startup failure](tsg/cacheserver-startup.md) |
+| Pods are Ready but the client cannot connect | [Cache discovery or client connectivity failure](tsg/cache-connectivity.md) |
+
+## Initial diagnostics
+
+Examples assume namespace `dacs-system`, Helm release `dacs`, Cache
+`cache-sample`, and the default `namePrefix=dacs`. Substitute your actual values
+and resource names. Replace angle-bracket placeholders before running commands.
+Individual guides include their own variable setup so they can be used directly.
+
+```bash
+export NAMESPACE="dacs-system"
+export RELEASE="dacs"
+export CACHE_NAME="cache-sample"
+
+kubectl config current-context
+kubectl cluster-info
+helm list -n "$NAMESPACE" --all
+helm status "$RELEASE" -n "$NAMESPACE"
+kubectl get deployments,daemonsets,statefulsets,pods,services -n "$NAMESPACE" -o wide
+kubectl get cache -n "$NAMESPACE"
+kubectl get events -n "$NAMESPACE" --sort-by=.metadata.creationTimestamp
+```
+
+If a resource does not exist yet, its `NotFound` error identifies the stage to
+investigate. If `kubectl` cannot reach the cluster, fix cluster access first.
+
+## Safety precautions
+
+- Record events and logs before restarting pods or retrying Helm.
+- Keep the original values file and chart version. Do not retry with defaults
+  that silently change storage mode, node selection, or sizing.
+- Do not delete the namespace, Cache CRs, CRDs, or storage as a diagnostic step.
+  Replacing an ephemeral cache pod loses its cached data.
+- NVMe setup can format unused disks. Confirm node and disk ownership before
+  installing or rerunning the prerequisite chart. Do not use `mkfs`, `wipefs`,
+  or force-create a RAID array as a troubleshooting shortcut.
+
+## Escalation checklist
+
+If the issue persists, collect:
+
+- Installation command, timestamp/time zone, chart version and `appVersion`,
+  and relevant values overrides.
+- Cluster/Kubernetes version, node OS/architecture/VM size, namespace, release,
+  Cache name, and storage mode.
+- Helm status/history, pod inventory, affected pod descriptions and events.
+- Current and previous logs for the affected controller, CacheServer, or prereq
+  init container.
+- Cache CR specification/status and, for storage failures, read-only host
+  diagnostics from the [NVMe setup guide](tsg/nvme-setup.md).
+- Expected versus actual behavior and the first failing verification step.
+
+Review and redact diagnostics before sharing. Do not include credentials,
+kubeconfig files, tokens, Kubernetes Secret contents, or TLS private keys.
+Logs, values, CRs, and host output may also contain sensitive storage URLs and
+private endpoints.
+
+Use the channels in [SUPPORT.md](../SUPPORT.md). Report suspected vulnerabilities
+through [SECURITY.md](../SECURITY.md), not a public issue.

docs/operations-troubleshooting.md

@@ -0,0 +1,49 @@
+# DACS post-installation troubleshooting guides (TSGs)
+
+Use these guides for basic issues after DACS has been installed. Each issue has
+its own Symptoms, Description, Steps to Mitigate, Verification Steps, and
+Contact Information sections.
+
+| Symptom | Troubleshooting guide |
+| --- | --- |
+| Reads become slow or throughput drops | [Slow reads or low throughput](tsg/slow-reads.md) |
+| Repeated reads continue to miss the cache | [Repeated cache misses](tsg/cache-misses.md) |
+| Cache writes fail or nodes report disk pressure | [Cache capacity or disk pressure](tsg/cache-capacity.md) |
+| Replica count does not change as expected | [Unexpected cache scaling](tsg/cache-scaling.md) |
+| Cache access degrades after a node restart or replacement | [Recovery after node disruption](tsg/node-recovery.md) |
+| Remote Blob requests fail with authentication or access errors | [Remote storage access failure](tsg/storage-access.md) |
+| Writes fail or data is absent from remote storage | [Client write failure or unexpected persistence](tsg/client-writes.md) |
+| Dashboards or client metrics are missing | [Missing metrics](tsg/missing-metrics.md) |
+| Reads report checksum mismatches | [Checksum validation failure](tsg/checksum-failure.md) |
+
+For initial setup failures, use the
+[installation troubleshooting index](installation-troubleshooting.md).
+Existing guides for [client connectivity](tsg/cache-connectivity.md),
+[CacheServer startup](tsg/cacheserver-startup.md), and
+[controller startup](tsg/controller-startup.md) also apply to runtime incidents.
+
+## Before making changes
+
+- Record the failure time, affected workloads, recent configuration changes,
+  node maintenance, and relevant logs/events before restarting anything.
+- Examples use namespace `dacs-system`, Cache `cache-sample`, and
+  `namePrefix=dacs`. Substitute actual names and replace angle-bracket
+  placeholders. Each guide includes the variables it needs.
+- Change Helm-managed settings through the original values file and pinned chart
+  version. Direct changes to generated StatefulSets may be reconciled away.
+- Cache eviction, pod replacement, scaling, and node replacement can remove
+  cached data. Do not treat cache-only writes or pending asynchronous uploads as
+  durable Blob data. Confirm the authoritative copy before destructive actions.
+- Do not delete Cache resources, CRDs, namespaces, or cache directories to
+  diagnose an issue. Do not disable checksum, TLS, or authorization checks.
+- Redact credentials, tokens, SAS query strings, and sensitive paths from
+  diagnostics. Use the existing
+  [escalation checklist](installation-troubleshooting.md#escalation-checklist).
+
+## Verification
+
+Verify the affected application operation, not just pod status. Where
+appropriate, also run the
+[quickstart smoke test](quickstart.md#run-the-end-to-end-smoke-test).
+It verifies cache upload/download and checksum validation; it does not verify
+remote Blob authorization, application write durability, or production performance.

docs/quickstart.md

@@ -17,6 +17,8 @@ This configuration is for evaluation only. Cache data is lost when the
 CacheServer pod or release is removed. For NVMe/RAID storage, production sizing,
 scaling, authentication, troubleshooting, and uninstall details, see the
 [deployment user guide](../user-guide.md).
+If installation or the smoke test fails, use the
+[installation troubleshooting guide (TSG)](installation-troubleshooting.md).
 
 ## Set variables
 

docs/tsg/admission-webhook.md

@@ -0,0 +1,53 @@
+# DACS Admission Webhook Failure
+
+## Symptoms
+
+Cache creation or updates fail with `failed calling webhook`,
+`no endpoints available`, connection refused, a timeout, or an `x509` error.
+Alternatively, the webhook rejects an invalid Cache field.
+
+## Description
+
+The DACS admission webhook defaults and validates Cache resources. The Kubernetes
+API server must reach its Service and trust its serving certificate. By default,
+the controller manages webhook certificates; cert-manager is not required for
+the default installation. A field-validation rejection is not a connectivity
+failure.
+
+## Steps to Mitigate
+
+1. Inspect the Service, endpoints, and controller logs. These names assume
+   `namePrefix=dacs`; substitute your actual names if different:
+
+   ```bash
+   export NAMESPACE="dacs-system"
+   kubectl get service dacs-webhook-service -n "$NAMESPACE"
+   kubectl get endpointslices -n "$NAMESPACE" \
+     -l kubernetes.io/service-name=dacs-webhook-service
+   kubectl logs deployment/dacs-manager -n "$NAMESPACE" -c manager --tail=200
+   ```
+
+2. For no ready endpoints, restore
+   [controller readiness](controller-startup.md).
+3. For timeouts, verify API-server connectivity to the webhook Service, including
+   cluster firewall and network policies. The default Service exposes TCP `443`
+   and targets controller port `9443`.
+4. For certificate errors, inspect certificate-management logs and confirm that
+   the webhook configuration references the intended Service and namespace.
+   Do not print TLS private keys or delete certificate Secrets blindly.
+5. If optional integrations were enabled, verify their prerequisites separately.
+6. For validation rejection, correct the Cache field named in the error.
+   Do not disable admission validation or change the failure policy merely to
+   bypass the failure.
+
+## Verification Steps
+
+Confirm the webhook Service has ready endpoints. Retry the original Cache
+creation or update and confirm it is accepted without connection or certificate
+errors. For invalid input, verify the corrected resource passes validation.
+
+## Contact Information
+
+Contact the cluster administrator for API-server connectivity or policy changes.
+For DACS help, use [SUPPORT.md](../../SUPPORT.md) and the
+[escalation checklist](../installation-troubleshooting.md#escalation-checklist).

docs/tsg/cache-capacity.md

@@ -0,0 +1,65 @@
+# DACS Cache Capacity or Disk Pressure
+
+## Symptoms
+
+Cache writes fail with `No space left on device`, nodes report `DiskPressure`,
+pods are evicted, or frequent cache eviction causes repeated remote reads.
+
+## Description
+
+Configured cache capacity and physical filesystem free space are different
+limits. Other node workloads, logs, or inode exhaustion can consume the backing
+filesystem. DACS eviction controls cache retention; it does not clean unrelated
+host data. Ephemeral storage also consumes node-local capacity.
+
+## Steps to Mitigate
+
+1. Inspect the Cache, affected pod, node, and eviction logs:
+
+   ```bash
+   export NAMESPACE="dacs-system"
+   export CACHE_NAME="cache-sample"
+   kubectl describe cache "$CACHE_NAME" -n "$NAMESPACE"
+   kubectl describe pod <cache-pod> -n "$NAMESPACE"
+   kubectl describe node <cache-node>
+   kubectl logs <cache-pod> -n "$NAMESPACE" -c cacheserver --tail=300
+   ```
+
+2. With approved host access, check the backing filesystem and inodes using
+   read-only commands. Substitute the configured host path:
+
+   ```bash
+   findmnt --mountpoint /var/lib/ssd/cacheserver
+   df -h /var/lib/ssd/cacheserver
+   df -i /var/lib/ssd/cacheserver
+   ```
+
+   For ephemeral mode, inspect the node filesystem backing the pod's `emptyDir`
+   through your cluster's normal node diagnostics.
+3. If the expected NVMe mount is absent, use the [NVMe setup TSG](nvme-setup.md).
+   Do not assume an existing directory is backed by the data disk.
+4. Review `cache.serverSizeInGB` and `cache.eviction` settings, including
+   `enableAccessTimeEviction`, `evictionMaintainFreeSpace`, `sizeThreshold`,
+   `sizeTarget`, and `passFrequency`. Keep the target below the threshold and
+   leave filesystem headroom. Inspect logs for invalid-setting warnings and
+   eviction progress.
+5. If the working set exceeds safe capacity, add suitable capacity or revise
+   retention/sizing through the original values file. Have the administrator
+   address unrelated host disk use; do not delete arbitrary host/cache files.
+6. Before reducing `cache.serverSizeInGB`, assess data impact: the startup
+   space-check script can clear the cache directory when existing contents exceed
+   the configured limit. Confirm authoritative remote copies and pending writes
+   before any size change, eviction-policy change, or restart.
+
+## Verification Steps
+
+Confirm filesystem space and inodes have headroom, node disk pressure clears,
+and affected pods remain stable. Run a bounded application write/read and
+observe capacity and eviction through at least the configured eviction interval.
+Verify pressure does not recur immediately under the representative workload.
+
+## Contact Information
+
+Contact the cluster/storage administrator for host capacity. For DACS help,
+use [SUPPORT.md](../../SUPPORT.md) and the
+[escalation checklist](../installation-troubleshooting.md#escalation-checklist).

docs/tsg/cache-connectivity.md

@@ -0,0 +1,66 @@
+# DACS Cache Discovery or Client Connectivity Failure
+
+## Symptoms
+
+CacheServer pods are Ready, but a client cannot discover servers or connect.
+The first request may fail with DNS resolution errors, connection errors,
+timeouts, or a failed quickstart smoke-test Job.
+
+## Description
+
+Clients use the cache discovery Service to locate CacheServers and then reach
+cache endpoints. Successful discovery alone does not prove endpoint
+connectivity. Service DNS normally resolves inside the cluster, not directly
+from a workstation.
+
+## Steps to Mitigate
+
+1. Inspect pods, the discovery Service, endpoints, and network policies:
+
+   ```bash
+   export NAMESPACE="dacs-system"
+   export CACHE_NAME="cache-sample"
+   kubectl get pods -n "$NAMESPACE" \
+     -l "dacs.azure.com/cache-name=$CACHE_NAME" -o wide
+   kubectl get service "${CACHE_NAME}-discovery" -n "$NAMESPACE"
+   kubectl get endpointslices -n "$NAMESPACE" \
+     -l "kubernetes.io/service-name=${CACHE_NAME}-discovery"
+   kubectl get networkpolicy -n "$NAMESPACE"
+   ```
+
+2. For a missing Service, inspect
+   [Cache creation and reconciliation](cache-creation.md).
+3. For no ready endpoints, restore
+   [CacheServer readiness](cacheserver-startup.md) and verify Service selectors.
+4. Use the correct namespace in the client endpoint, for example
+   `cache-sample-discovery.dacs-system.svc.cluster.local`. Substitute the cluster
+   DNS suffix if it differs from `cluster.local`.
+5. Test DNS from the affected workload using its available tools. Review policies
+   in both client and cache namespaces, DNS egress, and access to discovered
+   cache endpoints on the configured data port (default TCP `9065`).
+6. For failed quickstart Jobs, inspect events and logs:
+
+   ```bash
+   kubectl describe job dacs-smoke-upload -n "$NAMESPACE"
+   kubectl logs job/dacs-smoke-upload -n "$NAMESPACE"
+   kubectl describe job dacs-smoke-download -n "$NAMESPACE"
+   kubectl logs job/dacs-smoke-download -n "$NAMESPACE"
+   ```
+
+   These names apply to the quickstart; use your actual Job names otherwise.
+   Inspect the affected Job pod for startup failures. Use matching client images
+   and the same key prefix for upload and download.
+
+## Verification Steps
+
+Run the [quickstart smoke test](../quickstart.md#run-the-end-to-end-smoke-test).
+Both Jobs must report Complete, their logs must contain
+`All iterations succeeded: true`, and the checksum-validated download must
+succeed. A deployed Helm release or a Running pod alone does not establish
+end-to-end success.
+
+## Contact Information
+
+Contact the cluster/network administrator for DNS and policy issues.
+For DACS help, use [SUPPORT.md](../../SUPPORT.md) and include redacted diagnostics
+from the [escalation checklist](../installation-troubleshooting.md#escalation-checklist).