Repository navigation
docs: add DACS installation and operations troubleshooting guides - #145
Pawan Prakash Sharma (pawanpraka1) wants to merge 2 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
kapilupadhayay
left a comment
kapilupadhayay
left a comment
There was a problem hiding this comment.
Two findings from the TSG review, checked against the implementation at this revision. The cache-downsizing deletion behavior was also reproduced using isolated synthetic data.
| cluster administrator to review required privileges. Do not weaken | ||
| cluster-wide policy as a workaround. | ||
| 4. For package failures, verify host package-repository access. Supported host | ||
| package managers are Azure Linux `tdnf` and Ubuntu `apt-get`. |
There was a problem hiding this comment.
[P2] Distinguish legacy package installation from the current bundled-tools setup.
The current prerequisite path does not install packages on the host at runtime: Docker/CacheSetup.Dockerfile:17-20 bundles e2fsprogs, mdadm, and util-linux into the setup image, and helm/dacs-prereq/scripts/setup-raid.sh:66-71 checks for the required tools and fails if one is missing. Host apt-get/tdnf repository access therefore is not the appropriate diagnostic for this version. The package-metadata-refresh explanation in step 5 is also not applicable to the current script.
Please gate that advice explicitly to older deployments. For the current setup, have operators inspect the deployed init-container image/tag and missing-tool errors, and investigate any OOM against the actual failing operation rather than assuming a package refresh.
There was a problem hiding this comment.
Addressed in d49ce13. The NVMe TSG now checks the deployed init-container image/tag, describes the current bundled-tools setup and missing-tool errors, and explicitly limits package-repository/metadata-refresh advice to older runtime-install deployments. OOM diagnosis now starts with the actual failing operation and deployed resources.
| expected ownership. Follow the [NVMe setup TSG](nvme-setup.md) for host | ||
| storage; do not apply broad permissions changes blindly. | ||
| 5. For `No space left on device` or `DiskPressure`, inspect the backing filesystem | ||
| and node ephemeral storage. Reduce cache sizing or add capacity; do not delete |
There was a problem hiding this comment.
[P1] Warn about full-cache deletion before recommending downsizing.
Reducing cache.serverSizeInGB is not a non-destructive response to disk pressure. The generated startup command invokes /root/check_space_overuse.sh before starting CacheServer (src/subsystems/cache-controller/internal/controller/templates/cacheserver-app.yaml:173-174). When existing contents exceed the configured limit, that script deletes every entry in the cache directory, rather than only trimming excess data (Docker/check_space_overuse.sh:32-36). I reproduced this behavior with an isolated synthetic cache.
Please require confirmation of authoritative remote copies and pending asynchronous writes before recommending a size reduction, and link to the safeguards already in docs/tsg/cache-capacity.md:49-52. Otherwise an operator following this standalone TSG can lose cache-only or not-yet-persisted data on the subsequent restart.
There was a problem hiding this comment.
Addressed in d49ce13. The standalone startup TSG no longer recommends immediate downsizing. It explicitly requires authoritative remote copies and resolution of pending asynchronous writes before considering a reduction, warns that the startup check deletes every entry rather than trimming excess, and links to the capacity/downsizing safeguards.
Address review feedback on PR #145. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Validation
Related issue