Put the loading index on a node-local NVMe ephemeral volume (blocked on #280)#281
Open
gaurav wants to merge 1 commit into
Open
Put the loading index on a node-local NVMe ephemeral volume (blocked on #280)#281gaurav wants to merge 1 commit into
gaurav wants to merge 1 commit into
Conversation
The index is the volume that wants speed -- it takes the whole write load of the indexing run and then the read-and-write of the optimize -- and it is also the one volume we can afford to lose, since once the tarball exists the index is worthless and could be rebuilt from the synonym files next door. That makes it an exact fit for nvme-ephemeral: created with the pod, destroyed with it. data/ stays on a persistent PVC. It holds the ~130G download and the backup tarball, which is the only thing the exercise produces. fsGroup 1000 matches the image's nru user; without it the volume arrives root-owned and Solr cannot write to it. nameres-loading-solr.k8s.yaml is deleted rather than kept as a fallback: a manifest nobody applies is a manifest nobody maintains, and it would sit there slowly drifting out of step with the pod it is supposed to pair with. The README explains how to get it back out of git history in the one case that needs it -- no node with 600Gi of local NVMe free. Blocked on #280: the namespace cannot currently create these volumes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This needs
nvme-ephemeralgeneric ephemeral volumes to work in the data-loadingnamespace, and they currently do not — see #280. Merging before that is sorted out
leaves a loading pod that cannot be scheduled.
Stacked on
speed-up-loads(#278), so the diff here is only the NVMe change. Once#278 merges and its branch is deleted, GitHub retargets this to
main.What this does
Moves the loading Solr index onto a node-local NVMe ephemeral volume, declared inline
in
nameres-loading.k8s.yaml, instead of thenameres-loading-solrPVC on networkstorage.
After the parallelisation work in #278, a load is bounded by its two disk-bound
phases: merging, and the final
optimize=true, which reads ~130G and writes ~130G ina single forced merge. This is the volume that absorbs all of it, so its throughput
decides how long a load takes — more than anything left to tune in the Makefile.
Only the index moves, because the two volumes want opposite things:
/var/solrbenefits most from being fast, and is pure intermediate state. Oncethe tarball exists the index is worthless, and losing it with the pod costs only the
time to rebuild it from the synonym files on the other volume.
data/holds the ~130G download and the finished backup tarball — the onlything the whole exercise produces — and stays persistent.
So deleting the pod throws away the fast, rebuildable half and keeps the slow,
irreplaceable half.
fsGroup: 1000matches the image'snruuser; without it the volume arrivesroot-owned and Solr cannot write to it.
Notes
nameres-loading-solr.k8s.yamlis deleted, not kept as a fallback. A manifestnobody applies is a manifest nobody maintains, and it would drift out of step with
the pod it pairs with. The README documents the one-command recovery from git
history for the only case that needs it: no node with 600Gi of local NVMe free.
speed-up-loadsfor an index well above the ~127Gimeasured for Babel 2025nov4, since releases grow and the sizing has to hold for the
release being loaded.
optimize=truewrites the new segment before dropping the oldones, so peak is 2–3x the finished index, and running out of room happens at the
last step of a multi-hour load.
What to check once #280 lands
128Gi. If not, lower the request, but not below roughly 2x the finished index.
the Solr volume still flat-tops, the claim did not bind to an NVMe class — check
with
kubectl get pvcwhile the pod runs.