Improve NameRes documentation: Translator Guide, Babel data guide, and doc drift fixes#262
Draft
gaurav wants to merge 50 commits into
Draft
Improve NameRes documentation: Translator Guide, Babel data guide, and doc drift fixes#262gaurav wants to merge 50 commits into
gaurav wants to merge 50 commits into
Conversation
Creates documentation/TranslatorGuide.md with Translator-specific guidance: what to do when lookup results are unexpected (highlighting, type/prefix/taxa filters, autocomplete), when to use /synonyms vs. NodeNorm, and performance tips (bulk-lookup, filtering, caching). Updates README.md with a Colab badge link to the notebook and a structured Getting Started / Documentation section layout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix typo and improve conflation paragraph in openapi.yml; update Babel Conflation.md link from master to main - Add conflations list and conflation_url to /status response, driven by a CONFLATIONS env var (default: GeneProtein,DrugChemical) so each deployment can self-describe - Fix incomplete sentence and improve conflation section in API.md; add new /status fields to example response - Add /status to the quick decision table in TranslatorGuide.md Closes #202 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Translator-facing documentation and makes deployment conflation settings more discoverable, while reorganizing top-level docs entry points for NameResolution.
Changes:
- Adds
documentation/TranslatorGuide.mdwith Translator-specific troubleshooting,/synonymsvs NodeNorm guidance, and performance tips. - Updates
README.mdto add a Colab badge and clearer “Getting started” / “Documentation” sections. - Enhances
/statusoutput (and related docs) to include active conflations and a conflation documentation URL.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds a Getting Started section and Colab badge link; reorganizes documentation links. |
| documentation/TranslatorGuide.md | New Translator-focused usage guide with troubleshooting and performance guidance. |
| documentation/API.md | Clarifies conflation behavior and documents new /status fields (conflations, conflation_url). |
| api/server.py | Extends /status response with conflations and conflation_url (configurable via env). |
| api/resources/openapi.yml | Updates API description to reference conflation docs on main and points to /status for active conflations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| conflation enabled (to match the conflation used by NameRes): | ||
|
|
||
| ``` | ||
| GET https://nodenormalization-sri.renci.org/get_normalized_nodes?curie=UniProtKB:A0A0S2Z3B5&conflate=true&drug_chemical=true |
… guard The Solr schema was defined twice -- in setup_solr.sh at load time and again in the Helm restore script at serve time -- and had already drifted. Make the checked-in configset (configsets/name_lookup/conf) the single source of truth, generated from a real Solr so behaviour is unchanged, with the malformed `types` field fixed, a larger index buffer, autoSoftCommit off, and a grown queryResultCache (closes #266). Rewrite setup-and-load-solr.sh to stream files to Solr in parallel with a single deferred commit (no per-file commit, no sleeps), and guard against dropped data by counting input documents before the load and comparing against Solr's count afterward, with curl --fail on every upload. The core is now created from the configset (see the Makefile / CI), so this script no longer sets up the schema and setup_solr.sh is deleted. Convert the test fixture to JSON-lines to match the Babel production format so CI exercises the line-counting guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run Solr in standalone mode everywhere (no ZooKeeper for a one-node job): drop -cloud from the loader Dockerfile and -DzkRun from docker-compose and CI. The Makefile now creates the core from the checked-in configset, loads in parallel, optimizes the index before export (closes #256), and tars the whole core -- config, schema and index -- into a self-contained snapshot.backup.tar.gz. Restoring is now just "untar into the Solr home and start Solr", so the old replication-backup targets and the solr-restore/ helper are removed. CI creates the core from the configset and runs the parallel loader end to end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite data-loading/README.md around the self-contained standalone-core flow, with an "Options considered" section explaining why we chose it over the status quo and PR #249, and the list of issues it closes. Update the test-run commands in CLAUDE.md to create the core from the configset first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
/status looked for a core called name_lookup_shard1_replica_n1, which is the name SolrCloud gives it. In standalone mode the core is just name_lookup, so /status reported "Expected core not found." with no document counts -- and because the endpoint still answers 200, the Helm probes stayed green while it did. tests/test_status.py caught this. The core name now comes from SOLR_CORE (default name_lookup) and is used for the query URLs too, so there is one place to change it. If that name is not present but Solr reports exactly one core, we report on that one, so backups built by the old cloud pipeline keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…names Three problems with the guard, in rough order of how much they would hurt: - It compared the input line count against Solr's *total* document count. Every load assigns fresh UUIDs, so loading is additive: a second load against a non-empty core would double the data and then blame the count. Compare the increase instead, and say so when the core is not empty. - The wait for Solr was unbounded, so a Solr that never starts hung the build (six hours in CI) instead of failing it. - The glob was expanded with default IFS, so a path containing a space split into two non-existent files, which nullglob then dropped -- the count would quietly come up short. Empty IFS during expansion fixes it. Blank lines are no longer counted as documents, since Solr ignores them. Also corrected a comment: --fail does not catch malformed JSON. Solr answers 200 and indexes nothing for a file of plain text; only the count notices. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
data/solr.pid is now an order-only prerequisite of core.done and backup.done. Optimizing needs a running Solr, and the stamp files outlive a restarted container while the server does not, so `make data/backup.done` after a restart used to curl into nothing. Order-only, because a restarted Solr must not make the core look stale: re-running `solr create` on an existing core is an error. The tarball is compressed with pigz when it is available (added to the image). A 130 GB Lucene index is already largely compressed and gzip is single-threaded, so this was hours of CPU for a few percent. Dropped tar -v, which wrote a line per index file into the build log, and added pipefail so a tar failure is not hidden by a successful gzip. Uncompressing now uses find -exec, so rerunning the download step after a partial run is a no-op rather than a "no such file" failure, and the wait for the PID file sleeps and gives up rather than spinning a CPU forever. Solr 9 wants Java 11 at a minimum but ships and tests on 17 (the official 9.10 image runs 17), so the loader image uses 17 as well. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
configoverlay.json carried one setting, update.autoCreateFields=false. That file is written by Solr's Config API, so shipping it in the configset invites exactly the drift this configset exists to prevent. Set the default on the add-unknown-fields chain in solrconfig.xml instead -- same behaviour, verified by posting a document with an unknown field and getting a 400. Also dropped size/initialSize from queryResultCache: as the comment two lines above it in the same file says, both are ignored once maxRamMB is set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The guard is what stands between us and a silently half-loaded 130 GB index, and the backup is the actual product of data-loading/, but neither was tested anywhere automated. Two steps, a few seconds each on the 89-document test core: - the loader must exit non-zero on a file that does not load - tar the core, drop it into a fresh Solr, and serve it: 89 documents with no collection creation and no schema setup The wait for Solr is also bounded now, and dumps the container log when it gives up, rather than hanging the job until the six-hour timeout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pin by role. The builder has to name an exact release tarball, so SOLR_VERSION stays at 9.10.0, and it is the floor for anything that serves the backup. The servers -- docker-compose, CI, the Helm chart -- now track 9.10, so they pick up patch releases (any 9.10.x reads a 9.10.x index) and stay at or above the builder without anyone remembering to bump them. Deployment.md still told people to run solr-restore/restore.sh, which this branch deletes. Restoring is now "extract the backup into ./data/solr and start Solr". Fixed the mount description while there: ./data/solr is mounted at /var/solr/data, the Solr home, which is why the core is found at all. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…heap Two settings were being chosen from numbers that do not describe the container they run in. Parallelism came from getconf _NPROCESSORS_ONLN, and pigz defaulted to one thread per core. Both report the *node's* CPU count inside a container: in the loading pod, 8 CPUs' worth of quota on a 64-core node meant 64 parallel uploads and 64 compression threads, all throttled. available-cpus.sh reads the cgroup quota instead (v2 and v1, falling back to the node count when unlimited), and both the loader and pigz now use it, so raising the pod's cpu limit is picked up without a second setting to remember. SOLR_MEM was 220G, which `solr -m` turns into -Xms220G -Xmx220G -- committed, not a ceiling. A bulk load does not want that: Lucene buffers into ramBufferSizeMB (512 MB) and streams merges through the OS page cache, so the heap was taking memory away from the only thing that would have used it. It is now 31G, which also stays under the compressed-oops threshold, and the serving side already measured 11-13Gi of RSS against 111Gi of page cache for the same index. The tunables use ?= so a pod spec can override them from the environment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pod asked for 220G/256G of memory and 6-8 CPUs, which was the right shape for the old serial loader and the old 220G heap: almost all of the memory went to a heap that could not use it, while the parallel loader is short of the CPU it now knows how to use. It asks for 96Gi and 16 CPUs instead, with requests == limits so a multi-hour load is not evicted partway through. The PVC comments described the pipeline as it was in 2022-2023. The Solr volume needs 2-3x the finished index because optimize writes the new segment before dropping the old ones, and the data volume no longer stages an uncompressed ~119G copy of the backup, since it is tarred straight out of the Solr volume. Added a README covering what actually costs time in a load (CPU while indexing, disk while merging and optimizing), how each resource should be sized, which Grafana panels answer "would more of this help", and what every knob does. Also replaced the keep-alive loop with sleep infinity and said why the pod overrides the image's entrypoint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
data/core.done and data/setup.done recorded facts about the Solr volume ("the
core exists", "it has been loaded") but lived on the data volume. That was
harmless while both were persistent PVCs with the same lifetime. It stops being
harmless the moment the Solr volume becomes ephemeral: a recreated pod would
find an empty index and a pair of stamps swearing it was loaded, skip straight
to optimizing a core that no longer exists, and fail somewhere confusing.
The three stamps that describe the Solr volume now live on it, so they vanish
with it and make sees the truth. data/synonyms/done and data/backup.done stay
where they are, because they describe the data volume.
Logs move the other way, from the Solr volume to data/logs, since they are most
useful exactly when the run has died and the Solr volume has gone with it. That
includes Solr's own logs, via SOLR_LOGS_DIR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The index is the volume that wants speed -- it takes the whole write load of the indexing run and then the read-and-write of the optimize -- and it is also the one volume we can afford to lose, since once the tarball exists the index is worthless. That makes it an exact fit for nvme-ephemeral: created with the pod, destroyed with it. data/ stays on a persistent PVC. It holds the ~130G download and the backup tarball, which is the only thing the whole exercise produces; that does not belong on a volume that dies with its pod. So deleting the pod now throws away the fast, rebuildable half and keeps the slow, irreplaceable half. fsGroup 1000 matches the image's nru user, without which the volume arrives root-owned and Solr cannot write to it. The old Solr PVC is kept as a documented fallback for when no node has enough local NVMe, since 400Gi of local disk is a real scheduling constraint in a way that network storage is not. README.md covers the split, the fallback, and what survives a pod restart. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The namespace has roughly 90 CPUs and 1000Gi spare, so 16/96Gi was leaving most of the available speed unused for a job that runs once and finishes. Stopping at 32 rather than taking the lot: the request has to fit on a single node, alongside the NVMe volume, and namespace quota says nothing about whether any one node can do that. Raising it further is a matter of checking node capacity -- the loader picks up the new CPU count on its own. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The buffer is a budget shared across indexing threads, so what it sets in practice is how large a segment grows before it is flushed: roughly the budget divided by the number of concurrent uploads. At 512 MB and 32 parallel uploads that is ~16 MB a segment; at 2 GB it is ~64 MB, so four times fewer segments for the merges to chew through afterwards. Safe at both ends. It is a budget rather than a reservation, so serving pods (which never index) allocate nothing from it, and 2 GB is a small fraction of the 31G load heap and well under Lucene's 1945 MB per-thread hard limit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every run rebuilt the image from nothing: apt upgrade, seven apt installs and a ~250MB Solr tarball fetched through an Apache mirror redirect. That is the difference between the 2-3 minute runs in the history and the 25 minute ones -- any Dockerfile edit near the top invalidates everything after it, and there was no cache to fall back on. cache-from/cache-to with the GitHub Actions backend fixes that; mode=max keeps intermediate layers, so editing the COPY lines at the bottom of the Dockerfile no longer re-downloads Solr. Buildx is already set up with the docker-container driver, which the gha backend needs. Also dropped the get_version step: it used ::set-output, which is deprecated, to compute a value nothing referenced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The publish workflow tags `latest` only on a published release, so `latest` is still the pre-PR image: old Makefile, old loader script, 220G heap, no available-cpus.sh. All of those live inside the image rather than in this repo's working copy, so a correct pod spec would have run the old pipeline anyway. pr-278 is what the pull_request build publishes. Revert to latest after release. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Makefile, the loader and the configset are all baked into the data-loading image, so the `image:` tag decides which version of the pipeline runs and editing a checkout does nothing. That is easy to miss, and the cost of missing it is an eight-hour load of the wrong thing, so the README now says it and gives three commands that tell you what you actually have. Also notes the escape hatch: the configset is only read when the core is created, so editing solrconfig.xml inside the pod before `make all` still counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The configset was captured wholesale from a running Solr, so it carried the whole of Solr's _default: 44 files, 70 field types and 69 dynamic fields, where NameRes uses 6 field types and 13 fields. 42 of those files were language stopword and contraction lists that could never have done anything here -- Babel synonyms carry no language, and neither of our field types stems, removes stopwords or applies synonyms, so nothing referenced them but the stock text_* types we never query. That left most of the file pinned without having been chosen, with no story for what to do when Solr moves. Both files are now written for NameRes instead of copied: 1593 lines to 218, 44 files to 2, and every element carries a comment saying why it is there. What remains is our data model plus a handful of tuning decisions, which are ours to own whatever Solr does. Upgrades are now "bump the image and let CI create a core from this", which is a check we already run. Two behaviours tightened along the way: - The schema is immutable: schemaFactory declares mutable=false, so the Schema API refuses writes and a live core cannot drift from this file. Undeclared, Solr defaults to a mutable managed schema, which is the trap that made us delete configoverlay.json. The one thing that touches a live index is the Helm blocklist, which deletes by query and needs no schema change. - An unexpected Babel key is a hard error rather than an invented field, since there are no dynamic fields and no schemaless chain. Kept deliberately, both easy to delete by accident: the named uuid update processor, which is what gives every document its id, and the /query handler, which the load guard and the CI roundtrip use to count documents. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The build fetched Solr through https://www.apache.org/dyn/closer.lua, which redirects to a randomly chosen Apache mirror. Mirrors range from fast to barely moving, and ADD-from-a-URL has no timeout, no retry and no resume, so a bad draw hung the build until the job gave up -- 25 minutes on one run, still going at 18 on another, against 2-3 minutes historically. Same Dockerfile every time; the only variable was which mirror answered. Nothing verified the download either. Copying /opt/solr out of the official image removes the whole failure mode: it arrives over a CDN, the daemon verifies it by digest, and it caches like any other layer. The JDK comes with it, so the openjdk-17-jre install goes too, and there is no 250MB tarball to extract. The Solr that builds a backup is now bit-for-bit the Solr that serves it. The stage is pinned to linux/amd64 to match the RENCI base image, which is published for amd64 only. Without that, a build on an arm64 machine resolves this stage to arm64 while the final stage falls back to amd64, and every java call fails with "No such file or directory" -- found the hard way. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Building the image locally and running `make all` inside it -- which nothing had done before, since CI exercises the loader against a stock Solr rather than through the Makefile -- turned up two failures, both fatal to a real load. `solr create -d configsets/name_lookup` cannot work: a *relative* -d is resolved against Solr's own server/solr/configsets directory, not the working directory, so core creation died with "Can't find resource 'solrconfig.xml'". CONFIGSET is now absolute via $(CURDIR). The start target waited for `solr status` to print a PID. In a container Solr reports "No Solr nodes are running" while serving happily on 8983 -- the stock solr image does the same, so this is not something about our image -- and the loop ran to its limit and failed. Nothing ever used that PID: stop-solr stops Solr by port, and the file is only a make stamp. It now waits for the admin endpoint to answer, which is the actual question being asked, and the stamp is renamed solr.started rather than solr.pid because it no longer holds a PID. Verified in the built image: make all loads 89 documents, make data/backup.done optimizes and writes a tarball through pigz, and that tarball restores into a stock Solr and serves all 89 with /status reporting ok. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The namespace cannot create nvme-ephemeral generic ephemeral volumes, so the loading pod could not be scheduled. Issue #280 asks for that; the NVMe version is preserved on nvme-ephemeral-loading and will come back as its own PR rather than blocking a load that is otherwise ready to run. Sized up from 400Gi to 600Gi while reverting. The 400Gi figure was 3x the ~127Gi index measured for Babel 2025nov4, but the Makefile now points at 2026jul22 and releases grow. optimize=true writes the new single segment before dropping the old ones, so peak usage is 2-3x the finished index, and at a 180Gi index 3x is 540Gi. Running out of space happens during the optimize, which is the last step of a multi-hour load, and takes the whole load with it -- cheap insurance. fsGroup stays, because a freshly provisioned PVC is root-owned and Solr runs as nru, but it now uses fsGroupChangePolicy: OnRootMismatch. The default chowns every file on every mount, which on a volume holding a ~130G index is minutes of pod startup for no reason. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two that would have bitten in production: Backup ownership. Solr writes to the core it serves -- it takes a write lock and refuses to load the core if it cannot. The loading image runs as nru (1000) and every Solr that serves the backup runs as solr (8983), so a restored core was owned by a user that does not exist on the serving side. Verified against a real restore: without this the core fails with SolrException: /var/solr/data/name_lookup/data/index/write.lock at the end of a multi-hour restore. CI papered over it with a chown, and Deployment.md did not mention it at all. tar now stamps BACKUP_UID/GID onto every member, which fixes every restore path at once instead of asking each one to remember. The CI roundtrip now chowns the source core to 1000 first -- standing in for the loading image -- and drops the chown on the restore side, so it exercises the fix rather than hiding it. tlog is excluded while we are here: the optimize commits, so there is nothing to replay. Additive loads. Loading assigns fresh UUIDs, so re-running a load over the same files indexes everything twice, and the document-count guard cannot see it because the delta is still exactly right. That is the one way the loader can hand back a corrupt index while reporting success, so a non-empty core is now a hard error (LOAD_APPEND=1 to override). It fires in the two cases that happen: a load re-run after one died partway through, and make re-running the load because a stamp file went missing. Both READMEs claimed a dead pod "resumes exactly where it left off"; resumption is per step and the load is one step, so they now say to clear the core. Smaller fixes: - /status used core['startTime'] where every field around it uses .get(). - `make clean` ran `mkdir data` on a directory that still existed. - wget now accepts only *.txt/*.txt.gz. Everything in data/synonyms is loaded, so a stray robots.txt became a file the loader tried to index -- caught by the count guard, but only after a multi-hour load. - The split loop swallowed a failed split that was not the last one. - The loading image's entrypoint still asked for a 64G heap. Doc drift: ramBufferSizeMB was described as 512 MB (it is 2 GB), SOLR_VERSION as 9.10.0 (it is 9.10.1), LOAD_PARALLELISM as defaulting to the CPU count (it is the cgroup limit, which is the entire point), and the useColdSearcher comment promised warming that nothing performs. Checked against a real Solr 9.10: core creates from the configset, 89 docs load, the malformed-file and non-empty-core guards both abort, the backup round-trips into a fresh Solr with no chown and serves queries, and pytest is 28/28.
The measured index for Babel 2026jul22 (the release SYNONYMS_URL loads) is ~111Gi -- smaller than 2025nov4's ~127Gi, because it has fewer proteins. That falsifies the "newer releases are bigger" claim the 600Gi sizing rested on, so reword both places to note release size moves both ways and the headroom is for a possibly-larger future release. No size change: 600Gi stays comfortably above 2-3x either figure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
maxRamMB is a CaffeineCache feature. CaffeineCache is Solr's default cache today, so this is a no-op now, but if a later release changed that default to an impl without RAM accounting, maxRamMB would be silently ignored and the cache would grow unbounded. Pin the class so the configset's intent survives a Solr upgrade. Verified the core still creates from the configset on solr:9.10. Addresses a Copilot review comment on #278. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The load guard and the CI backup-roundtrip check both scrape numFound out of /query with a regex that assumed "numFound":<digits> with no space. Solr 9.10 emits exactly that today (verified: "numFound":1), so this is defensive, but /query runs with indent=true and a JSON-writer change to "numFound": 1 would turn the guard into a false failure. Allow optional whitespace after the colon; head -1 still skips the sibling numFoundExact key. Addresses two Copilot review comments on #278. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-setting checks (ls available-cpus.sh, grep ramBufferSizeMB/SOLR_MEM) were a crutch for the unversioned transition image; with released version tags the image tag itself identifies the pipeline. Drop them, keep the durable facts (pipeline is baked into the image, `latest` is the previous pipeline until a release is cut, files are editable in the pod before make), and point at the org.opencontainers.image.revision label as the reliable tag-to-commit check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…278) ## What & why Building a NameRes instance loads ~130 GB of Babel synonyms into Solr and ships a backup a worker restores. The old pipeline had three problems: a slow serial load (per-file hard commit + `sleep 60`), a Solr schema **defined twice** (loader *and* the Helm restore script) that had already drifted, and ZooKeeper running for what is a one-shard, one-node job. This PR reworks the whole flow around a **self-contained standalone core**: the backup is a whole Solr core — config + schema + index — so restoring it is just "untar into the Solr home and start Solr". No ZooKeeper, no collection creation, no restore-time schema setup. This supersedes #249. ## Key changes - **Single source of truth for the schema:** new checked-in configset at `data-loading/configsets/name_lookup/conf` — two files, ~220 lines, **written for NameRes rather than copied from Solr's `_default`**. Field types are carried over verbatim so analysis behaviour is unchanged, the malformed `types` field is fixed, the index buffer is larger, `autoSoftCommit` is off and `queryResultCache` is grown. Everything in it is a decision we made, so there is nothing to reconcile against upstream on a Solr upgrade; the copy it replaced was 44 files, 70 field types and 69 dynamic fields, of which NameRes uses 6 field types and 13 fields (42 of the files were language stopword lists, which Babel synonyms can never use). The schema is now **immutable** (`mutable=false`), and schemaless mode is off in `solrconfig.xml` rather than in `configoverlay.json`, which Solr's Config API rewrites — so no file in the configset is one Solr can rewrite behind us. An unexpected Babel key is now a hard `unknown field` error instead of a silently invented field. - **Fast, guarded load:** `setup-and-load-solr.sh` streams files to Solr in parallel with a single deferred commit (no per-file commit, no sleeps). Because a parallel load of a huge dataset is scary, it **counts input documents before loading and compares the increase in Solr's count afterward**, with `curl --fail` on every upload — a drop or a bad file aborts with a non-zero exit. The count is the load-bearing half: Solr answers **200 and indexes nothing** for a file of plain text, so `--fail` alone would let it through. - **Standalone everywhere:** loader `Dockerfile`, `docker-compose.yml`, and CI drop cloud mode; Solr bumped to 9.10. The Makefile creates the core from the configset, optimizes the index, and tars the core into `snapshot.backup.tar.gz` (with `pigz` when available — single-threaded gzip on an already-compressed 130 GB index was hours for a few percent). - **`/status` fixed for standalone:** it looked up the SolrCloud core name `name_lookup_shard1_replica_n1`, so under standalone it reported `Expected core not found.` with no document counts — quietly, since the endpoint still answers 200 and the Helm probes only check that. The core now comes from `SOLR_CORE` (default `name_lookup`), with a fallback to the single core present so old cloud-built backups keep working. - **Version pinning by role:** the builder names an exact release tarball (`SOLR_VERSION=9.10.0`) and is the floor for anything serving the backup; the servers (docker-compose, CI, Helm chart) track the `9.10` line, so they pick up patch releases and stay at or above the builder automatically. - **Deleted:** `setup_solr.sh`, `solr-restore/` and `configoverlay.json` (all obsolete). - **Load sized to the container, not the node:** parallelism and `pigz` now come from the cgroup CPU quota rather than `nproc`, which inside a pod reports the node's cores — 8 CPUs of quota on a 64-core node used to mean 64 throttled uploads. The loading heap drops from 220G (`solr -m` commits both `-Xms` and `-Xmx`) to 31G, since indexing wants page cache rather than heap, and `ramBufferSizeMB` rises to 2G — the buffer is a budget shared across indexing threads, so at 32 parallel uploads that is ~64 MB per segment instead of ~16 MB, and four times fewer segments to merge. - **Kubernetes loading job reworked:** the pod goes to 32 CPUs / 128Gi, and `/var/solr` is sized at 600Gi (the optimize writes the new segment before dropping the old ones, so it peaks at ~2x the finished index). Moving the index onto a node-local **NVMe ephemeral volume** is the change that would actually speed this up — it takes the whole write load of the indexing run and the optimize, and it is the one volume we could afford to lose — but it is blocked on #280 (the namespace cannot create `nvme-ephemeral` volumes), so **both volumes are persistent PVCs today** and both need deleting by hand. Makefile stamp files now live on the volume whose state they describe, so they disappear with the index if that volume is ever made ephemeral. New [`data-loading/kubernetes/README.md`](data-loading/kubernetes/README.md) covers the volume split, sizing, the Grafana panels that answer “would more of this help”, and every knob. - **CI image builds cached:** the data-loading image was rebuilt from scratch every run — `apt upgrade`, seven `apt install`s and a ~250MB Solr tarball — taking ~25 minutes. It now uses the GitHub Actions cache. - **Helm chart** (in `translator-devops`, edited copy under `data/name-lookup/`): standalone Solr, idempotent `download.sh`, `restore.sh` gutted to blocklist-only, heap lowered for page cache. ## Validation CI now runs the whole thing end to end, against a Solr loaded by the new pipeline: - **`pytest`: 28/28 pass** (including `test_status`, which the standalone switch initially broke). - **Configset verified against a real Solr:** core creates from the two files, load succeeds (so the `uuid` processor and `/query` survived the trim), an unknown field is rejected with a 400, a Schema API write is refused with `schema is not editable`, and highlighting still returns spans. - **The loader must abort** on a file that does not load — the guard is what stands between us and a silently half-loaded index, so it is tested rather than assumed. - **The loader must refuse a core that already has documents.** Loading assigns fresh UUIDs, so a re-run over the same files doubles the index, and the count guard cannot see it — the delta is still exactly right. This is the one way the loader can return a corrupt index while reporting success, so it is a hard error (`LOAD_APPEND=1` to override). - **Full backup roundtrip in CI:** tar the core → drop it into a fresh Solr → 89 documents served with **zero schema setup and no `chown`**. The roundtrip deliberately chowns the source core to uid 1000 first, standing in for the loading image, so it proves the tarball's ownership stamping rather than hiding it behind a fixup. Also checked by hand: an unexpected field is rejected with a 400 (schemaless really is off), filenames with spaces and blank lines are counted correctly, and `make data/backup.done` on a restarted container restarts Solr instead of failing — without re-running `solr create` on a core that already exists. Also confirmed the failure the ownership fix prevents: a backup tarred without it restores into a core that will not load, with `SolrException: /var/solr/data/name_lookup/data/index/write.lock` — Solr takes a write lock on the core it serves, and the loading image's uid (1000) is not the serving image's (8983). ## Issues Closes #238, closes #185, closes #256, closes #266. Addresses helxplatform/translator-devops#609. Progresses #265 (heap knob + smaller index; final heap/GC tuning and #272 still need load testing). Out of scope: #267, running natively on SLURM/HPC. ## Pre-merge cleanup — done - [x] **Deleted the `pull_request:` trigger** from `.github/workflows/release-nameres-loading.yml` (c563ded). It only existed to publish a `pr-278` image so a load could run against this branch without merging. - [x] **Reverted `nameres-loading.k8s.yaml` to `image: …:latest`** (b6458cc). Caveat carried over: the publish workflow only tags `latest` on a published *release*, so `latest` becomes this pipeline only once a release is cut — run the next load after this is merged and released. ## The load that already ran — done A full Babel 2026jul22 load (~111Gi index, ~331M docs) was built with the `pr-278` image and is now restored and serving on the cluster, so the workarounds this section used to list are spent. That run predated the review fixes, so the backup's ownership was stamped by hand — the merged Makefile now does it automatically with `tar --owner`. The load ran to completion (no partial-load restart, no stray files to skip). Two problems surfaced during it, both now fixed: the backup's uid ownership (in this branch) and the serving chart's Solr image tag being read as `9.1` (see the reviewer note). ## Note for reviewers The `data/name-lookup/` Helm chart is gitignored here (it's a working copy); its edits live in `translator-devops`. One caveat found while bringing the load up: the Solr image tag must be **quoted** (`tag: "9.10"`) — unquoted, YAML parses `9.10` as the float `9.1` and pulls `solr:9.1`, which cannot init a 9.10-built core (it fails with `_version_ … not searchable`). That fix still needs applying in `translator-devops`. Point `dataUrl` at a backup built by the new loader — old backups won't restore under the new flow. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The links in documentation/Deployment.md are written as if the file sat at the repository root, so ./docker-compose.yml, ./data-loading/README.md and ./Dockerfile all 404 from inside documentation/. documentation/Scoring.md pointed at a Babel README anchor (?tab=readme-ov-file#how-does-babel-choose-a-preferred-label-for-a-clique) that no longer exists: that section moved into Babel's docs/Understanding.md. Point at the file rather than at an anchor within it, so a future reorganization of that document does not break the link again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NameRes inherits a lot of behaviour from Babel that is visible through this
API but explained nowhere here: which Babel synonym files an instance is
actually built from, why conflation cannot be turned off per query, what the
leftover-UMLS singletons in the results are, why `label` may not be the label
of the CURIE you looked up, why `synonyms[0]` is not the best synonym, what
`clique_identifier_count` really counts, why an empty `taxa` is not the same
as "not taxon-specific", and why there are no descriptions.
Collect all of that in one document, along with where to file a bug (concept
wrong -> Babel, service wrong -> NameRes) and the fact that an index is a
snapshot of a single Babel release rather than live output.
Babel.md is now the only file in this repository that links to a specific
file inside the Babel repository, so a reorganization there is a one-file fix
here rather than a hunt through five. Bare links to the Babel repository root
stay where they read naturally -- those cannot rot.
Also fixes two sentences that were broken where they sat: the conflation
section of API.md ended mid-example ('if you search for ""'), and the OpenAPI
description read "Note that the returned by this service".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Six of the OpenAPI endpoint descriptions linked into documentation/API.md with anchors that do not exist. Five used "#Conflation" where GitHub generates "#conflation", and GET /synonyms pointed at "#bulk-lookup". - documentation/Deployment.md's configuration list omitted BABEL_VERSION, BABEL_VERSION_URL, BIOLINK_MODEL_TAG and LOGLEVEL. The first three are the ones an operator most needs: they are what /status reports about the data an instance is serving, and there was no way to learn they existed. - CLAUDE.md listed two endpoints that do not exist (/lookup-curies, and /reverse-lookup rather than the actual deprecated /reverse_lookup), named api/resources/openapi.yml with a leading dot, and pinned a line count for api/server.py that no longer matched. The endpoint list now matches the five paths FastAPI actually generates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Verified against a local Solr loaded with tests/data/test-synonyms.json: - "No taxa" is not represented the same way by both endpoints. /lookup builds a LookupResult and so always emits taxa, defaulting to []; /synonyms returns the raw Solr document, which omits the field. Say so, and point at Babel's taxon_specific flag for code that wants the distinction as a boolean. - The /synonyms return list omitted taxon_specific and shortest_name_length, both of which are in every document. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The response-shape differences between the two endpoints are documented as current behaviour rather than as intent, so point at the issue that tracks removing them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The remote branch and this local work were two independent passes over the same documentation, forked from different points. Both restructured README's doc list, both rewrote API.md's conflation section, and both independently fixed the same two broken sentences (API.md's 'if you search for ""' and the OpenAPI 'Note that the returned by this service'). Resolved in favour of combining rather than choosing: - README.md keeps the remote's Getting started / Documentation structure and the Colab badge, with Babel.md added to the list and the reporting-a-problem section kept. - API.md's conflation section keeps the remote's bold headings and the pointer to /status, but describes GeneProtein conflation using the NCBIGene:1756 document already shown above it rather than asserting what a live search for "dystrophin" returns, which is a ranking claim we can't make from the docs. - The OpenAPI description states the conflations, points at /status for the active set, and links to documentation/Babel.md. Two Babel links arriving on the remote side used /blob/main/; Babel's default branch is master, which is what every other Babel link in this repository uses. Both resolve today, but they are normalized to master here. Also documents CONFLATIONS, the new environment variable behind the /status field, alongside the other /status variables in Deployment.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Babel, NameResolution and NodeNormalization all use main as their default branch; Babel's was renamed from master relatively recently. NameResolution has no master branch at all. Links to /blob/master/ still resolve today only because GitHub keeps a redirect after a default-branch rename. That redirect is exactly the kind of thing that disappears quietly, which makes these the link rot this branch is supposed to be removing rather than an acceptable alias. This rewrites 24 links: the 12 pre-existing NameResolution self-links in api/server.py, the seven Babel links in documentation/Babel.md, the conflation_url reported by /status, the two Colab badge URLs, and the sample /status payloads in the docs. Reverses the normalization direction taken in the previous merge commit, which went to master on the strength of a stale origin/HEAD ref in a local Babel clone rather than the actual remote default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gaurav
changed the title
The conflations field, its CONFLATIONS override and the environment variables that /status reports were all untested — /status is the only record of which Babel data an instance is serving, so a silent regression there is invisible. Also records in CLAUDE.md why cross-repo links must use /blob/main/, and why a local clone's origin/HEAD is the wrong place to check a default branch: it is cached at clone time and does not follow a remote rename, which is how the previous commit came to normalize these links in the wrong direction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- api/apidocs.py returned `app.openapi_schema()`, calling a dict. The branch is unreachable on the happy path (server.py builds the schema once at import), but any second call raises TypeError. Covered by tests/test_apidocs.py, which fails against the old line. - tests/test_docs_links.py checks, offline, that relative links resolve, that heading anchors exist, that the API.md anchors used in the endpoint descriptions exist, and that no link uses /blob/master/. Four of the six broken links this branch fixed by hand would have been caught mechanically. It runs in the existing pytest CI job rather than needing a new workflow, and it makes no network requests -- a link check that fails when GitHub is slow is one people learn to ignore. It immediately found nine more /blob/master/ links in releases/, which also used the pre-rename TranslatorSRI org; those are now NCATSTranslator/main. The remaining TranslatorSRI PR and issue links in old release notes are left alone: an org-rename redirect is durable, and rewriting historical release notes is churn. - data-loading/README.md's "Options considered" and "Issues addressed" were an argument for a merged PR, not documentation. Replaced with the three decisions a future reader could undo by accident; the rest is in the history. - documentation/TranslatorGuide.md now links to Babel.md where it talks about missing concepts and about caching across Babel releases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example in documentation/API.md still showed nameres_version v1.5.1 while the OpenAPI spec is at 1.7.0. The accompanying test treats the two kinds of drift differently. A stale patch or minor version is cosmetic and only warns. A stale major version fails, because a major release is when someone should re-run this example against a real instance and confirm the endpoint still behaves as documented. The field names are compared strictly in both cases: that is what catches a field added to /status and never written down, which is how `conflations` came to be undocumented in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
Collaborator
Author
|
Some field evidence for the
So ITRB prod is serving an index from Nov 2024 out of a NameRes old enough to predate those Two other divergences worth knowing, neither of which this PR changes:
No action needed on this PR — recording it so it isn't rediscovered. |
NameResolution, Babel and NodeNormalization all moved from TranslatorSRI to NCATSTranslator. 77 links across the eight release-note files still named the old org. They resolve through GitHub's org-rename redirect, which is more durable than the branch-rename redirect cleaned up earlier, but they still point at an org that no longer owns the code. The new test is scoped to those three repositories by name rather than banning the TranslatorSRI string. TranslatorSRI/RENCI-Python-image (used by data-loading/Dockerfile), TranslatorSRI/babel-validation and TranslatorSRI/r3 really do still live under that org -- NCATSTranslator equivalents are 404s -- so a blanket ban would push someone to "fix" three correct links into broken ones. The comment says so, since that is the trap here. Closes #292. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation pass over NameRes, covering both what NameRes does itself and what it inherits from Babel.
Important
#259 is stacked on this branch and cannot merge until this does.
If you squash-merge this PR,
improve-documentation's commits will not exist inmain, andGitHub's auto-retarget of #259 will leave it showing duplicated content. In that case #259 needs
git rebase --onto main improve-documentation add-claude-skillbefore it is reviewable again —which is correct for a merge commit too, so it is safe to run either way. Ping me and I'll do it.
New documentation
documentation/TranslatorGuide.md— Translator-specific guidance: what to do when lookup results are unexpected (highlighting, type/prefix/taxa filters, autocomplete), when to use/synonymsvs. NodeNorm, and performance tips (bulk-lookup, filtering, caching).documentation/Babel.md— where the data comes from, and the Babel behaviour that is visible through this API but was explained nowhere here:synonyms/directory shipsGeneProteinConflatedandDrugChemicalConflatedinstead of per-typeGene/Protein/SmallMoleculefiles, plus Babel's leftover-umlscompendium. This is why conflation can't be turned off per query, and why single-identifierUMLS:results show up in searches.labelis the clique's preferred name and need not be the label of the CURIE you looked up.synonyms[0]is not the best synonym — Babel orders shortest-first except for conflated cliques, which is almost everything we load.clique_identifier_countmeasures how many sources cover a concept, not importance, and it multiplies the search score.taxameans "no source asserted one", which is whyonly_taxakeeps untaxoned results.It is also the only file in this repository that links to a specific file inside Babel, so a reorganization there is a one-file fix rather than a hunt.
README.md,API.md,Scoring.mdand the OpenAPI description link into it instead.API changes
/statusnow reports the conflations baked into the deployment:The set comes from a new
CONFLATIONSenvironment variable (defaultGeneProtein,DrugChemical) — it is not detected from the data, so it needs setting if an index is ever built with a different set.Fixes
documentation/Deployment.md, written as if the file sat at the repository root.Scoring.md, pointing at a README anchor that moved intodocs/Understanding.md.API.md#Conflation(GitHub generates#conflation) and/synonymspointing at#bulk-lookup.API.md's conflation section ended mid-example (if you search for ""), and the OpenAPI description read "Note that the returned by this service".BABEL_VERSION,BABEL_VERSION_URL,BIOLINK_MODEL_TAG,LOGLEVELand the newCONFLATIONSare what/statusreports about the data an instance is serving, and there was no way to learn they existed.CLAUDE.mddrift: two endpoints that don't exist (/lookup-curies, and/reverse-lookuprather than the real/reverse_lookup), a filename with a stray leading dot, and a stale line count.main. All three repos usemainas their default branch, and NameResolution has nomasterat all — those links resolved only via GitHub's post-rename redirect.Verification
Tests run against a local Solr 9.10.1 with the checked-in configset and
tests/data/test-synonyms.json(89 docs): 36 passed, up from 27 onmain. The documented/statusenvironment variables were confirmed to work end to end, and every cross-repo link was checked for a 200.Cleanups this branch turned up
api/apidocs.pyreturnedapp.openapi_schema()— calling a dict. Unreachable on the happy path,TypeErroron any second call. Fixed, with a test that fails against the old line.tests/test_docs_links.pychecks offline that relative links resolve, that heading anchors exist, that theAPI.mdanchors used in the endpoint descriptions exist, and that nothing uses/blob/master/. Four of the six broken links fixed by hand here would have been caught mechanically. It runs in the existing pytest job and makes no network requests. It immediately found nine more/blob/master/links inreleases/, which also used the pre-renameTranslatorSRIorg.data-loading/README.md's "Options considered" / "Issues addressed" were an argument for an already-merged PR rather than documentation. Replaced with the three decisions a future reader could undo by accident.documentation/TranslatorGuide.mdnow links toBabel.mdwhere it discusses missing concepts and caching across Babel releases.Follow-up
Filed #291 —
/lookupand/synonymsdescribe the same concept in two different shapes (label/preferred_name,synonyms/names,biolink:-prefixed types vs bare, absent vs emptytaxa, Solr internals leaking through/synonyms). Documented as current behaviour here with a pointer to the issue, so those caveats can be deleted rather than maintained once it lands.Closes #133