Optional dataset gate for presenter template injection (partial-appDb builds) - #131
Open
jbrestel wants to merge 2 commits into
Open
Optional dataset gate for presenter template injection (partial-appDb builds)#131jbrestel wants to merge 2 commits into
jbrestel wants to merge 2 commits into
Conversation
… execution that validation triggers for any DatasetPresenter not found in apidb.Datasource table. Earlier getContacts and getModelReferences were running unconditionally for every presenter in the presenters XML directory, regardless of whether that dataset was actually loaded into the specific database instance being processed, leading to error when Presenter xml has extra entries
datasetPresenters/*.xml is deliberately a superset of what any one instance has
loaded, but injection ignored that: it built a DatasetInjector for every presenter
it parsed, so a presenter whose dataset is absent failed the build. Seen building
PlasmoDB against the unidb_shu_a subset:
Error: A datasetInjector for class ...ClonedInsertEnds in DatasetPresenter null
is missing the required property sourceIdField
Five of the ten ClonedInsertEnds presenters have no ApiCommonDatasets declaration
on this branch, so they get no prop file and no sourceIdField.
Commit 1289792 added the same kind of gate to DatasetPresenterSetLoader, but that
keys on getFoundInDb() -- a fact from a different lifecycle phase (presenterWriteToDb)
that TemplatesInjector cannot have, since its only inputs are directories. That gate
stays; it covers the DB-aware phase. This covers the build phase.
The predicate becomes a LoadedDatasetSource, consulted once in
DatasetPresenterSet.createFromPresentersDir so every consumer -- dataset injectors,
contact validation, model references, the loader, anything added later -- inherits
it without knowing it exists. That is what the previous fix lacked, and why it
landed in only one of two paths.
Off by default (AllDatasetsLoaded): no database call, injection identical to before.
A developer opts in with a system property:
-Dpresenter.dataset.gate=on # appDb from model-config.xml
-Dpresenter.dataset.gate=jdbc:postgresql://host:5432/db # or name one explicitly
-Dpresenter.dataset.gate.always=a_RSRC,b_RSRC # inject these regardless
A property rather than an environment variable because the two build paths do not
share an environment and only a property reaches both: rebuilder discards the
environment on startup but parses --gusjvmopts afterwards, while wb inherits
GUSJVMOPTS from the site's etc/setenv. Both exec the same installed
presenterInjectTemplates, which now forwards GUSJVMOPTS to the JVM -- previously
that variable was set by rebuilder and consumed by nothing.
Matching mirrors the loader's "ds.NAME like ?" exactly, including '_' as a
single-character wildcard, so the two gates can never disagree about membership.
A presenter's datasetNamePattern is resolved the same way the loader resolves it,
and one match is enough since a pattern presenter fans out to many datasets.
Enabled but unable to read apidb.Datasource is a hard failure: degrading to
"inject everything" would reproduce the failure this exists to prevent. Skipped
presenters are always reported -- count to stderr, names to
gus_home/lib/wdk/presentersNotLoaded.txt, because a gate that drops presenters
silently turns a loud build failure into a website that builds green with searches
missing.
Verified on cedar against unidb_shu_a: off => byte-identical pre-change behaviour;
on => 220 datasets read, 3171 presenters skipped, rebuilder reaches SUCCESS and
deploys. 21 unit tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an optional gate to presenter template injection so a site with a partial appDb
can build. Off unless explicitly enabled; no behavior change for full-data or production
builds.
Why
datasetPresenters/*.xmlis deliberately a superset of what any one instance has loaded,and injection ignores that: it builds a
DatasetInjectorfor every presenter it parses.Harmless on a full-data site. On a dev instance whose appDb holds only a subset, it breaks
the build — presenters for absent datasets have no prop file, so injection fails on props
those datasets would have supplied, or the model later resolves a question whose generated
attribute never appeared. Typical symptoms, all naming a dataset that was never loaded:
A datasetInjector for class ... is missing the required property <prop>duringpresenterInjectTemplatesSummary attribute field [...] defined in question [...] is invalidduring model loadrelation "eda.attributevalue_s<hash>_..." does not existfrom an injected queryHow
Enabled with a JVM system property, forwarded via
GUSJVMOPTS:It skips presenters whose dataset (or
datasetNamePattern, matched with SQLLIKEsemantics) has no row in
apidb.Datasource— the workflow root graph's output, so gatingon it gates on what the workflow actually loaded without the build system needing to know
about
ApiCommonWorkflow. Credentials come from the site'smodel-config.xml, never thecommand line.
Deliberate design points worth reviewing:
reproduce the problem the gate exists to prevent.
$GUS_HOME/lib/wdk/presentersNotLoaded.txt— so a gate that drops too much is visiblerather than silent.
-Dpresenter.dataset.gate.always=<names>injects a presenter whosedata is absent. Only works for a dataset declared in ApiCommonDatasets (so
propertiesFromDatasetsgenerated its prop file) and merely not loaded.The first commit (Bindu's) is the narrower precursor: it stops
getContacts/getModelReferencesvalidation — and the injector execution that validation triggers —from running unconditionally for presenters absent from
apidb.Datasource.Tests
TestDatasetGateandTestLikePattern(~300 lines) cover the gating decision and theLIKE-pattern matching.Note for review
Master's
8d72fac3("Filter dataset injectors by project when a presenter declares morethan one") touches the same
addToDatasetInjectorSetmethod. The cherry-pick auto-mergedcleanly and both filters are present; they narrow the injector set independently, but
that method is the one place these two changes meet.
Provenance
Cherry-picked onto current
masterfromdnaseq-merge-experiments, where this workhappened to land. It is unrelated to the variants/DNA-seq work on that branch — hence a
separate PR.
mvn -pl DatasetPresenter -am compilepasses.Related PRs
The variants work spans four repos. These four merge together —
ApiCommonModel's record rename is a contract the other three depend on by name:ApiCommonModelRename the Variation record to Variant; add a precomputed SNV characteristics search ApiCommonModel#212 — record rename + precomputed SNV characteristics searchApiCommonWebServiceFix SNV density and dN/dS in the sample-set SNV search, and revive its test suite ApiCommonWebService#20 — density and dN/dS fixes, test-suite repairApiCommonWebsitePrefix HSSS result IDs with Variant_ instead of NGS_SNP. ApiCommonWebsite#310 — HSSS result-ID prefix (NGS_SNP.→Variant_)web-monorepoShow the Variant record id without the record-type prefix web-monorepo#1841 — record-heading override, filename tracking the renameCoupling worth knowing when sequencing the merges: #1841's customization is resolved by
recordClass.fullName, so without #212 it matches nothing and the override silently stops applying — no error, just the default heading back. #310's prefix and #20's separator together produce the ids #212's record class resolves; any two without the third leaves HSSS emitting ids the record class rejects.Related but independent, cherry-picked onto
master— merge before the four above:EbrcModelCommonOptional dataset gate for presenter template injection (partial-appDb builds) #131 — optional dataset gate for presenter template injection. Not part of the variants work; needed to build a site against a partial appDb.The release note originally in VEuPathDB/ApiCommonWebsite#310 has been deferred (reverted in
965009fdd) because itsdateattribute was an unresolvable placeholder. Recover the drafted text withgit show a039aeee9in ApiCommonWebsite and re-PR it once the release date is known.