Skip to content

Rename the Variation record to Variant; add a precomputed SNV characteristics search - #212

Open
jbrestel wants to merge 112 commits into
masterfrom
dnaseq-merge-experiments
Open

Rename the Variation record to Variant; add a precomputed SNV characteristics search#212
jbrestel wants to merge 112 commits into
masterfrom
dnaseq-merge-experiments

Conversation

@jbrestel

@jbrestel jbrestel commented Aug 8, 2026

Copy link
Copy Markdown
Member

Model-side of the SNV work. Must merge together with ApiCommonWebService PR #20 — that PR's plugin advertises a span_snp_density column and validateColumns rejects it unless this PR's wsColumn declaration is present.

Renames the Variation record to Variant

The record class was Variation while every source_id it holds reads Variant_<sequence>_<location>, and its own attributes already said "Variant Type" and "Variant Call Set". "Variation" is the phenomenon; a record is one concrete allele at a locus.

Renames the WDK layer only — record class, question set and its five searches, query sets, param set, category ontology nodes, and six model files — and sets displayName to Short Variant / Short Variants, naming what the record actually holds (SNVs and small indels).

Deliberately unchanged: urlName stays variation so existing record URLs resolve (the feature is not public yet, so this can move separately); variation_sample_meta{,_a,_b} and chromosomeOptionalForVariations are contracts with the HSSS plugins and renaming them breaks the searches at run time, not build time.

The four HSSS searches now lead with SNV — HSSS computes over single-nucleotide variants only and cannot return an indel however the record class is named. VariantBySourceId is not prefixed, because an ID lookup goes to the tuning table and does return both.

Adds GenesByVariantCharacteristics

A precomputed sibling to GenesByNgsSnps. That one runs HSSS over a user-selected sample set and takes minutes; this reads apidbtuning.GeneVariationSummary, returns in milliseconds, and reports the same numbers as the gene record page. Neither replaces the other — dropping the HSSS search would delete the sample-set comparison, and without this one the record page and the search disagree about the same gene with no explanation.

Filtering is one filterParam over sixteen statistics, not sixteen range params. The range-param version was built first and abandoned: a numberRangeParam cannot express "untouched", so every range applies on every run and any gene NULL in that column is excluded — and GeneVariationSummary NULLs its frequency statistics below their allele floors by design, which for tbruTREU927 is 100% of rows for three columns. Working around that needed a full-span escape clause with bounds hardcoded in two files, which then broke when NumberRangeParamHandler rounded a value onto the declared ceiling. The filterParam has none of it, and distributions come from the data rather than from hand-declared bounds.

Result columns reuse the transcript record's own attributes rather than new dynamic ones — gene_pi_n_pi_s there is gvs.pi_n_pi_s_snpeff, the same column the filter reads, so the filtered value and the displayed value agree by construction.

Corrects GenesByNgsSnps labels

Its statistics changed in PR #20, so the strings describing them changed with them: density is now genuinely CDS with the old gene-span value beside it, the ratio says it is site-normalized, "Nonsynonymous SNPs" → Missense SNVs (it never included stop-gained), "Non-coding SNPs" → Unclassified SNVs (class 0 means no product byte was available). All eight result columns carry a (sample group) qualifier, since a results table can hold them next to the whole-cohort columns from the gene record.

Also adds span_snp_density to both postCacheUpdateSql blocks, not just the wsColumn list — those blocks enumerate result columns explicitly and backfill sibling transcripts, so omitting it would leave siblings NULL while matched transcripts carried a value.

Verification

Built and exercised on a dev instance throughout. Category tree keeps all 59 nodes after the rename with no stale VariationRecordClass references; all five renamed searches resolve with expandParams; the new search's filter distributions match SQL (impact classes sum to all 5,579 pfal genes; piN/piS reports exactly the 2,370 genes that have a value).

Note on scope

This branch carries ~90 prior commits of dnaseq work authored before this session — CNV search ports, the GeneVariationSummary tuning table, gene-record variation wiring. They are on the branch but are not part of this change.

🤖 Generated with Claude Code


Related PRs

The variants work spans four repos. These four merge togetherApiCommonModel's record rename is a contract the other three depend on by name:

Coupling worth knowing when sequencing the merges: #1841's customization is resolved by recordClass.fullName, so without #212 it matches nothing and the override silently stops applying — no error, just the default heading back. #310's prefix and #20's separator together produce the ids #212's record class resolves; any two without the third leaves HSSS emitting ids the record class rejects.

Related but independent, cherry-picked onto mastermerge before the four above:

The release note originally in VEuPathDB/ApiCommonWebsite#310 has been deferred (reverted in 965009fdd) because its date attribute was an unresolvable placeholder. Recover the drafted text with git show a039aeee9 in ApiCommonWebsite and re-PR it once the release date is known.

jbrestel and others added 30 commits July 29, 2026 23:42
IsolatesHTS no longer describes how DNASeq datasets are processed on this branch,
and both of its methods fail on the new shape:

- injectTemplates() derived per-sample gbrowse/jbrowse databases from getSampleList(),
  which keys samples on organismAbbrev + datasetClassCategory + experimentName. The
  new dnaseqExperiment class carries an empty category while its samples span two
  ("Genetic variation" for SNPs, "Structural variation" for CNVs), so the key cannot
  match and getSampleList() throws "No sample names found for dataset ...".
- addModelReferences() registered SnpQuestions / SnpRecordClasses references. That XML
  is not in the compiled model and is to be superseded by the variation record.

Both bodies are commented out rather than deleted, and the class stays in place as a
no-op so the presenters naming it keep resolving. getPropertiesDeclaration() is left
alone: presenters still supply hasCNVData, and DatasetInjector.addPropValues validates
supplied props against the declaration.

Note the deeper issue for whoever writes the replacement: keying the experiment ->
sample lookup on a single datasetClassCategory cannot work for an experiment that
aggregates SNP and CNV samples, even once dnaseqExperiment declares a category.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Commit 643ddb4 commented out the CommunityCount intermediate table and the
communityCount column of OrganismAttributes in apiTuningManager.xml (it read from a
hardcoded usercomments_userdb_betan.mappedComment), but left every consumer in place,
so model load failed with:

  Database error while attempting to parse sqlQuery OrganismAttributes.organismAttrs:
  ERROR: column oa.communitycount does not exist

Consumers brought into line with the producer, following the snpcount/popsetcount
precedent already in these files:

- organismAttributeQueries.xml: the communitycount column declaration and its select
  expression are commented out.
- organismRecords.xml: the communitycount columnAttribute is commented out.
- organismQuestions.xml: communitycount removed from five summary attribute lists,
  which is mandatory since WDK resolves every attribute a summary names.

community_flag is kept, as a literal 'none' rather than derived from the missing
column. That is exactly what "CASE oa.communitycount WHEN 0 THEN 'none'" produced for
a zero count, so the organism page renders as it would with no comments, rather than
losing an attribute that organismRecords.xml:305 and its red-dot <img> still use.
Deriving it from NULL would have been worse than removing it: CASE NULL WHEN 0 falls
through to ELSE, so every organism would have advertised comments it does not have.

The individuals.txt ontology entry is left alone, matching how snpcount and
popsetcount were handled: the category tree tolerates entries for absent attributes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These two questions were written by hand in geneQuestions.xml while the attribute
their summaries name is generated by the presenter:

  summary="...,exprGraphAttrpfal3D7_microarrayExpression_Derisi_TimeSeries_RSRC_expr_graph"

Expression.java:136 builds that name and injects the textAttribute that defines it. So
a hand-written question depended on generated output, coupled only by a name string
with nothing enforcing that both exist. On an instance where this dataset is not
loaded, the presenter is skipped and the question survives, referencing an attribute
nobody generated:

  Summary attribute field [exprGraphAttrpfal3D7_..._expr_graph] defined in question
  [GeneQuestions.GenesByMicroarraypfal3D7_..._RSRC] is invalid

The questions are curated rather than generic -- they use the PFTimeSeries vocabulary
queries, an extra samples_fc_comp_generic param, and fold_change_chosen_display in
place of fold_change_chosen -- which is why the presenter sets hasMultipleSamples,
hasPercentileData and hasPageData to false and why they were hardcoded in the first
place. So they move to their own microarrayDeRisiTimeSeries.dst rather than being
folded into the generic expression templates, and MicroarrayDeRisiTimeSeries injects
them. Question and attribute now appear and disappear together.

Only three values are parameterised -- ${datasetName} for the question and graph
attribute names, ${includeProjects} and ${includeProjectsExcludeEuPathDB} for scoping,
both already set by Expression.java to precisely the literals that were hardcoded.
Display strings stay literal because they describe this experiment. addModelReferences
now derives the question names from getDatasetName() rather than spelling out the
dataset, so the references cannot drift from what is injected.

Not yet verified: that the generated question is equivalent to the deleted static XML.
Nothing is generated on an instance where this dataset is absent, so this branch can
only show that its absence is handled. Before this reaches a full-data site, run a
build with the dataset declared and diff the generated question against the static
version in this commit's parent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CompoundQuestions.CompoundsByFoldChange named one dataset's graph column in its summary:

  summary="...,metaboliteProfiles_LlinasMetabolites_RSRC_metabolite_graph"

That attribute is injected per metabolomics dataset by metabolomics.dst, so naming a
particular dataset's copy in a generic search has two consequences: model load fails
wherever that dataset is not loaded ("Summary attribute field [...] is invalid"), and
where it is loaded the search shows that one dataset's graph regardless of what the
user searched for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One row per variant locus (4,390,908 in unidb_shu_a), backing a new
"variation" WDK record that replaces the deprecated snp record.

Holds only columns that are derived, aggregated, or require a join.
Intrinsic per-locus facts -- variant_type, is_coding, all 22 snp_*/indel_*
allele columns, the strain counts -- are read directly from
apidb.VariationFeature, which is already one row per locus with a unique
source_id. That split keeps ~570MB of derived strings out of the tuning
table and gives every column one known source.

Both internalDependency elements are load-bearing: built before
TranscriptAttributes the gene aggregate is silently empty rather than an
error, and built before GenomicSeqAttributes the project_id/organism join
drops every row.

Notable derivations:
- gene_ids is a string_agg, not a lookup: 25,545 loci overlap more than
  one gene, so the multi-gene case is correct by construction.
- most_severe_impact and effect_summary are split per caller, because
  snpeff and product_call disagree on 19% of paired calls and that
  disagreement is content rather than noise.
- collapsed_allele renders both classes for a MIXED locus (e.g.
  "A>C; A>AC"), which a single major/minor pair would have destroyed.

The SELECT was verified against unidb_shu_a before being committed:
4,390,908 rows, 100% join coverage against GenomicSeqAttributes, and the
expected values at three loci covering SNV, INDEL, and MIXED.

Design: docs/superpowers/specs/2026-07-30-variation-record-design.md and
plans/2026-07-30-variation-record.md in agentic-veupath-dev.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record class, alias query, and the tuning-backed identity/location
attributes. Builds green and loads by ID; 13 attributes registered.

sequence_source_id and location are exposed as first-class non-internal
attributes rather than being parsed back out of source_id, because they
are the coordinate a future VCF/tabix lookup and an EDA sample join both
key on.

Two things differ from the written plan, both discovered by building it:

- The snp imports in apiCommonModel.xml sit inside an "UNCOMMENT WHEN
  SNPS are AVAILABLE" comment block, so inserting after them would have
  silently disabled the variation record. Imports are placed after that
  block instead.
- variationTableQueries.xml is not created yet. WDK's RELAX NG schema
  requires a querySet to contain at least one query, so the intended
  empty-but-valid placeholder fails validation. The file arrives with its
  first real query.

Temporarily reads jbrestel.VariationAttributes; see Task 14 of the plan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First attribute query reading apidb.VariationFeature directly, per the
sourcing rule: intrinsic per-locus facts come from the base table, only
derived/joined columns come from the tuning table.
Both allele classes are preserved rather than collapsed: 129,850 MIXED
loci populate snp_* and indel_* simultaneously and describe the same
strains two ways, so a single major/minor pair would silently lose half
the data.

HGVS attributes carry help text explaining that the major-allele HGVS is
empty whenever the major allele equals the reference.
Grouped together deliberately: these aggregates are what the future
per-strain VCF-backed table will detail, so it lands beside them.
…lumns

Gene linkage is an aggregate rather than a lookup, so the 25,545
multi-gene loci are correct by construction instead of by a later fix.

Effect rollups stay split by caller: snpeff and product_call disagree on
19% of paired calls, 62% of which are product_call's strain-aware
downstream_frameshift.

Collapsed allele and MAF columns exist only because WDK tables render on
record pages, so results pages need pre-aggregated values.
Review of the Task 8 commit found two issues.

most_severe_impact_snpeff and most_severe_impact_product_call hold text,
so WDK sorted them alphabetically (HIGH, LOW, MODERATE, MODIFIER) --
contradicting the severity ranking their own help text advertises. Adds
computed rank columns and points sortingColumn at them, entirely within
the WDK query so no tuning table change or rebuild is needed.

Also completes collapsed_minor_allele_frequency's help text, which named
itself derived but omitted the pointer to the canonical SNP Alleles and
Indel Alleles sections that the spec requires and collapsed_allele has.
Overview shows both allele sections so a MIXED locus reads honestly.
Summary columns use the collapsed allele and MAF, since results pages
cannot render tables.
One row per transcript and observed codon. This is where gene strand
lives, since strand is only unambiguous per transcript once a locus can
overlap two genes.
The help said "Number of strains observed with this codon". It is not
per-codon: it is per amino-acid PRODUCT, repeated across every codon row
sharing that product.

At Variant_Pf3D7_01_v3_29514 four synonymous V codons (GTA/GTC/GTG/GTT)
all read 162 while the single A codon reads 1. Summing per-codon gives
649 strains at a locus whose distinct_strain_count is 160.

Verified at scale rather than inferred from one locus: across all
1,192,971 product groups having more than one codon row, strain_count is
constant within the group -- zero exceptions.

A scientist summing this column would have over-counted badly, so the
displayName now says "(per product)" and the help states the repetition
explicitly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One table with a visible Source column, not two tables, so caller
provenance travels with every row and disagreements stay legible.

LEFT joins TranscriptAttributes because na_feature_id is null for
intergenic calls; an inner join would drop them silently.
Adds SNP Alleles, Indel Alleles, and Strain Statistics category nodes so
the two allele classes read as distinct sections on the record page.

Requires wb ontology; wb model alone does not regenerate the OWL and
would leave every attribute uncategorized with no error.
The overview used '<dt><b>SNP Alleles</b></dt><dd></dd>' as a section
header, but the client DROPS any dt/dd pair whose dd is empty. Both
headers therefore vanished, and a MIXED locus rendered two identical
Reference/Major/Minor triplets with nothing indicating which was the SNP
view and which the indel view -- precisely the ambiguity the two-section
design exists to prevent.

Verified on Variant_Pf3D7_01_v3_12, which reads:
  SNP Reference A | SNP Major A (0.9316) | SNP Minor C (0.0598)
  Indel Reference A | Indel Major A (0.9316) | Indel Minor AC (0.0085)

The same empty-dd rule is why Gene(s) and Most Severe Impact (Product
Call) are correctly absent on that locus: both are genuinely null there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A textAttribute template of "$$allele$$ ($$frequency$$)" renders a bare
" ()" when both operands are null, so every pure-SNV record page showed
"Indel Major ()" and "Indel Minor ()" -- about 4M of the 4.39M records,
reading as a rendering bug rather than as "no indel at this locus".

Moving the concatenation into SQL with a CASE yields NULL instead, and
the client already drops any dt/dd pair whose dd is empty (the same rule
that correctly hides Gene(s) on an intergenic locus).

Verified on all three variant types via the service:
  SNV   -> snp_* populated,   indel_* null
  INDEL -> indel_* populated, snp_* null
  MIXED -> both populated
and on the rendered SNV page: zero occurrences of "()" in the overview.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Final review found a critical layout defect plus four cheaper ones.

CRITICAL -- the three new category nodes were attached to the ontology
ROOT, not to DNA polymorphism. individuals.txt column 2 is the parent IRI
and column 3 its label; the hand-typed rows left both empty and put the
IRI in column 4 (recordClassName) with 'category' in column 5, so each
row asserted a nonexistent record class and declared no parent. SNP
Alleles, Indel Alleles and Strain Statistics rendered as top-level peers
of Genomic Location, ahead of it. Now verified as
"Genetic variation > DNA polymorphism > SNP Alleles" in the assembled
tree, with H2/H3/H4 nesting on the page.

record_overview was parented to topic_2885 with record+download scope,
unlike all 15 other record classes; it rendered as an ordinary attribute
row labeled "Overview" inside the DNA polymorphism subsection and became
a selectable report column carrying ~4KB of HTML per record. Now
topic_0219 / record-internal.

The tuning table joins DatasetPresenter but never declared the
dependency, the exact failure mode its own comment warns about -- an
undeclared dependency yields a silently stale or absent join rather than
an error.

linkedGeneIds rendered text byte-identical to gene_ids, so the page and
column chooser showed two columns both named "Gene ID(s)" with the same
content. Dropped; gene_ids is used directly. The four *_and_freq columns
also duplicated their base column's displayName and are now
"... Allele + Frequency".

effect_summary_* had no help, so a blank "Effects (Product Call)" next to
a populated SnpEff column read as "this caller found no effect" rather
than "did not call here" -- true for 2,700,000 of 4,390,908 loci.

Also set displayOrder within the allele and statistics sections: they
sorted alphabetically, which put "Reference Allele" LAST. Now
reference -> major -> minor -> HGVS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ence)"

The old label "Strain Count" sat beside "Called Strain Count" (159) and
"No-Call Strain Count" (57) and read as though those two should sum to it.
They do not, and are not meant to.

called + no_call is the genotyped panel -- 216 at Pf3D7_01_v3:29514, and
call_rate = 159/216 = 0.7361 confirms that denominator on all 4,390,908
loci. distinct_strain_count is called + 1, verified as exactly +1 on every
row, the extra strain being the reference, which carries the reference
allele by definition. It is the denominator behind the allele frequencies.

Renamed in both places it appears -- the attribute displayName and the
record overview -- so the same number is not shown under two names. 160
now sits directly beside 159, making the +1 self-evident without help text.

Reference-strain explanation confirmed by John rather than inferred.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the temporary jbrestel.VariationAttributes stub with
ApidbTuning.VariationAttributes now that the tuning job has built it, and
removes the TEMPORARY STUB comment from both query files. 6 SQL
references flipped: 4 in variationAttributeQueries.xml, 2 in
variationTableQueries.xml.

The real table was compared against the stub before flipping rather than
assumed equivalent: identical column sets and ordering, and EXCEPT in
both directions returned 0 rows over all 4,390,908 rows. Aggregates
populated as expected (gene_ids on 2,879,337 loci -- zero would have
meant the TranscriptAttributes dependency failed to resolve).

Record pages verified unchanged across all five test loci: SNV, INDEL,
MIXED (both allele classes), multi-gene, and a caller-disagreement locus.
Assembled SQL confirmed to read ApidbTuning via wdkQuery -showQuery.

grep -rn jbrestel over Model/lib/wdk and Model/lib/xml returns nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These were authored in the agentic-veupath-dev control-plane checkout because
that is where the build commands run from, but they describe work that lands
entirely here. Moved to sit beside the code they specify, matching the
convention already used for the EDA data tables and the fungidb strain-segment
record.

Content is carried over verbatim; the plan's progress block is stale relative
to the commits on this branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The progress block had been stale since Task 4, claiming Task 5 was next while
tasks 5-14 were all committed on this branch, and all 91 step checkboxes were
unchecked. Replaced with a task-to-commit map covering 0-14, and checked the
boxes.

Also retires the stub warning. Both halves are settled: Task 14 (2723cda)
flipped all six query references to ApidbTuning.VariationAttributes, and a
tuning run built the real table on unidb_shu_a (variationattributes1121, 1688 MB,
4,390,908 rows, plus the view; all 17 va.* columns the model reads resolve).
jbrestel.VariationAttributes no longer exists in the database. The plan's
original assumption that the flip would cost dev-instance buildability no longer
holds, and said so misleadingly.

Task bodies keep their jbrestel snippets — that is what was executed — with a
banner not to copy them forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the question-side scaffolding for the variation record -- a
variationParams paramSet, a VariationsBy querySet, a VariationQuestions
questionSet, and their apiCommonModel.xml imports -- plus the first
search, VariationBySourceId, ported from the deprecated
NgsSnpBySourceId.

Settled decisions worth the reader's time: the nine-project
includeProjects list is deliberate and must not be narrowed to the
subset appDb's loaded three; ID matching is exact source_id only, with
the alias query named as the home for any future tolerance; the
question inherits the record's default summary columns rather than
overriding them.

The four remaining snp searches are out of scope and explained: they are
HSSS processQuery searches over per-strain data, which is the seam the
record spec deferred.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five tasks: create the three question-side XML files, import them and
confirm wb model loads, add the individuals.txt row and run wb ontology,
verify the search in the browser, close out the spec.

Uses a cheapest-rung-first verification ladder (xmllint, wb, the WDK
service, then the browser) in place of unit tests, since the change is
declarative model XML and the sanity model is dead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three new files -- a variationParams paramSet holding the ID datasetParam,
a VariationsBy querySet, and a VariationQuestions questionSet -- giving the
variation record its first search, ported from the deprecated snp record's
NgsSnpBySourceId.

Not yet imported into apiCommonModel.xml, so this commit changes nothing
that builds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"See spec section 5.1" did not say which of the two specs in
docs/superpowers/specs/ it meant. The claim is accurate against the searches
spec, but the record spec's section 5 has no subsections, so a maintainer who
opens the record spec -- the likelier bookmark when editing this record --
finds nothing and concludes the comment is stale.

Names the spec, matching the disambiguation style already used in
variationQueries.xml. Comment-only; no element, attribute, value or SQL change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wires the new variation search files into the model. wb model succeeds and
the assembled PlasmoDB model now contains
VariationQuestions.VariationBySourceId.

The search is not yet categorized, so it will not appear in the searches
menu until the individuals.txt row lands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Places the search under edamontology topic_0199 with menu and webservice
scopes, matching the Popset ID search and the parent already used by the
variation record's gene-linkage and effect-rollup attributes.

Verified by running wb ontology and grepping the freshly built
gus_home categories_merged.owl: the class appears exactly once, with
rdfs:subClassOf topic_0199, targetType search, and scopes menu and
webservice. The /service/ontologies/Categories check could not be run --
the browser is blocked at the veupathdb.org pre-release autologin gate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scaffolding and VariationBySourceId are built and verified on the plasmodb
dev instance: wb model and wb ontology both pass, the search appears in the
searches menu, a single ID goes straight to the record page, and a multi-ID
run shows the record's default summary columns.

Records in the plan what was substituted and what remains unrun -- the WDK
service endpoints were unreachable behind the autologin gate so server-side
equivalents were used, and the empty-project case still needs a site whose
project lacks variation data.

The four HSSS-backed searches remain out of scope pending the per-strain seam.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jbrestel and others added 26 commits August 7, 2026 17:27
EDA stores the dataset attribute as a dataset stable ID (DS_1d17c1883c) - the
right thing to store, an opaque thing to show. The Dataset facet listed four
DS_ hashes. Both metadata queries now LEFT JOIN apidbtuning.datasetpresenter and
select COALESCE(dp.display_name, av.string_value).

Applied to SamplesMetadataByStudy as well as SamplesMetadataByStudyWithRef, so
cnv_sample_meta gets it too. The fork between those two queries is about which
samples exist, not about presentation, and a facet reading DS_302d3e3bc0 in the
CNV searches but a sentence in the variation searches - for the same samples -
would be a worse bug than the one being fixed.

COALESCE rather than the join value alone, so a sample survives when no presenter
row exists; on a gated dev instance datasetpresenter holds only loaded datasets.
The join is narrowed by provider_label rather than left to match the DS_ shape,
which works today only by coincidence. dataset_presenter_id is unique, so no row
multiplication - counts are identical before and after (3771/216, 3774/217).

Verified on the running instance: the HSSS filters show four display names
including "Genome Sequence and Annotation" for the reference, CNV shows the same
three isolate datasets and no reference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The record class was named Variation while every source_id it holds reads
Variant_<sequence>_<location>, and its own attributes already said Variant
Type / Variant Call Set. "Variation" is the phenomenon; a record is one
concrete allele at a locus, so it is a variant.

Renames the WDK layer only - record class, question set and its five
searches, query sets, param set, category ontology nodes, and the six model
files - and sets displayName to Short Variant / Short Variants, naming what
the record actually holds (SNVs and small indels) rather than variation in
general.

Deliberately unchanged:
 - urlName stays "variation" so existing record urls keep resolving; the
   feature is not public yet, so this can move in a separate change.
 - variation_sample_meta{,_a,_b} and chromosomeOptionalForVariations are
   contracts with the HSSS plugins in ApiCommonWebService; renaming them
   breaks the searches at run time, not build time.
 - apidb.Variation* and apidbtuning.VariationAttributes /
   GeneVariationSummary are database objects, out of the model's reach.

Verified on a dev instance after wb ontology: all five searches resolve
with expandParams, and the category tree keeps all 59 nodes in their
sections with no stale VariationRecordClass references.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
HSSS computes over single-nucleotide variants only, so these four cannot
return an indel no matter what the record class is called. VariantBySourceId
keeps its neutral name because an ID lookup goes to the tuning table and does
return both.

Prefixes shortDisplayName too: a strategy box showing bare "Location" or
"Two Groups" is exactly where the SNV-only scope would otherwise be lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"SNV Gene ID(s)" and "SNV Genomic Location" named their input; all four
HSSS searches compute differences within a sample group, so they now say so:
"SNV Differences for Gene(s)" and "SNV Differences for Genomic Location".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…search

The sibling of GenesByNgsSnps. That one runs HSSS over a sample set the user
picks and takes minutes; this one reads apidbtuning.GeneVariationSummary,
returns in milliseconds, and reports the same numbers the gene record page
shows. Neither replaces the other: dropping the HSSS search would delete the
sample-set comparison, and not having this one leaves the record page and the
search disagreeing about the same gene with no explanation. GenesByNgsSnps is
renamed to "SNV Characteristics Within a Group of Samples" to say so.

Filtering is ONE filterParam over sixteen statistics, not sixteen range params.
The range-param version was written first and abandoned: a numberRangeParam
cannot express "untouched", so every range applies on every run and any gene
NULL in that column is excluded - and GeneVariationSummary NULLs its frequency
statistics below their allele floors on purpose, which for tbruTREU927 is 100%
of rows for three columns. Working around that needed a full-span escape clause
with bounds hardcoded in two files, which then broke when NumberRangeParamHandler
rounded a value onto the declared ceiling. The filterParam has none of it: a gene
with no value contributes no metadata row, distributions come from the data, and
adding a statistic is one ontology row plus one VALUES tuple.

Also adds organismVQ.withVariantSummary, scoping the organism tree to organisms
that actually have a summary row rather than to a datasource type.

Verified against unidb_shu_a on a dev instance: the impact distribution sums to
all 5,579 pfal genes (HIGH 2,797, matching SQL), piN/piS reports 2,370 - exactly
the genes with a non-null value - and filtered runs match direct SQL counts
(piN/piS >= 1.5 -> 271 genes/272 transcripts; n_lof >= 5 -> 317/319).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ones

TranscriptRecordClass already publishes this tuning table's sortable subset,
and gene_pi_n_pi_s there is gvs.pi_n_pi_s_snpeff - the same column the search's
filter reads. The dynamic columns were showing identical numbers under parallel
names, which is a divergence waiting to happen.

Drops the dynamicAttributes block and the gvs_* columns from the id query,
leaving it returning only the four columns WDK needs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Not just the wsColumn: both postCacheUpdateSql blocks enumerate the result
columns explicitly in their INSERT and SELECT lists, so without adding it
there the sibling transcripts they backfill would carry NULL span density
while the matched transcripts carried a value.

Required by the plugin change in ApiCommonWebService 7a61e99 - getColumns now
advertises span_snp_density, and PluginExecutor.validateColumns rejects a
plugin advertising a column the wsColumn list lacks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds SNVs per kb (gene span) alongside the now-actually-CDS density, renames
the ratio to say it is site-normalized, and corrects three labels that never
matched the classifier: nonsynonymous excludes stop-gained, and non-coding is
really unclassified - class 0 means no product byte was available, which
covers unclassifiable positions as well as genuinely non-coding ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Density is CDS density, the ratio is site-normalized, and the class enum terms
follow the corrected column names. Internal enum values are unchanged - they
are a contract with legalParams and with the filter's branches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The PlasmoDB description promised normalized Dn/Ds in a future release; it is
here. Also states in both descriptions that these statistics are sample-set
scoped and so not comparable to the precomputed ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…cription

The bullet keeps saying the ratio is site-normalized; it no longer quotes the
17.49% figure or the 1.43x factor at the user.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These are computed over the samples the user selected, but a results table can
carry them alongside the whole-cohort columns from the gene record (Total
Variants, Variants per kb, piN/piS) with nothing to tell them apart. Each now
carries a (sample group) qualifier.

The two density labels swap their existing parentheses for commas so no header
ends up with two bracketed clauses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CASE in SampleOntologyByStudy translates EDA data_type values into WDK
filter-param types, and NULL is meaningful: it marks a BRANCH node. So an
unmapped type does not degrade gracefully - it makes a LEAF look like a
branch, and WDK throws "ontology items have no children ... but have a null
item type", 500ing the whole search.

'date' had no arm, which broke the 13 dnaseq studies carrying a Collection
date. The partial test appDb held none of them, so nothing exercised it.

Add the date arm and record the invariant in the comment: every data_type EDA
can emit needs a case here. The full appDb uses category/date/integer/number/
string, all now mapped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
IsolatesHTS was a no-op, so no dnaseq:: properties were injected at all and
/service/jbrowse/dnaseq/<org> answered {"tracks":[]} for every organism.

Re-enable only the per-dataset half. The per-sample half is not repaired but
removed: it called getSampleList(), which keys samples on organismAbbrev +
datasetClassCategory + experimentName, and the merged dnaseqExperiment class
carries an empty category while its samples span two ("Genetic variation" for
SNPs, "Structural variation" for CNVs), so that key cannot match and the call
throws. Dropping the per-sample path removes its only caller.

The sample list, and which measures each sample has, now come from the
webservices tree at request time - the pipeline that writes those files is the
only thing that knows them, and LOH in particular exists only for diploid
organisms, which no per-dataset property could express.

Retiring the template also means retiring its anchor: presenterInjectTemplates
fails on an anchor naming a template that no longer exists, and no amount of
re-running fixes it.

hasCNVData is no longer emitted. 118 presenters say true and 77 say false, but
the pipeline now writes _normalisedCoverage.bw regardless, so the flag stopped
describing the data. It stays declared in getPropertiesDeclaration because
presenters still pass it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both classes derive label, id and track type from the dataset, which suits one
track per dataset but not a caller emitting several per sample. label is the
JBrowse track id and must be unique across the whole response, so uniqueness
needs to come from construction rather than from sample names happening to be
distinct.

SingleCoverageTrackConfig: honour an explicit label, and an explicit
track_type_display. The latter matters because the existing
"Coverage (ploidy Normalized)" default is applied to every order-less,
dbid-less caller - correct for the CNV track it was written for, wrong for a
caller emitting SNP density or LOH.

MultiBigWigTrackConfig: honour explicit label and id, applied last so they also
beat the jbrowse2 branch. Add has_alignment => 0 for callers with no
unique/non-unique read split at all; without it every track claims
"unique and non-unique" and shows up under the RNA-Seq Alignment facet.
track_type_display is deliberately NOT overridable here - the ::Density and ::XY
subclasses set it after SUPER::new returns, so anything set here is clobbered,
and a hook that silently does nothing is worse than no hook.

All defaults are unchanged, so existing callers are unaffected: the only other
caller of either class, JbrowseRnaAndChipSeqTracks, passes none of these.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shape by variant type and colour by predicted effect, both resolved per feature
so one track can mix them: substitutions get a diamond, indels a box (which also
renders at the variant's real width, where an SNV is always one base).

Colour follows the varianteffect plugin's convention - green synonymous, purple
non-synonymous, red truncating, blue intron/other - read from snpEff's
Annotation (ANN field 2). Impact (field 3) was tried first and rejected: it has
only four ordinal levels, and as a warm ramp its top two were indistinguishable
at glyph size.

The exact hexes are load-bearing. The convention's nominal green/red pair sits at
deltaE 3.3 under deuteranopia, a hard fail on the most consequential pair here -
a silent change against a truncation. Under deuteranopia red and green separate
mainly by lightness, so red is deliberately dark (#b71c1c), lifting the worst
all-pairs separation to deltaE 9.7 with all four clearing 3:1 on white.

Two traps this file sets for anything added to it, both silent:

  - A blank line inside a function body terminates the multi-line value. The
    fragment then fails to compile and the key is left a plain string, so the
    track renders in a default colour with no error anywhere. Neither function
    here contains one; verify typeof is 'function' after building.
  - Do not deref track.browser.config in a function used for glyph, the way
    snpColorFxn does. getConfForFeature resolves glyph with a null track, so the
    deref throws. Both functions here are self-contained and feature-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
glyph and track_type_display become optional args defaulting to today's values,
so a caller can supply a per-feature glyph function while the existing
per-sample ebi_VCF tracks are unchanged. The old default also hardcoded
"VCF from VectorBase" as the user-visible track type, which is simply untrue on
any other site.

Raise maxFeatureScreenDensity to 50. This - not chunkSizeLimit - is what
produces "Too many features to show" when zoomed out: CanvasFeatures.fillBlock
refuses a block whose feature density exceeds it, and the JBrowse default is 0.5
features/pixel. A merged call set runs about 6.8 variants/kb, so in a ~1000px
window a 640kb chromosome measures ~2.4 features/pixel and the largest ~3.3Mb
P. falciparum chromosome ~12 - both well past 0.5. 50 clears a whole chromosome
with headroom, and is deliberately not the 9999999 RNASeqJunctionTrackConfig
uses, which would have JBrowse attempt 100k+ glyphs at genome scale.

Raise the store fetch ceilings to match the size of a merged multi-sample call
set, and emit all of these as numbers rather than the quoted string
chunkSizeLimit used to send.

Note VCFStore also sets a chunk_size_limit that nothing reads, so it does not
participate in any of this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rewrite around a single glob of the organism's dnaseq/bigwig directory rather
than a build-time injected sample list. The old arrangement had the track list
and the files it pointed at coming from separate sources of truth; they diverged
and the endpoint silently served nothing. Reading the tree makes "a track
exists" and "its file exists" the same fact - and is the only workable source
for LOH, which the pipeline emits per sample for diploid organisms only
(tbruTREU927 has it, the haploid pfal3D7 does not).

The presenter still owns what only it knows - display name, summary,
attribution, category - so the two sources of truth are split by what each can
actually know.

Per sample: coverage and ploidy-normalised coverage as individual tracks, the
two density measures overlaid in one autoscaling XY track, and LOH standalone.
Coverage is deliberately not paired with anything: raw coverage runs to
3367-50590 against normalised coverage's ~5, and a ~200x gap on a shared axis
renders the smaller trace as a flat line. Pooling every sample's densities into
one plot was tried and reverted - with only two colours available, one per
measure, a few hundred overlaid samples are mutually indistinguishable. Data
volume was never the constraint; legibility was.

Also emits the merged annotated VCF as a variant track. It is per organism, a
merge across all experiments, so it is built once outside the dataset loop and
carries its own metadata, there being no single presenter to inherit from. Its
summary doubles as the legend, since shape and colour are the only channels
distinguishing type and effect on screen.

Removed with the old design: the BAM alignment track (b71 ships no BAMs), the
${organismAbbrev}_HTS_SNP_(.+)_RSRC regex (cannot match _dnaseqExperiment_RSRC
names), and a hardcoded afumAf293/V157-47 exclusion.

A missing bigwig directory for an organism that has dnaseq datasets is now a
hard failure naming the path, because an unreadable mirror must not look
identical to "this organism has no DNASeq data" - that equivalence is what let
the original bug hide. A declared dataset with no files is still skipped
quietly, since presenters are deliberately a superset of any one build.

Note this module's stderr is merged into its own JSON response by
JBrowseService.responseFromCommand, so a Perl warning corrupts the output. That
is why cov_max_score_default is always passed explicitly (the setter does
"$value + 0", and undef there both warns and stores 0) and why the VCF track
passes an empty-string rather than undef attribution.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The include had been commented out long before this branch, so the genome
browser never asked for dnaseq tracks even once the endpoint returned them -
they would have existed and been unreachable. The phenotypeTracks set already
included it.

This is what makes the volume user-visible: pfal3D7 contributes 1612 tracks,
taking the faceted selector from 1392 to about 3000.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The accessor pair and the setChunkSizeLimit(10000000) call in new() wrote a
chunk_size_limit that nothing ever read: the value the track config actually
carries is emitted by VcfTrackConfig. Raising it here therefore looked effective
and did nothing, which is worse than it being absent, so leave a note saying
where the live one is.

AlignmentsTrackConfig keeps its identically-named pair - that one is on a
different key and its own getJBrowseObject genuinely emits it.

Verified behaviour-neutral: the dnaseq endpoint's output is canonically identical
before and after (byte comparison is meaningless here, since Perl randomises hash
key order between processes).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two reasons, either sufficient on its own.

It was misplaced. The merged annotated VCF is one file per organism, a merge
across every DNASeq experiment, so it never belonged in a per-dataset module - it
had to be emitted outside the dataset loop with hand-written metadata precisely
because no single presenter owns it.

And it was slow where it mattered. The gene page's SNP links use the
geneticVariationTracks set, which includes organismSpecific but not dnaseq. With
the VCF on the dnaseq endpoint those links had to load and parse ~1600 per-sample
coverage and density track configs to draw variants; via organismSpecific the same
set is ~100. This also lets the geneticVariationTracks/syntenyTracks branch of
jbrowseTracks stay as it was - an earlier attempt added dnaseq to it, which would
have dragged all 1600 into the synteny view too.

Replaces the "SNPs by coding potential" unified SNP track, removed here. It was
backed by the SNP:Population REST feature, part of the SNP infrastructure the
Variant record supersedes, and was already unreachable: no organism in build 71
sets the hasSnp property that gated it, and the gate could not have fired anyway,
since the fallback is an empty hashref and {} == 1 compares a ref address to 1.
UnifiedSnpTrackConfig is deleted with it, having no remaining references.

Verified: dnaseq 1612 -> 1611 tracks with zero merged-VCF tracks, organismSpecific
102 -> 103 with exactly one, all of its glyph, colour, fetch and density settings
carried over intact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SnpsGbrowseUrl was commented out and targeted the gbrowse_img CGI and the
HtsAlignmentSNPsByClass gbrowse track, neither of which exists. Restore it against
the merged annotated VCF, and rename it "Short Variants in the Genome Browser" -
it covers indels as well as substitutions, so "SNPs" undersold it.

snpsJbrowseUrl and snpsJbrowseFullUrl were live but pointed at "SNPs by coding
potential", which no organism in build 71 emitted, so those links were already
dead rather than merely dated. Same for dynSpanRecord's span link.

Track sets are chosen per link rather than uniformly: the gene page's SNPs view and
snpsJbrowseUrl use geneticVariationTracks (~100 track configs), while
snpsJbrowseFullUrl keeps the full default set, which is the point of a "full" view.

On dynSpanRecord the organism-dependent track name is appended in the two consuming
URLs rather than inside the nested jbrowseTracks attribute they interpolate: there
is no precedent in this model for a macro inside a composed textAttribute, so
recursive expansion is unproven, whereas $$organismAbbrev$$ demonstrably expands in
the consumers. jbrowseTracks keeps only the static part.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GenesByVariantCharacteristics covers indels as well as substitutions, so naming it
"SNV Characteristics" was too narrow.

GenesByNgsSnps keeps its "SNV Characteristics Within a Group of Samples" name
deliberately: the per-sample-group statistics really are SNV-only.

Also updates the four help texts that referred to the renamed search by its display
name, which would otherwise send readers looking for a search that no longer exists.
The fifth such reference is left alone - it points at the per-sample-group search,
whose name is unchanged. Note "SNV Characteristics" is a prefix of "SNV
Characteristics Within a Group of Samples", so these are anchored rather than
blind-replaced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant