Skip to content

Commit 0dabbab

Browse files
JamesKaneclaude
andauthored
Feat/ancestral origin icicle (#31)
* Return the caller's own sample guid with an IBD suggestion `/api/v1/ibd/attest` gates on `owns_sample(attester_did, claimed_sample)`, but a self-publishing Edge client has no way to learn its server-side `core.biosample.sample_guid` — the suggestions payload returned only the *candidate's* guid, so Navigator could never fill in `claimed_sample` and the attest endpoint was unreachable from the edge. `suggestions_for_did` already joins on `ms.target_sample_guid` (that is how the per-DID scope is enforced), so surfacing it costs nothing and leaks nothing: the caller owns that sample by construction. `suggested_sample_guid` supplies the `counterpart_sample` of the same report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tree): one-off script to fill Y tree branch-row build coordinates The de-novo loader reuses a catalog row only when it matches on `coordinates @> {'hs1': ...}`, so markers whose hs1 coordinate had not been lifted yet missed the match and got a fresh hs1-only row — and that row is what `tree.haplogroup_variant` points at. `variant-name-reconcile` later adopted the marker's name onto the branch row but never touches `coordinates`, leaving the tree with correct names and, on the 2026-08-04 prod dump, GRCh38 for only 44,181 of 203,983 branch SNPs (21.7%). The Navigator places each source in its native build with no liftover (`place_y_consensus_decodingus`), and `parse_decodingus_json(json, build_key)` drops every locus lacking that build's coordinate — so a GRCh38 subject saw 3,413 of 11,421 Y nodes. All 33 backbone nodes survived; what vanished was the terminal tree (node visibility 81% at depth 0-10 → 21% at 31-40), so GRCh38 subjects placed plausibly but shallow rather than failing outright. Copy GRCh38/GRCh37 from the marker's catalog row (same canonical_name AND identical hs1 site + alleles) rather than chain-lifting: the catalog row holds YBrowse's own values, and a lift would re-derive them and can mismap in the inverted / ampliconic Y blocks. All 132,183 candidate rows agree with their twin on position and alleles (0 swapped), so the copy is exact. Rows whose twin disagrees are not filled — the match is enforced in the join. Verified against a restored copy of the 2026-08-04 prod dump and against decodingus_cutover: 130,553 rows filled in 34s, GRCh38 44,181 → 174,734, node visibility 3,413 → 11,104/11,421, flat across every depth band. Idempotent — a re-run fills 0 and leaves tree_revision alone (the ~60 MB tree payload should not be invalidated for a no-op). Residual ~27.6k rows have no named twin and need variant-coord-lift, which requires the reverse hs1->GRCh38 chain staged to establish its pivot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(tree): ancestral-origin icicle for the genealogical era The public tree answers where a clade sits phylogenetically and, in the "Geography & Time" panel, where its samples were collected. Neither answers what a surname project actually asks: as this branch splits, where do the lines go? ytree.net answers it by putting geography onto the phylogeny itself. This adds that view — the Big Tree's top-down icicle, containment carrying descent, with each band filled by where its men's most distant known ancestors came from. The AppView had nothing to draw it from. Its only locality datum is `core.specimen_donor.geocoord`, and of 9,642 placed Y samples 1,380 carry one — all of them ancient or academic. The 7,882 `cohort=bigy` D2C tips, which are the entire genealogical era, have 3 between them. So this also adds the substrate: a `com.decodingus.atmosphere.ancestralOrigin` lexicon, mirrored to `fed.ancestral_origin` (migration 0074) by the existing Jetstream consumer. MDKA as publishable data is not new policy — `biosample-identifier-dedup.md` already records it as "genealogical context, not PII". What is new is that the AppView now *enforces* that rather than asserting it. Five gates run at ingest and REJECT the record, never store-and-hide, because a row that exists is a row some future read path can leak: a single-token surname (particles allowed, so `van der Berg` survives and `Thomas Michael Kane` does not), `birthYear <= 1900`, country-only when no birth year establishes the ancestor is long dead, coordinates re-coarsened to ~1 km whatever the client sent, and a join key that is never rendered. The bulk-load exclusion in `import_kit_identifiers.rs` stands: origins enter only when a PDS publishes them, so this ships dark until the Navigator half lands. The two migration headers that say otherwise (`0012_fed_reporting` here, `0030_mdka` in Navigator) are deliberately NOT edited — both repos run `sqlx::migrate!`, which checksums applied migrations, so changing even a comment would fail every existing database. `proposals/ancestral-origin-icicle.md` §2 is the amendment of record and 0074's header points at it. The D4 assertion-store rail rejecting MDKA_IS is untouched: it governs assertions about a *living* subject, which is the distinction that keeps the two apart. Resolution runs through `core.biosample_identifier`, not the at-uri. Zero placed samples carry an at-uri — the tips were bulk-loaded, not federated — while 7,548 carry an FTDNA row from migrations 0059/0060, which exist precisely to match a re-published donor to its existing biosample. Both paths are unioned so at-uri works as federation grows. Three things came from measuring rather than reasoning: * Place normalization was validated against all 3,356 real MDKA strings, which caught what hand-picked cases missed. US ZIPs went unstripped, making dozens of singleton "admins" (`Va 24521`, `Wv 26801`) that are all one state; the country table was too short; and a parenthetical qualifier dropped the row entirely. 705 distinct admin strings fold to 457, with 0 unresolvable and 6 no-country out of 3,356. * A band spans its PARENT's TMRCA to its own, not its own `formed_ybp`. The obvious choice is wrong: the two are independent point estimates under no monotonicity constraint, agreeing on 898 of 10,252 edges while 4,243 (41%) have the child forming before its parent's split. Rendering the real tree put `R-A13318` at exactly its parent `R-S764`'s y. Parent-TMRCA → own-TMRCA has zero inversions, so containment holds by construction. * Nothing is dropped silently. A sample on a de-novo node used to contribute to nothing, so every band above it understated itself; it now climbs to the nearest named ancestor as sample tips already do. Branches with no origin beneath them are pruned — unpruned, R-S764 drew 175 bands across 7,944px to show 10 origins; pruned it is 37 in 768px — and the count, the placed total, and the samples with no published origin are all stated on the page. Colour is the validated 8-slot categorical palette, fixed order, never cycled; a ninth locality folds into a reserved neutral that also carries "no locality recorded", since an absence is not an identity. Both modes pass the checker; light mode's contrast warning is met by the always-on band labels and the table view. Tests: 14 normalizer, 6 ingest-gate, 18 layout. Suites green (du-db 52, du-jobs 46, du-web 93). Verified end to end against the dev DB with synthetic records on real FTDNA kits, which resolved to real placed nodes; those rows were removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(tree): preview data for the origins icicle, and fit labels to their boxes The icicle ships dark — `import_kit_identifiers.rs` reserves MDKA for records a PDS publishes, and the Navigator publisher is not built — so there was no honest way to put the view in front of a reviewer. `seed-ancestral-origins.sql` fabricates a plausible cohort against REAL tree placements: 708 rows over three era-gated clades (R-DF85, R-S764, R-Z3000), every row stamped `did = 'did:plc:preview'` so the set removes with one delete and can never be mistaken for a contributor's record. The mix is deliberately awkward rather than tidy, because the tidy version hides the cases that matter: more than eight distinct counties (so the fold into the reserved "Other" slot is exercised), US diaspora beside Irish counties, country-only rows, and men with no locality at all. Rows are written already conformant to the ingest gates — single-token surnames, birth years at or before 1900, 2dp coordinates, and no place or coordinate where there is no birth year — so the preview shows what ingest would actually have kept rather than a state it would have rejected. Seeding it immediately exposed a bug that the layout tests could not: the boxes are sized by the phylogeny, not by the text. A tip is at most 74px and holds about 13 characters, while `Sullivan · Co. Limerick` is 23, so labels ran straight through their neighbours and the tip row was unreadable. `fit()` now truncates band and tip labels to their box with an ellipsis, and the unabbreviated form moves to the hover title so nothing is lost. The rectangle assertions all passed throughout — this was only ever visible by rendering it. Also recorded from the same run, unfixed: with dense data pruning stops helping. R-DF85 draws 266 bands across 11,220px because 278 of its 283 samples carry an origin, and the page scrolls horizontally like the Big Tree and FTDNA's block tree do. That is inherent to one box per man at a legible width, not a defect, but it is the shape a width-reduction pass would target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tree): make the origins icicle readable on a real clade R-DF85 drew 266 branches across an 11,220px canvas — six screens of horizontal scrolling for a chart nobody could read. Canvas width is driven by the number of leaf branches, so pruning empty ones only helps where data is sparse; on a well-covered clade (278 of 283 samples carry an origin) it does nothing. Three changes, all found by looking at the rendered page rather than the tests. A DEPTH BOUND, defaulting to 4 levels with a selector (1-8). This is a legibility bound, not a data one: branches past the depth are marked `hidden` rather than dropped, which routes them through the same path that already handles de-novo nodes — their men are attributed to the nearest drawn ancestor. So every band's composition is identical at every depth and only the visible branching changes, which `folding_by_depth_preserves_composition_exactly` pins. R-DF85 goes 11,220px → 3,498px at the default, 2,016px at depth 2. Folded bands carry a "+" and say so on hover, so "this branch is simple" is never confused with "you are not being shown its shape". TIPS TOO NARROW TO LABEL ARE DROPPED AND COUNTED. Men share their band's width, so a band holding forty of them produced forty 8px slivers that hid the composition bar above them rather than adding anything. Below 26px the box goes and the man is counted instead — reported on the page, never silently. "NO LOCALITY RECORDED" IS NOW DRAWN. It was counted, legended, and then left as bare band background, so the chart disagreed with its own legend and a branch of unrecorded men looked like a branch with fewer men. It is now a real segment, always last so absence sits at the same end of every bar. That exposed a second problem: slot 0 carries both "Other" (localities past the eight palette slots) and "no locality recorded", and the legend called both of them the latter. They share a colour but not a meaning, so `Segment::unknown` now separates them in the legend, the tooltips and the table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(tree): blocks show their equivalent SNPs; origin colour belongs to the men Two corrections to what a block means. A BLOCK IS ITS SNPs. The branch's phylogenetically equivalent mutations are unordered — nothing separates them — so the list *is* the block, exactly as the Big Tree draws it. They were missing entirely; a block was a bare rectangle. `variant_names_for` fetches them for the whole window in one query rather than one per branch, and they flow into as many columns as the block's width allows and as many rows as its height allows. The block is never grown to fit the list: its height is elapsed time and has to stay on the shared axis, so what does not fit is reported as "+N" on a line of its own. BLOCKS ARE NO LONGER TINTED BY ORIGIN. Colouring a clade by the composition of its descendants asserted something the data does not support — a branch has no locality, only the men standing on it do, and a modal-origin tint reads as a claim about the whole lineage. The colour now lives exactly where the claim does: on each man's box, keyed to his own most distant known ancestor. The legend and table still carry the composition, which is what explains those colours, so `Segment` became a tally rather than a drawn mark and lost its geometry. Two layout bugs that only rendering showed. A leaf block is exactly LEAF_W wide, which is narrower than one preferred SNP column, so flooring the column count gave it zero columns and dropped its SNPs — the common case, not an edge one; a block now always gets at least one column, sized to what it actually has. And the SNP list was anchored to the block's padding rather than to the name's baseline, so every block opened with its name and first SNP overprinted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tree): size a block by its SNP count, not by the age model A block with 16 equivalent SNPs was drawing as an 18px sliver. Sizing blocks by elapsed years does not survive contact with the data, and both obvious forms of it fail: * a node's own formed_ybp → its own tmrca_ybp draws children ON TOP of their parents — the two are independent point estimates under no monotonicity constraint, agreeing on 898 of 10,252 edges while 4,243 (41%) have the child forming before its parent's split (fixed earlier, in 4ffc3b8); * parent TMRCA → own TMRCA is monotone, so containment held — but it is degenerate. `formed_ybp == tmrca_ybp` on 41% of terminal branches and 26.5% of internal ones, collapsing those branches to a point. On R-DF85 at depth 4 that left 30 of 75 blocks unable to show a single one of their SNPs: R-BY18328 got 3px of span for 9 mutations, R-BY170664 16 SNPs in an 18px sliver, and a quarter of all blocks sat pinned at the minimum height. So the block is sized to its SNPs — one line each, nothing elided — which is what BACKLOG's own block-tree convention already says and what the Big Tree does. The list is the block; truncating it shortens the box, and a shortened box misreports how long the branch ran unbroken. Vertical position becomes cumulative, and the gutter rules off in mutations rather than calendar years. This loses nothing as a time axis. Mutations accrue at a roughly steady rate, and measured on this very tree branch length tracks SNP count at r = 0.975, about 69 years per mutation — better behaved than the per-branch estimate it replaces, which is missing or degenerate exactly where a block most needs a height. Ages keep their two real jobs: gating the view to the genealogical era, and labelling each block. Result on R-DF85 at depth 4: every one of 468 SNPs drawn, 0 blocks hiding any, and the canvas got SHORTER (706px → 439px) because blocks now take the height their content needs instead of whatever a sparse time axis handed them. The "+N did not fit" marker is gone with the problem it reported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tree): drop the cross-hatch, and stop hiding branches that lack origins Two pieces of scaffolding outlived what they were for. THE CROSS-HATCH marked branches with no age estimate. It was justified only because height *was* the age span — an unmeasured branch must not read as a short one. Height is SNP count now (79305f9), so the hatch distinguishes nothing and is just noise on 16% of blocks. Gone, along with the `dated` flag behind it. BRANCHES WITH NO PUBLISHED ORIGIN BENEATH THEM were pruned away. That was how the 11,220px canvas got tamed before the depth bound existed; the depth bound does that job now, and pruning is both redundant and wrong. Origins are an overlay on the tree, not a filter of it — a branch with no locality data is still part of the clade's shape, and dropping it misrepresents the phylogeny to make a sparse overlay look dense. Removing the prune surfaced the tip row's own version of the same confusion. Men were packed into a single row under their branch, so most became slivers narrower than their labels and were dropped and counted instead — 219 of 278 on R-DF85, and with the block tint gone they were then shown nowhere at all. Men now wrap into rows, sized for a readable box first and narrowed only when the branch demands it. But wrapping alone made one depth-4 boundary block absorb 179 folded men and stack them 90 rows deep, a tip row taller than the tree above it. So a man gets a box only where his own branch is drawn: folded upward he still counts in his ancestor's composition, the fold marker is the affordance, and drilling in draws him where he belongs. R-DF85 at depth 4: 76 branches (was 75 — the pruned one is back), all 471 SNPs, no hatch, 64 men drawn and every one legible, canvas 459px. At depth 6 the same view resolves 108 men, which is what drilling in is meant to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tree): the icicle follows the page's theme, and its type is readable Two problems, one of them mine to begin with. THE BLOCK HEADERS WERE ILLEGIBLE ON A DARK-MODE DESKTOP. The icicle carried a `prefers-color-scheme: dark` block, but the site itself is light-only — so on a dark OS the SVG surface went dark while the Bootstrap-derived block fills stayed light, and every branch name became dark text with a dark halo on a light block. Keying off the OS was wrong in the first place: a chart embedded in a page has to follow that page. The dark values now hang off `[data-bs-theme="dark"]` alone, which is what the app would set if it ever gains a dark theme, and `--surface-1` defaults to `--bs-body-bg` so it tracks whatever the page is. The label halo also now uses the block fill rather than the canvas, since the block is what sits behind the text. THE TYPE WAS TOO SMALL — 9px SNP names, 10px branch names. Names go to 12px bold, SNPs and men's labels to 11px, the ruler to 10px, and the geometry is now derived from those sizes rather than hard-coded beside them: line heights, tip height, minimum tip width and the leaf block width all follow the font constants, so changing the type cannot silently break the layout again. A leaf block widens 74px → 90px, which is what a 12-character SNP name (`14405732-C-T`) actually needs at 11px. R-DF85 at depth 4 costs 3,576px → 4,296px of width for it. That is the trade: legible at normal viewing distance instead of technically-present. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(tree): put the ancestral-origin icicle in the site's navigation The view existed but was reachable only by typing its URL. Three ways in now, each at the altitude it belongs to. PER CLADE — a button in the tree's SNP sidebar, beside the sample-map panel that answers the neighbouring question. It navigates rather than swapping a fragment: the icicle is a whole view, not a sidebar widget. PER LINEAGE — `/ytree/origins` and `/mtree/origins`, the Tools menu entry. These open on the tree's default root, which is always older than the era gate, so they render the signpost — and that turns out to be the right landing page rather than a dead end: it states the constraint and immediately offers the clades young enough to have origins. For that to be useful the signpost had to stop listing whatever the tree walk happened to return. It now ranks by how many men a clade holds and drops the ones holding none, so `/ytree/origins` opens on R-DF85 (283), R-S673 (239), R-S764 (232) rather than the alphabetically-first branches. 5,113 clades qualify under the Y root; 60 are shown and the page says so, because a capped list that looks complete is worse than one that admits its cap. Each entry carries its sample count and TMRCA, which is what a reader picks on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 2910309 commit 0dabbab

20 files changed

Lines changed: 3180 additions & 1 deletion

File tree

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
# Ancestral-origin locality icicle on the public Y tree
2+
3+
**Status:** proposed (2026-08-06). **AppView half only** — the Navigator publisher is deferred by
4+
the project owner, so this ships *dark*: schema, ingest, aggregate and view are built and tested
5+
against seeded records, and the surface fills when a PDS starts publishing.
6+
**Scope:** a new `com.decodingus.atmosphere.ancestralOrigin` lexicon, `fed.ancestral_origin`
7+
(migration 0074), a jetstream arm, `du_db::origins`, and a server-rendered icicle at
8+
`/ytree/node/:name/origins`. No change to `core.biosample`, no change to placement.
9+
**Privacy posture:** unchanged from `biosample-identifier-dedup.md` — the corpus carries no
10+
living-donor PII, and MDKA (surname / origin / birth-year of the earliest paternal-line ancestor) is
11+
**genealogical context, not PII**. This design does not relax that; it *enforces* it (§2).
12+
13+
## 1. Why
14+
15+
The public Y tree renders as a cladogram (`du-web/src/tree_layout.rs`) with a per-clade Leaflet map
16+
in the "Geography & Time" panel. Neither answers the question a surname project asks: **as this
17+
branch splits, where do the lines go?** ytree.net answers it by putting geography onto the phylogeny
18+
itself — depth down the page, a block's height its elapsed time, each block spanning its
19+
descendants. This adds that view with locality as the fill.
20+
21+
It only means anything in the genealogical era. Deeper than ~1,500 ybp every block aggregates to
22+
"Europe" and the counts explode, so the view is age-gated rather than offered tree-wide (§5).
23+
24+
**The data gap, measured on `decodingus_cutover` (2026-08-06).** The AppView's only locality datum
25+
is `core.specimen_donor.geocoord`; there is no country and no place text, and legacy had none
26+
either, so the ETL did not drop one. Of **9,642 placed Y samples, 1,380 carry a coordinate**, and
27+
that coverage is entirely ancient/academic:
28+
29+
| source | placed | with geocoord |
30+
|---|---:|---:|
31+
| `EXTERNAL` (ancient + academic) | 1,760 | 1,377 |
32+
| `STANDARD` (`cohort=bigy` D2C tips) | 7,882 | **3** |
33+
34+
The genealogical era — the only era this view is for — has effectively no locality data. That is
35+
what the lexicon exists to supply.
36+
37+
## 2. What may cross the wire, and how it is enforced
38+
39+
The posture is already the project's (`biosample-identifier-dedup.md` §Privacy posture). What is new
40+
is that this design **enforces** it at ingest rather than asserting it in prose. A record failing any
41+
gate is **rejected**, not merely un-rendered:
42+
43+
1. **Surname only.** No given name, ever. The publisher derives it; the AppView independently
44+
rejects a `surname` containing whitespace or more than one name token, so a buggy or hostile
45+
client cannot leak a given name through a field labelled `surname`.
46+
2. **Date ceiling — `birthYear <= 1900`.** A person born in 1900 is 126 today. This is the check
47+
that makes "not PII" verifiable rather than asserted.
48+
3. **Precision ladder when the birth year is absent.** With a birth year: place text + coarsened
49+
coordinate. Without one: **country only** — place text and coordinate are dropped at ingest.
50+
4. **Coordinates coarsened to 2 decimal places (~1 km).** Applied at publish *and* re-applied at
51+
ingest, because the client cannot be trusted to have done it. A county-scale view cannot use more
52+
precision; full precision plus a surname narrows to one family.
53+
5. **The join key is never rendered.** Resolution runs through an FTDNA kit id (§4), and every vendor
54+
namespace is `is_public = false` (`du_db::identifier::is_public_namespace`). The icicle shows
55+
`Kane · Co. Clare`; the kit number must not reach any public projection.
56+
57+
**The bulk-load exclusion stands.** `du-jobs/src/import_kit_identifiers.rs:17` records the decision
58+
that MDKA "enters only when a PDS publishes the sample, never from this bulk load." This design does
59+
not create a manifest or curator path. It is the reason the view ships dark, and that is accepted.
60+
61+
**Two migration headers say the opposite and are deliberately not edited.** Both repos use
62+
`sqlx::migrate!`, which checksums applied migrations — editing a comment in
63+
`0030_mdka.up.sql` (Navigator) or `0012_fed_reporting.sql` (AppView) would fail every existing
64+
database with `VersionMismatch`. **This document is the amendment of record**, and migration 0074's
65+
header points back to it.
66+
67+
The D4 assertion store's PII rail (`research.assertion` rejecting `MDKA_IS`) **stands unchanged**.
68+
It governs assertions made *about a living research subject* within a project, which is a different
69+
question from publishing a deceased ancestor's parish — and the rail is what keeps the two apart.
70+
71+
## 3. The record
72+
73+
`com.decodingus.atmosphere.ancestralOrigin`, one per `(biosample, lineage)`:
74+
75+
```jsonc
76+
{
77+
"biosampleRef": "at://did:plc:…/com.decodingus.atmosphere.biosample/…", // when federated
78+
"externalIds": [{ "namespace": "FTDNA", "value": "B5163" }], // the join that fires
79+
"lineage": "Y_DNA", // Y_DNA | MT_DNA
80+
"surname": "Kane", // single token; never a given name
81+
"originPlace": "Creegh South, Co. Clare, Ireland", // as recorded; normalized server-side
82+
"originCountry": "Ireland",
83+
"birthYear": 1830,
84+
"deathYear": 1908,
85+
"lat": 52.75, // 2dp
86+
"lon": -9.43,
87+
"createdAt": "2026-08-06T…Z"
88+
}
89+
```
90+
91+
**Place text is published as recorded and normalized in the AppView**, not at the edge. One
92+
implementation, fixable without a client release, and re-runnable over records already ingested. The
93+
normalizer is a pure function in `du_db::place` with a country/admin synonym table — the corpus needs
94+
it: `Ireland` / `Republic of Ireland` / `ireland`; `UK` / `United Kingdom` / `Scotland`; `Co. Cork`
95+
vs `Cork`; `VA` vs `Virginia`; UK postcodes embedded mid-string (`Moulin, Pitlochry PH16 5EP, UK`).
96+
705 distinct raw admin strings across the reference corpus.
97+
98+
## 4. Resolving a record to a placed sample
99+
100+
The obvious join — `core.biosample.atproto->>'uri' = biosample_ref`, as `discovery.rs:185` does —
101+
**matches nothing**: zero placed samples carry an at-uri, because the tips were bulk-loaded rather
102+
than federated.
103+
104+
The working key already exists. **All 7,548 placed bigy tips carry an `FTDNA` row in
105+
`core.biosample_identifier`** (migrations 0059/0060, built precisely to "match a re-published donor
106+
to its existing biosample"). So resolution is `(namespace, value)` against that table, with the
107+
at-uri as a fallback for genuinely federated samples. No re-federation, no new identity work.
108+
109+
## 5. The view
110+
111+
`/ytree/node/:name/origins` (+ the mt sibling), a server-rendered inline SVG — no client layout
112+
library, matching `tree_layout.rs`.
113+
114+
- **Geometry**: depth → y; each node a rect spanning its subtree's horizontal extent; children flush
115+
against the parent's underside, so containment carries descent and no connector is drawn.
116+
- **A block shows its equivalent SNPs, and its height is their count** — one line each, nothing
117+
elided, exactly as the Big Tree draws it. The mutations on a branch are unordered, so the list
118+
*is* the block. Vertical position is therefore cumulative: how far down a block sits is the
119+
mutations accrued along the path to it, and the left gutter rules that off in SNPs.
120+
121+
**Do not size blocks by the age model.** Both obvious forms were tried against real data and both
122+
fail:
123+
124+
1. *A node's own `formed_ybp` → its own `tmrca_ybp`.* These are independent point estimates under
125+
no monotonicity constraint; they agree with the parent's TMRCA on only **898 of 10,252 edges**,
126+
and **4,243 (41%)** have the child forming *earlier* than its parent's split — so children
127+
draw on top of their parents. `R-A13318` (formed 1622) landed at exactly its parent
128+
`R-S764`'s y.
129+
2. *Parent TMRCA → own TMRCA.* Monotone, so containment holds — but degenerate:
130+
`formed_ybp == tmrca_ybp` on **41% of terminal branches and 26.5% of internal ones**, collapsing
131+
the branch to a point. On R-DF85 at depth 4 that left **30 of 75 blocks unable to show a single
132+
one of their SNPs**`R-BY18328` got 3px of span for 9 mutations, `R-BY170664` 16 SNPs in an
133+
18px sliver.
134+
135+
SNP count never degenerates, and it is still a time axis: measured on this tree, branch length
136+
tracks SNP count at **r = 0.975, ≈69 years per mutation**. Ages keep their two real jobs — gating
137+
the view to the genealogical era, and labelling each block — they simply do not drive geometry,
138+
because a per-branch estimate is precisely what is missing or degenerate when a block most needs
139+
a height.
140+
141+
- **Colour belongs to the men, not the branches.** An early cut tinted each block by the composition
142+
of its descendants' origins. That asserts something the data does not support: a branch has no
143+
locality, only the men standing on it do, and a modal-origin tint reads as a claim about the whole
144+
lineage. Colour lives on each man's box, keyed to his own MDKA; the legend and table carry the
145+
composition that explains those colours.
146+
- **Tips**: one leaf box per placed sample carrying an origin — `Kane · Co. Clare`, coloured by his
147+
own locality. Never the kit id. A man too narrow to label is counted rather than drawn as an
148+
unreadable sliver, and the count is stated.
149+
- **"No locality recorded" is a visible category**, in the legend and the table, never bare
150+
background. A view of who published is not a view of where a branch is from, and the difference
151+
must be on screen.
152+
- **Colours**: categorical, colourblind-safe, legible in both themes; assigned by frequency rank
153+
*within the rendered subtree* (deterministic, tie-broken on name), top N distinct + a neutral
154+
"other".
155+
- **Era gate**: serves nodes with `tmrca_ybp <= 1500` (adjustable within bounds). Above the cutoff it
156+
renders the breadcrumb, one line of explanation, and links down to eligible children rather than
157+
drawing a block that means nothing.
158+
- **Every branch in the window is drawn, origins or not.** Origins are an overlay on the tree, not
159+
a filter of it: a branch with no locality data is still part of the clade's shape, and hiding it
160+
would misrepresent the phylogeny to make a sparse overlay look dense. Legibility is bounded by
161+
the depth selector instead (default 4 levels), which folds rather than drops — folded branches
162+
are marked, and their men still count in their nearest drawn ancestor.
163+
- **A man gets a box only where his own branch is drawn.** Attributed upward from a folded branch
164+
he still counts in the composition, but is not given a box under a branch that is not his; on
165+
R-DF85 one boundary block had otherwise absorbed 179 men and stacked them 90 rows deep. Drilling
166+
in draws him where he belongs.
167+
- **De-novo nodes stay hidden but their men still count.** A sample placed on an auto-named node is
168+
attributed to the nearest named ancestor, as the public tree already does for sample tips.
169+
Dropping it instead made every band above it understate its own composition.
170+
- **No silent caps.** Pruned branches, samples with no published origin, and the placed total are
171+
all stated on the page.
172+
173+
## 6. Phasing
174+
175+
1. `du_db::place` normalizer — pure, unit-tested. *(No wire format, nothing published.)*
176+
2. Migration 0074 + `fed::ancestral_origin` + the jetstream arm + every gate in §2.
177+
3. `du_db::origins` aggregate + `origins_layout` + route + template + i18n.
178+
4. **Deferred, Navigator:** the lexicon's client half — `AncestralOriginRecord`, the surname
179+
splitter, and the publish predicate (the workspace holds primary data for the subject **and**
180+
`ftdna_member.publicly_shares = 1` or there is no roster row). Measured on the reference
181+
workspace: 583 Y MDKA rows sit on subjects with primary data, 558 of them publicly-sharing —
182+
**the 25 that are not must never publish.**
183+
184+
## 7. Open items
185+
186+
- **The precision ladder (§2.3)** is a proposed default, not a derived rule. It withholds place-level
187+
detail for the ~57% of MDKA rows with no birth year.
188+
- **The 1900 ceiling** is a round number, not a legal standard. Cheap to set now, expensive to lower
189+
once records exist.
190+
- **Retraction.** A withdrawn consent needs the PDS record deleted *and* the mirror tombstoned. The
191+
jetstream `delete` path (`jetstream.rs:163`) is the mechanism; the workflow is unspecified.
192+
- **The view ships empty** until the Navigator half lands. That is a consequence of §2's bulk-load
193+
exclusion, not a defect.
194+
- **mtDNA** costs almost nothing extra (the lexicon is lineage-keyed). Ship Y first and validate there.
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
//! Mirrored ancestral-origin records (`com.decodingus.atmosphere.ancestralOrigin`) — one
2+
//! lineage's most distant known ancestor: surname, origin, dates. The locality substrate for the
3+
//! genealogical-era origins icicle. See [`super`] for the shared cursor/delete, and
4+
//! `proposals/ancestral-origin-icicle.md` for the design.
5+
//!
6+
//! **This layer is pure storage.** The privacy gates that make these records publishable —
7+
//! single-token surname, `birth_year <= 1900`, country-only without a birth year, coordinates
8+
//! coarsened — run in the consumer (`du_jobs::jetstream::build_ancestral_origin`) *before* a row
9+
//! reaches here, matching how every other `fed.*` module leaves record-shape extraction to
10+
//! du-jobs. A row in this table has already passed them.
11+
12+
use super::Common;
13+
use crate::DbError;
14+
use serde_json::Value;
15+
use sqlx::PgPool;
16+
17+
/// A mirrored ancestral origin, post-gate. Every field beyond the envelope is optional because
18+
/// the precision ladder legitimately produces a country-only record.
19+
pub struct AncestralOrigin {
20+
pub common: Common,
21+
/// at-uri of the parent biosample record — present only for genuinely federated samples.
22+
pub biosample_ref: Option<String>,
23+
/// `[{namespace, value}]` verbatim — the join key that actually fires for tree tips.
24+
pub external_ids: Value,
25+
pub lineage: Option<String>,
26+
pub surname: Option<String>,
27+
pub origin_place: Option<String>,
28+
pub origin_country: Option<String>,
29+
pub birth_year: Option<i32>,
30+
pub death_year: Option<i32>,
31+
/// Coarsened to 2dp by the consumer. `None` when the record carried no usable coordinate.
32+
pub lat: Option<f64>,
33+
pub lon: Option<f64>,
34+
}
35+
36+
pub async fn upsert(pool: &PgPool, o: &AncestralOrigin) -> Result<(), DbError> {
37+
sqlx::query(
38+
"INSERT INTO fed.ancestral_origin \
39+
(did, rkey, at_uri, cid, biosample_ref, external_ids, lineage, surname, \
40+
origin_place, origin_country, birth_year, death_year, geocoord, record_created_at, time_us) \
41+
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, \
42+
CASE WHEN $13::float8 IS NULL OR $14::float8 IS NULL THEN NULL \
43+
ELSE ST_SetSRID(ST_MakePoint($14::float8, $13::float8), 4326) END, \
44+
$15,$16) \
45+
ON CONFLICT (did, rkey) DO UPDATE SET \
46+
at_uri = EXCLUDED.at_uri, cid = EXCLUDED.cid, biosample_ref = EXCLUDED.biosample_ref, \
47+
external_ids = EXCLUDED.external_ids, lineage = EXCLUDED.lineage, \
48+
surname = EXCLUDED.surname, origin_place = EXCLUDED.origin_place, \
49+
origin_country = EXCLUDED.origin_country, birth_year = EXCLUDED.birth_year, \
50+
death_year = EXCLUDED.death_year, geocoord = EXCLUDED.geocoord, \
51+
record_created_at = EXCLUDED.record_created_at, time_us = EXCLUDED.time_us, \
52+
indexed_at = now() \
53+
WHERE EXCLUDED.time_us >= fed.ancestral_origin.time_us",
54+
)
55+
.bind(&o.common.did)
56+
.bind(&o.common.rkey)
57+
.bind(&o.common.at_uri)
58+
.bind(&o.common.cid)
59+
.bind(&o.biosample_ref)
60+
.bind(&o.external_ids)
61+
.bind(&o.lineage)
62+
.bind(&o.surname)
63+
.bind(&o.origin_place)
64+
.bind(&o.origin_country)
65+
.bind(o.birth_year)
66+
.bind(o.death_year)
67+
.bind(o.lat)
68+
.bind(o.lon)
69+
.bind(o.common.record_created_at)
70+
.bind(o.common.time_us)
71+
.execute(pool)
72+
.await?;
73+
Ok(())
74+
}

rust/crates/du-db/src/fed/mod.rs

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ use chrono::{DateTime, Utc};
1818
use sqlx::PgPool;
1919

2020
pub mod analytics;
21+
pub mod ancestral_origin;
2122
pub mod core;
2223
pub mod coverage;
2324
pub mod device_key;
@@ -45,6 +46,7 @@ pub const NS_INSTRUMENT_OBSERVATION: &str = "com.decodingus.atmosphere.instrumen
4546
pub const NS_PRIVATE_VARIANT: &str = "com.decodingus.atmosphere.privateVariant";
4647
pub const NS_DEVICE_KEY: &str = "com.decodingus.atmosphere.deviceKey";
4748
pub const NS_FEED_POST: &str = "com.decodingus.atmosphere.feed.post";
49+
pub const NS_ANCESTRAL_ORIGIN: &str = "com.decodingus.atmosphere.ancestralOrigin";
4850

4951
/// Every collection mirrored for reporting (the consumer's `wantedCollections`).
5052
pub const INGEST_COLLECTIONS: &[&str] = &[
@@ -61,6 +63,7 @@ pub const INGEST_COLLECTIONS: &[&str] = &[
6163
NS_PRIVATE_VARIANT,
6264
NS_DEVICE_KEY,
6365
NS_FEED_POST,
66+
NS_ANCESTRAL_ORIGIN,
6467
];
6568

6669
/// The `fed.*` reporting table backing a collection, or `None` if unsupported.
@@ -79,6 +82,7 @@ fn table_for(collection: &str) -> Option<&'static str> {
7982
NS_PRIVATE_VARIANT => "fed.private_variant",
8083
NS_DEVICE_KEY => "fed.device_key",
8184
NS_FEED_POST => "fed.feed_post",
85+
NS_ANCESTRAL_ORIGIN => "fed.ancestral_origin",
8286
_ => return None,
8387
})
8488
}

rust/crates/du-db/src/haplogroup.rs

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1209,3 +1209,34 @@ pub async fn pathway(pool: &PgPool, called_name: &str, dna_type: DnaType) -> Res
12091209
.collect();
12101210
Ok(Pathway { dna_type, called_name: called_name.to_string(), resolved_name: Some(resolved), steps })
12111211
}
1212+
1213+
/// Defining-SNP names for a set of nodes, in one query — the block contents for the origins
1214+
/// icicle, which draws a whole subtree at once and must not issue a query per branch.
1215+
///
1216+
/// Unnamed variants (`canonical_name IS NULL` — folded legacy homoplasy/duplicate rows) are
1217+
/// excluded, matching [`merge_candidates`]: they name nothing a reader could look up.
1218+
/// Names come back sorted so a block's contents are stable between requests.
1219+
pub async fn variant_names_for(
1220+
pool: &PgPool,
1221+
ids: &[i64],
1222+
) -> Result<std::collections::HashMap<i64, Vec<String>>, DbError> {
1223+
use std::collections::HashMap;
1224+
if ids.is_empty() {
1225+
return Ok(HashMap::new());
1226+
}
1227+
let rows: Vec<(i64, String)> = sqlx::query_as(
1228+
"SELECT hv.haplogroup_id, v.canonical_name FROM tree.haplogroup_variant hv \
1229+
JOIN core.variant v ON v.id = hv.variant_id \
1230+
WHERE hv.valid_until IS NULL AND hv.haplogroup_id = ANY($1) \
1231+
AND v.canonical_name IS NOT NULL \
1232+
ORDER BY hv.haplogroup_id, v.canonical_name",
1233+
)
1234+
.bind(ids)
1235+
.fetch_all(pool)
1236+
.await?;
1237+
let mut out: HashMap<i64, Vec<String>> = HashMap::new();
1238+
for (id, name) in rows {
1239+
out.entry(id).or_default().push(name);
1240+
}
1241+
Ok(out)
1242+
}

rust/crates/du-db/src/lib.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,10 @@ pub mod job_lock;
3131
pub mod merge;
3232
pub mod naming;
3333
pub mod notification;
34+
pub mod origins;
3435
pub mod pagination;
3536
pub mod pdf;
37+
pub mod place;
3638
pub mod proposal;
3739
pub mod publication;
3840
pub mod recruitment;

0 commit comments

Comments
 (0)