From 6c08f16adfb04569cdec11636a3b154e80293e00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Martynas=20Jusevi=C4=8Dius?= Date: Mon, 26 Oct 2026 10:25:14 +0100 Subject: [PATCH] Add SPARQL skills: write-ltlod-sparql (mechanics) + investigate-ltlod (recipes) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two agent skills that package how to query the LTLOD datasets, which CLAUDE.md (ETL/dev) and EXAMPLES.md (worked queries) don't: - write-ltlod-sparql — query mechanics, independent of the question: the named-graph- per-document data shape, how to run a query (the throttled live /sparql endpoint vs. a local TDB2 store built from the TriG — the preferred path for anything bulk), and the query-shape rules that decide empty-vs-right and fast-vs-hung (one GRAPH block per document, no union graph, no alt-path with unbound subject in a block, no per-row FILTER NOT EXISTS over the store, read values via triples not IRI string-parsing, --base always). Triggers on writing/debugging/optimizing any LTLOD SPARQL, incl. authoring app views and example queries. - investigate-ltlod — turns an investigative question into the right query: the join-key map (JAR code / asmens_id / AR codes / calendar connect the registers as the same IRI), where each domain lives, recipes (follow a company's public money; a politician's interests vs. contracts/funds; a supplier below the procurement radar; campaign donors), and the caveats that change a conclusion (matchMethod precision, suppressed <3 cells, links that dangle until the full JAR loads, the unmodeled hash-keyed gaps). Builds on and points to write-ltlod-sparql; references EXAMPLES.md / URI-SCHEME.md rather than copying. Co-Authored-By: Claude Opus 4.8 --- .claude/skills/investigate-ltlod/SKILL.md | 82 ++++++++++++++++ .claude/skills/write-ltlod-sparql/SKILL.md | 109 +++++++++++++++++++++ 2 files changed, 191 insertions(+) create mode 100644 .claude/skills/investigate-ltlod/SKILL.md create mode 100644 .claude/skills/write-ltlod-sparql/SKILL.md diff --git a/.claude/skills/investigate-ltlod/SKILL.md b/.claude/skills/investigate-ltlod/SKILL.md new file mode 100644 index 0000000..47abfaf --- /dev/null +++ b/.claude/skills/investigate-ltlod/SKILL.md @@ -0,0 +1,82 @@ +--- +name: investigate-ltlod +description: Answer an investigative or journalistic question from the LTLOD (Lithuanian Linked Open Data) datasets — who controls or funds what, how much public money a company or person receives, and which hidden connections the linked registers reveal. Use when asked to follow a company's public money, check a politician's declared interests against contracts or funds, find suppliers below the procurement radar, trace campaign donors, or otherwise connect the Lithuanian registers; it names the domains and join keys to use and the caveats that decide whether a finding is solid. +--- + +# Investigate LTLOD + +Turn an investigative **question** into the right query. For the mechanics of writing and +running the query (named-graph model, `run.sh` vs. the throttled endpoint, the query-shape +rules), load **`write-ltlod-sparql`** — every recipe here produces a query that must obey it. +49 worked queries with explanations are in **`queries/EXAMPLES.md`**; the URI map is in +**`etl/URI-SCHEME.md`**. Start from the closest example, don't write blank. + +## The join keys (this is the whole point) + +The datasets are separate registers; their investigative power is the keys that connect them. +**The same key is the same IRI in every domain, so joins need no lookup.** + +| Key | IRI | Connects (predicate → the shared IRI) | +|---|---|---| +| **JAR code** (9-digit company code) | `legal-entities/{code}/#this` | procurement buyer/contractor (`epo:playedBy`), taxpayer (`ltlod:taxpayer`), Sodra employer (`ltlod:employer`), campaign donor (`ltlod:donor`), EU-fund beneficiary/partner (`ltlod:beneficiary`/`ltlod:fundingPartner`), insolvent company (`ltlod:insolvent`), invoice buyer/supplier (`ltlod:spendingBuyer`/`ltlod:spendingSupplier`), land applicant (`ltlod:applicant`), VTEK interest org (`ltlod:interestOrganization`), lobbyist/agent (`ltlod:agent`) | +| **`asmens_id`** (Seimas person id) | `persons/{asmens_id}/#this` | Seimas members, officials, VMI/PINREG declarants, VRK candidates, lobbied contacts, membership/vote history | +| **AR code** | `admin-units/{code}/#this` | the municipality/county of an observation or location (`sdmx-dimension:refArea`, `schema:location`) | +| **calendar day/year** | `calendar/{yyyy}[/{mm}/{dd}]/#this` | every dated event/period (`dct:temporal`) — "what happened on/in …" | +| **Wikidata QID** | via `owl:sameAs` | aligned persons and places | + +So *"which company links procurement and EU funds?"* is just: the same `legal-entities/{code}/#this` +is the object of `epo:playedBy` in one graph and `ltlod:beneficiary` in another. + +## Where to look (container → what it holds) + +`legal-entities/` JAR register · `persons/` Seimas members & officials · `procurement-contracts/` +VPT contracts · `taxes/` VMI arrears + taxes paid · `employment/` Sodra employer stats · +`eu-funds/` 2014-2020 EU structural funds · `insolvency-cases/` bankruptcies/restructurings · +`public-spending/` public-sector invoice aggregates · `land-acquisitions/` NŽT land applications · +`campaign-finance/` + `election-results/` VRK · `interest-declarations/` PINREG private interests · +`lobbying-declarations/` + `lobbyists/` VTEK SKAIDRIS · `declarations/` VMI asset declarations · +`seimas` votes/bills/sittings · `admin-units/` + `calendar/` + `facilities/`. + +## Recipes (question → approach → start from) + +- **Follow a company's public money.** Join `legal-entities/{code}/#this` across procurement + (`epo:playedBy`), taxes (`ltlod:taxpayer` — arrears/paid), eu-funds (`ltlod:beneficiary`), + public-spending (`ltlod:spendingSupplier`), insolvency (`ltlod:insolvent`). → `eu-funds-and-procurement.rq`, + `taxes-contractors-with-arrears.rq`, `contractors-in-insolvency.rq`. +- **A politician's declared interests vs. public money.** VTEK `ltlod:interestOrganization` → + the same JAR IRI as a procurement contractor / eu-fund beneficiary / invoice supplier. Filter the + tie `?role skos:notation` to ownership/control (`akcininkas, dalininkas, savininkas, vadovas, + valdybos-narys, …`). → `vtek-interest-procurement.rq`, `pinreg-ties-eu-funds.rq`, + `pinreg-ties-public-spending.rq`. +- **A supplier below the procurement radar.** public-spending suppliers billed a lot via many + small invoices (group per supplier, avg invoice under the threshold) who hold **no** procurement + contract. → `suppliers-under-threshold.rq`. +- **Campaign donors and what they got.** campaign-finance `ltlod:donor` → procurement contractor / + later insolvent. → `finance-donors-procurement.rq`, `donors-later-insolvent.rq`. +- **New companies / land.** land-acquisitions `ltlod:applicant` × legal-entities registration date + × Sodra headcount. → `land-new-companies.rq`. +- **How a member acted vs. lobbying / their wealth.** seimas votes × VTEK lobbying / VMI + declarations, by `asmens_id`. → `lobbying-then-vote.rq`, and the story `app/stories/seimo-turtas-2024.ttl`. + +## Caveats that change the conclusion (state these in any finding) + +- **Name-based person links are marked.** Where a source has no shared code with `persons/` + (VMI/PINREG declarants, VRK candidates, lobbying contacts, officials), the link carries + `ltlod:matchMethod` → `name-and-office` (name + an office held then — strong) or `exact-name` + (name only — weaker; audited precision in CLAUDE.md). **Filter to `name-and-office`** for a solid + person claim; treat `exact-name` as a lead. **Code joins (JAR, `asmens_id`, AR, aligned QID) are + exact by construction** and carry no method. +- **Natural persons are counts, never named.** Land (and NMA farm subsidies) publish natural-person + figures only as per-cell aggregates, with **cells under 3 suppressed**. Don't infer individuals. +- **Links dangle until the full JAR is loaded.** A `legal-entities/{code}/#this` link is correct + even when the name doesn't resolve — the full register (`entities.trig`) must be loaded + (`make -C etl/legal-entities entities`) for names and `rov:*` details. "No name" ≠ "no match". +- **Snapshots and windows.** Tax **arrears** are the *current* snapshot (not the state at a past + contract date); **procurement** is the last 12 months ≥ 10 000 EUR; so "owes tax now" / "has no + contract" are as-of the data date, not history. +- **Known gaps — don't over-claim.** EU-fund **amounts** exist for only ~3 % of records (the + ~685 coded projects; the rest are code-less instrument payouts). EU-fund **repayments and + irregularities**, and **NMA farm-subsidy** applicant identities, are keyed by salted/unpublished + **hashes** that don't join to any public code — they are *not in the data*. The **2021-2027** EU + programme is not published. Public-sector invoice flows at the top by value are **state financial + transfers**, not purchases — filter to small-average-invoice suppliers for real procurement-like spend. diff --git a/.claude/skills/write-ltlod-sparql/SKILL.md b/.claude/skills/write-ltlod-sparql/SKILL.md new file mode 100644 index 0000000..10e271e --- /dev/null +++ b/.claude/skills/write-ltlod-sparql/SKILL.md @@ -0,0 +1,109 @@ +--- +name: write-ltlod-sparql +description: Write, debug or speed up a SPARQL query against the LTLOD datasets — the named-graph-per-document data shape, how to run a query (the throttled live endpoint vs. a local TDB2 store), and the query-shape rules that decide whether a query returns the right rows or hangs. Use whenever a query over the Lithuanian Linked Open Data returns empty or wrong results, is slow, needs a cross-document join, or is being authored for an app view (app/*.ttl) or an example (queries/*.rq). +--- + +# Write LTLOD SPARQL + +How to write a **correct, fast** query against the LTLOD datasets, independent of what +question you are asking. For *which* domains and keys answer a given investigative +question, use the `investigate-ltlod` skill — it builds on this one. + +## The data shape (read this first) + +**Entity per named graph.** Every document is one named graph whose URI is the document +URI `{base}{container}/{slug}/`; the entity is `{graph}#this`, and secondary entities are +`#fragment`s **in the same graph** (a person's memberships, their intervals and instants; +a contract's value and role fragments; a proceeding's `#period`/`#start`/`#end`). So: + +- **One `GRAPH ?g { … }` block per document.** Put `#this` and all its `#fragment`s in + one block; join *across* documents by sharing a variable between blocks. +- There is **no default graph** of instance data — everything is in named graphs. A query + with no `GRAPH` wrapper matches nothing. +- `alignments.trig` / `photos.trig` / `*-geo.trig` reuse the entity graph names, so they + merge with the main data on load — query them as if they were one graph. + +The full URI map (containers, slugs, `#fragment` types, which source each comes from) is in +**`etl/URI-SCHEME.md`**. 49 worked, explained queries are in **`queries/EXAMPLES.md`** — copy +the closest one rather than starting blank. + +## Running a query + +### Local, over the TriG files — preferred for development and anything bulk +`queries/run.sh` bulk-loads **all** `datasets/current/*/*.trig` into a throwaway TDB2 store +and runs the query against it (TDB2 answers an unbound `GRAPH ?g` from its quad indexes; +in-memory `arq` walks every one of ~190k named graphs and is unusable for GRAPH queries): + +```shell +queries/run.sh queries/some-query.rq # one query, CSV/table to stdout +queries/run.sh queries/some-query.rq --results=csv # pass extra tdb2.tdbquery args +BASE=https://linkeddata.lt/ queries/run.sh q.rq # override base (default: prod) +``` + +It rebuilds the store **every call** (~20 s for the committed data, minutes with the bulk +datasets). **For many queries, build the store once and reuse it:** + +```shell +J=/Users/martynas/WebRoot/apache-jena-5.6.0 # 5.6.0 or 6.x both query fine +"$J/bin/riot" --base=https://linkeddata.lt/ --syntax=trig --output=nquads \ + datasets/current/*/*.trig > /tmp/all.nq +"$J/bin/tdb2.tdbloader" --loc /tmp/tdb /tmp/all.nq +"$J/bin/tdb2.tdbquery" --loc /tmp/tdb --query q.rq --results=csv # repeat, fast +``` + +Load only the domains a query touches to save time (e.g. `datasets/current/procurement/*.trig +datasets/current/insolvency/*.trig datasets/current/legal-entities/*.trig` for a +procurement×insolvency join — plus the taxonomies if the query resolves a `skos:prefLabel`). + +**Datasets must be present.** Committed `.trig` are in the repo; the bulk ones (the full JAR +`legal-entities/entities.trig`, `procurement`, `seimas/votes`, `eu-funds`, `public-spending`, +`taxes/taxpayers`, …) are **gitignored** and built with `make -C etl ` (see CLAUDE.md +Commands). A query that joins a missing bulk domain returns empty, not an error. + +**Always pass `--base`.** Committed TriG is base-**relative** with no `@base`; without a base, +relative IRIs resolve to `file://…` garbage and joins silently fail. + +### Live endpoint — `https://linkeddata.lt/sparql` — throttled, for a few interactive queries only +```shell +curl -sG https://linkeddata.lt/sparql --data-urlencode "query=$(cat q.rq)" -H "Accept: text/csv" +``` +It is **rate-limited by nginx** (default `2r/s`, burst 10; 2 connections per client, 8 total) +and answers **`429`** beyond that, keyed on the client address. Fine for a handful of queries; +**do not** loop or bulk-query it — use `run.sh` / a local TDB2 for that. (On the local stack, +RFC-1918/loopback clients are exempt, so you won't see 429 there.) + +## Query-shape rules (these decide empty vs. right, fast vs. hung) + +- **One `GRAPH ?g` block per document; join across blocks on shared variables.** A block is + evaluated on its own, so a `BIND`/`FILTER EXISTS` inside it **cannot see** a variable bound + outside it (it silently matches everything) — put such a test *after* the block, with the + `GRAPH` inside the `EXISTS`. +- **A property path that crosses documents can't sit in one block** — split it into hops + (e.g. settlement → county is a UNION of the 2- and 3-hop `dct:isPartOf+` chains). +- **No alternative path `(a|b)` with an unbound subject inside a `GRAPH` block** — ARQ won't + resolve it through the quad indexes. Use `?p` + `FILTER(?p IN (…))`, or a `UNION` of the two + plain triples. +- **Never `FROM `** (Fuseki/ARQ-specific) — use standard `GRAPH` patterns. +- **Read a value from a triple, never by string-parsing an IRI.** No `REPLACE(STR(?x), …)` / + `STRAFTER`/`SUBSTR(STR(?x), …)` to pull a code or label out of a resource URI: a concept's + code is `skos:notation`, its label `skos:prefLabel` (filter `lang`), a legal entity's code + `dct:identifier`. String ops on genuine *literals* are fine. Matching a *known* resource + base-agnostically with `FILTER(STRENDS(STR(?x), "/…/#this"))` is OK (the TriG is base-relative, + so a bare absolute IRI wouldn't match) — that selects a resource, it doesn't extract a value. +- **Avoid per-row `FILTER NOT EXISTS` over the whole store** — it is quadratic and hangs. + Group/aggregate first so the check runs over a few thousand rows, or dump both URI sets with + two SELECTs and `comm -23`. +- **Dates link to the calendar.** Events carry `dct:temporal` → `calendar/{yyyy}/{mm}/{dd}/#this` + (or the year doc) plus a literal; periods are a `time:Interval` with `time:Instant`s that each + carry `dct:temporal`. Join on the day/year doc for "what happened on/in …". +- **Labels / language.** Lithuanian labels are `@lt`; filter `FILTER(LANG(?l) = "lt")`. A legal + entity's name is `skos:prefLabel`/`rov:legalName`; a person's is `foaf:name`. + +## When a query returns empty — checklist +1. Is every pattern inside a `GRAPH ?g { … }`? (No default graph.) +2. Is the joined **bulk** domain actually loaded? (`entities.trig`, `procurement`, the new + domains, and the **taxonomies** a `skos:prefLabel` join needs — e.g. `insolvency-types`.) +3. Did you pass `--base` and the same base everywhere? +4. Does a cross-document property path need splitting into hops? +5. Is a value being read by IRI string-parsing (returns nothing if the slug scheme differs) + instead of from `skos:notation`/`dct:identifier`?