Repository navigation
Add public-sector invoice spending domain (buyer-supplier-year aggregates) - #85
Merged
Merged
Conversation
…ates)
New opt-in etl/public-spending/ domain: public-sector invoice totals per
(buyer, supplier, year) from the State Treasury (NBFC) open data
(nbfc/viesojo_sektoriaus_saskaitos). ltlod-public-spending streams ~14.4M invoices
(Saskaitos) and ~32M party rows (SaskaituSalys), keyset-paginated, into a temporary
on-disk SQLite db and aggregates there (the party rows cannot be windowed by invoice
year, so the whole table is read — ~40-90 min). Join: SaskaituSalys.sf_id =
Saskaitos.sf_id normalized to an LT prefix; tipas Pirkėjas is the buyer, Tiekėjas the
supplier, each by validus_jar_kodas. Party names are pseudonymised in the source, so
names come from legal-entities/ by the code.
One ltlod:SpendingAggregate per document (public-spending/{buyer}-{supplier}-{year}):
ltlod:spendingBuyer/spendingSupplier -> legal-entities/{code}/#this, dct:temporal ->
the calendar year, measures ltlod:invoiceCount / ltlod:billedTotal (incl. VAT) /
ltlod:billedNet. At the default MIN_TOTAL=1000 EUR floor (drops the ~63% zero-total
credit-note/VAT pairs and the trivial tail) the set is ~432k documents; years are
restricted to the calendar's 1977-2027.
app/ns.ttl: the class + the two party links (each with its own inverse view, buyer /
supplier) + the measure properties. app/public-spending.ttl container with a top-flows
chart (no children listing). SHACL in etl/shapes/public-spending.ttl. NOT in `make all`
(the ~46M-row fetch is heavy); aggregates.trig is gitignored bulk output (~418 MB).
Two cross-domain cases joined on the JAR code: suppliers paid via many sub-threshold
invoices with no procurement contract, and politicians' companies billed by the public
sector. Repayments/irregularities are not modelled here; see the eu-funds note for the
hash-key limitation that also applies conceptually.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
app/stories/ pages (like seimo-turtas-2024) that narrate the investigative cases the company-side Tier-1 domains open up, each with live charts pinned to the current data: - rangovai-skolos-bankrotai — public contractors that owe overdue tax or are in the insolvency register (procurement × taxes × insolvency). - es-fondai-politiku-imones — EU structural funds to companies tied to politicians, and companies drawing both EU funds and public contracts (eu-funds × PINREG × procurement). - saskaitu-skaidymas — suppliers paid large sums via many sub-threshold invoices with no procurement contract (public-spending × procurement). All figures and company links are verified against the loaded data. Listed automatically under the Stories container (app/stories.ttl ChildrenView); installed with `make install`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Final Tier 1 source. A new opt-in
etl/public-spending/domain — public-sector invoice totals per (buyer, supplier, year) from the State Treasury (NBFC) open data (nbfc/viesojo_sektoriaus_saskaitos), joined tolegal-entities/by the bare JAR code. Opens the sub-threshold / "money past the procurement register" cases.Scale & method
Saskaitos) + ~32M party rows (SaskaituSalys). The party rows can't be windowed by invoice year (theirdatais a 2020-era registration date), soltlod-public-spendingstreams both feeds (keyset-paginated) into a temporary on-disk SQLite db and aggregates there — low memory, ~40–90 min fetch.SaskaituSalys.sf_id=Saskaitos.sf_idnormalized to anLTprefix;tipasPirkėjas = buyer, Tiekėjas = supplier, each byvalidus_jar_kodas. Party names are pseudonymised in the source, so names come fromlegal-entities/.Model
ltlod:SpendingAggregateper document (public-spending/{buyer}-{supplier}-{year}/):ltlod:spendingBuyer/ltlod:spendingSupplier→legal-entities/{code}/#this,dct:temporal→ the calendar year, measuresltlod:invoiceCount/ltlod:billedTotal(incl. VAT) /ltlod:billedNet.app/ns.ttl: the class + both party links (each with its own inverse view — buyer / supplier) + the measures.app/public-spending.ttlcontainer with a top-flows chart (no children listing for the ~432k docs). SHACL inetl/shapes/public-spending.ttl.MIN_TOTAL=1000EUR floor (like procurement'sMIN_VALUE): drops the ~63% zero-total credit-note/VAT pairs and the trivial tail → ~432,570 documents (full set ~3.9M). Years restricted to the calendar's 1977–2027 (28 malformed-date rows dropped).Verification
legal-entities/dangle until the full JAR register is loaded — expected.make all(the ~46M-row fetch is heavy);aggregates.trigis gitignored bulk output (~418 MB).EXAMPLES.mdupdated with the two sections only.🤖 Generated with Claude Code