Skip to content

Repository navigation

chore: move STATISTICAL livestock production into attribute, anchor to FAOSTAT ids #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeffseif merged 5 commits into from
Nov 6, 2026
Merged

chore: move STATISTICAL livestock production into attribute, anchor to FAOSTAT ids #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
23ca2eb
Move STATISTICAL's livestock stock/production from attribute to trace…
jeffseif Nov 5, 2026
b1bdf22
Stand up JD's commodity -> FAOSTAT mapping in a new taxonomy module -…
jeffseif Nov 5, 2026
3fea6ad
Stand up S's commodity -> FAOSTAT mapping
jeffseif Nov 5, 2026
f58ed2b
Bring FAOSTAT taxonomy into validation
jeffseif Nov 5, 2026
af2cb8f
update docs
jeffseif Nov 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 54 additions & 54 deletions docs/coverage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,66 +240,66 @@ Taiwan is its own country, `TWN`, following CEDA's convention. The World Bank la

### sLUC (statistical)

The statistical pipeline covers the following 32 crops, identified by IFPRI MapSPAM crop code.
The statistical pipeline covers the following 32 crops, identified by IFPRI MapSPAM crop code. "Published as" is the `commodity_id` and `commodity_name` each crop's rows carry in the emissions-factors table (see [`data.md`](data.md#3-emissions-factors-parquet)): a FAOSTAT item, or a negative code of our own where FAOSTAT has none. The two coffees share 656 and the two millets share 79, so each pair is published as one row.

| Crop | MapSPAM code |
| -------------- | ------------ |
| ARABICA_COFFEE | ACOF |
| BANANA | BANA |
| BARLEY | BARL |
| BEAN | BEAN |
| CASSAVA | CASS |
| CHICKPEA | CHIC |
| COCONUT | CNUT |
| COTTON | COTT |
| COWPEA | COWP |
| GROUNDNUT | GROU |
| LENTIL | LENT |
| MAIZE | MAIZ |
| OILPALM | OILP |
| OTHER_OILCROPS | OOIL |
| OTHER_PULSES | OPUL |
| PEARL_MILLET | PMIL |
| PIGEONPEA | PIGE |
| PLANTAIN | PLNT |
| POTATO | POTA |
| RAPESEED | RAPE |
| RICE | RICE |
| ROBUSTA_COFFEE | RCOF |
| SESAME_SEED | SESA |
| SMALL_MILLET | SMIL |
| SORGHUM | SORG |
| SOYBEAN | SOYB |
| SUGARBEET | SUGB |
| SUGARCANE | SUGC |
| SUNFLOWER | SUNF |
| SWEET_POTATO | SWPO |
| WHEAT | WHEA |
| YAM | YAMS |
| Crop | MapSPAM code | Published as |
| -------------- | ------------ | ----------------------------------- |
| ARABICA_COFFEE | ACOF | 656 (Coffee, green) |
| BANANA | BANA | 486 (Bananas) |
| BARLEY | BARL | 44 (Barley) |
| BEAN | BEAN | 176 (Beans, dry) |
| CASSAVA | CASS | 125 (Cassava, fresh) |
| CHICKPEA | CHIC | 191 (Chick peas, dry) |
| COCONUT | CNUT | 249 (Coconuts, in shell) |
| COTTON | COTT | 328 (Seed cotton, unginned) |
| COWPEA | COWP | 195 (Cow peas, dry) |
| GROUNDNUT | GROU | 242 (Groundnuts, excluding shelled) |
| LENTIL | LENT | 201 (Lentils, dry) |
| MAIZE | MAIZ | 56 (Maize (corn)) |
| OILPALM | OILP | 254 (Oil palm fruit) |
| OTHER_OILCROPS | OOIL | -201 (OTHER_OILCROPS) |
| OTHER_PULSES | OPUL | -202 (OTHER_PULSES) |
| PEARL_MILLET | PMIL | 79 (Millet) |
| PIGEONPEA | PIGE | 197 (Pigeon peas, dry) |
| PLANTAIN | PLNT | 489 (Plantains and cooking bananas) |
| POTATO | POTA | 116 (Potatoes) |
| RAPESEED | RAPE | 270 (Rape or colza seed) |
| RICE | RICE | 27 (Rice) |
| ROBUSTA_COFFEE | RCOF | 656 (Coffee, green) |
| SESAME_SEED | SESA | 289 (Sesame seed) |
| SMALL_MILLET | SMIL | 79 (Millet) |
| SORGHUM | SORG | 83 (Sorghum) |
| SOYBEAN | SOYB | 236 (Soya beans) |
| SUGARBEET | SUGB | 157 (Sugar beet) |
| SUGARCANE | SUGC | 156 (Sugar cane) |
| SUNFLOWER | SUNF | 267 (Sunflower seed) |
| SWEET_POTATO | SWPO | 122 (Sweet potatoes) |
| WHEAT | WHEA | 15 (Wheat) |
| YAM | YAMS | 137 (Yams) |

### sLUC livestock (statistical)

The statistical pipeline also divides pasture's share of each cell among livestock commodities, by livestock units from Global Pasture Watch's headcounts.
The statistical pipeline also divides pasture's share of each cell among livestock commodities, by livestock units from Global Pasture Watch's headcounts. "Published as" is as for the crops.

| Commodity | Grazers |
| ----------- | ---------------------------------------------------------------------------------- |
| BEEF_CATTLE | Cattle |
| PASTURE | The residual: buffalo, goats, horses and sheep, and pasture no grazer is mapped on |
| Commodity | Grazers | Published as |
| ----------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------- |
| BEEF_CATTLE | Cattle | 867 (Meat of cattle with the bone, fresh or chilled) |
| PASTURE | The residual: buffalo, goats, horses and sheep, and pasture no grazer is mapped on | -3 (PASTURE) |

### jdLUC (jurisdictional-direct)

The jurisdictional-direct pipeline (USA only) covers the following 11 crops, mapped to USDA NASS CDL crop classes.
The jurisdictional-direct pipeline (USA only) covers the following 11 crops, mapped to USDA NASS CDL crop classes. "Published as" is as for the statistical crops, and each code is the one the statistical crop of the same name publishes under.

| Crop | USDA NASS CDL class(es) |
| --------- | --------------------------------------- |
| BARLEY | BARLEY |
| BEAN | DRY_BEANS |
| COTTON | COTTON |
| MAIZE | CORN |
| POTATO | POTATOES |
| RICE | RICE |
| SORGHUM | SORGHUM |
| SOYBEAN | SOYBEANS |
| SUGARBEET | SUGARBEETS |
| SUGARCANE | SUGARCANE |
| WHEAT | DURUM_WHEAT, SPRING_WHEAT, WINTER_WHEAT |
| Crop | USDA NASS CDL class(es) | Published as |
| --------- | --------------------------------------- | --------------------------- |
| BARLEY | BARLEY | 44 (Barley) |
| BEAN | DRY_BEANS | 176 (Beans, dry) |
| COTTON | COTTON | 328 (Seed cotton, unginned) |
| MAIZE | CORN | 56 (Maize (corn)) |
| POTATO | POTATOES | 116 (Potatoes) |
| RICE | RICE | 27 (Rice) |
| SORGHUM | SORGHUM | 83 (Sorghum) |
| SOYBEAN | SOYBEANS | 236 (Soya beans) |
| SUGARBEET | SUGARBEETS | 157 (Sugar beet) |
| SUGARCANE | SUGARCANE | 156 (Sugar cane) |
| WHEAT | DURUM_WHEAT, SPRING_WHEAT, WINTER_WHEAT | 15 (Wheat) |
7 changes: 5 additions & 2 deletions docs/data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,11 +83,14 @@ emissions = xarray.open_zarr("<scratch-root>/<cache-key>.zarr", consolidated=Fal

Deposited per data version as `emissions-factors.parquet` (see [Access](#access)).

The final per-(jurisdiction, commodity) table produced by the `trace` stage. Indexed by `(admin_level, admin_id, commodity_name, methodology)` — all strings — where `admin_level` is `PROVINCIAL` (World Bank admin-1) or `NATIONAL` (admin-0, summed from provincial rows) and `methodology` is `JURISDICTIONAL_DIRECT` or `STATISTICAL`. Four `commodity_name` values are not crops. `BEEF_CATTLE` is a livestock commodity: it carries its share of pasture's emissions and area, and production like a crop's. `PASTURE` keeps the rest of pasture's share, with emissions and area but no production. `DROPPED` and `UNATTRIBUTED` appear in both legs and are not commodities at all. `UNATTRIBUTED` holds emissions that count toward the jurisdiction's total but that none of the commodities we model takes, so the commodity rows plus `UNATTRIBUTED` add up to that total. `DROPPED` holds carbon released where, as far as our maps show, the land became neither cropland nor pasture; it does not count toward the total and comes on top of it. `PASTURE`, `DROPPED` and `UNATTRIBUTED` have no production figure, so none of them has an emissions factor. See "Rows that name no commodity" in [`methodology.md`](methodology.md).
The final per-(jurisdiction, commodity) table produced by the `trace` stage. Indexed by `(admin_level, admin_id, commodity_id, methodology)`, where `admin_level` is `PROVINCIAL` (World Bank admin-1) or `NATIONAL` (admin-0, summed from provincial rows), `methodology` is `JURISDICTIONAL_DIRECT` or `STATISTICAL`, and `commodity_id` is an integer: the FAOSTAT item code where FAOSTAT has an item, and a negative code of our own where it does not. `commodity_name` holds FAOSTAT's item name (`Maize (corn)`, `Soya beans`), or our own name for a negative code. Both legs use the same codes, so the same crop has the same `commodity_id` in each. Where MapSPAM splits one FAOSTAT item in two, the statistical leg publishes one row: arabica and robusta coffee add together as 656 (`Coffee, green`), and pearl and small millet as 79 (`Millet`). The statistical leg's `OTHER_OILCROPS` (-201) and `OTHER_PULSES` (-202) are MapSPAM groups of several FAOSTAT items, so they take codes of our own. [`coverage.md`](coverage.md#crops) lists each leg's crops with their codes.

Four rows are not crops. Beef, 867 (`Meat of cattle with the bone, fresh or chilled`), is a livestock commodity: it carries its share of pasture's emissions and area, and production like a crop's. `PASTURE` (-3) keeps the rest of pasture's share, with emissions and area but no production. `DROPPED` (-1) and `UNATTRIBUTED` (-2) appear in both legs and are not commodities at all. `UNATTRIBUTED` holds emissions that count toward the jurisdiction's total but that none of the commodities we model takes, so the commodity rows plus `UNATTRIBUTED` add up to that total. `DROPPED` holds carbon released where, as far as our maps show, the land became neither cropland nor pasture; it does not count toward the total and comes on top of it. `PASTURE`, `DROPPED` and `UNATTRIBUTED` have no production figure, so none of them has an emissions factor. See "Rows that name no commodity" in [`methodology.md`](methodology.md).

| Column | Type | Units | Description |
| ---------------------------------- | ------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| `jurisdiction_name` | string | — | Display name (e.g. `United States of America \| Iowa`) |
| `commodity_name` | string | — | FAOSTAT's item name for a positive `commodity_id`, our own name for a negative one |
| `commodity_hectares` | float64 | ha | Area the commodity occupies (land occupation), on production's discounted 2000–2020 window |
| `peatland_commodity_hectares` | float64 | ha | Commodity area on peatland |
| `emissions_mt` | float64 | t CO₂e | Total allocated LUC emissions |
Expand All @@ -107,4 +110,4 @@ import pandas
emission_factors = pandas.read_parquet("emissions-factors.parquet")
```

The `attribute` stage writes an intermediate rollup parquet with the same index and the non-derived subset of these columns (`commodity_hectares`, `peatland_commodity_hectares`, `emissions_mt`, `peatland_occupation_emissions_mt`, `forest_emissions_mt`, `grassland_emissions_mt`, `peatland_conversion_emissions_mt`, and — statistical only — `production_mt`); `trace` joins production and appends the ratio columns (`production_kg`, `yield_kg_per_ha`, `emissions_factor_kgco2e_per_kg`).
The `attribute` stage writes an intermediate rollup parquet indexed by `(admin_level, admin_id, commodity_name, methodology)`, where `commodity_name` is the leg's own name for the commodity (a CDL crop for the direct leg, a MapSPAM crop for the statistical leg, `BEEF_CATTLE` for beef), with the non-derived subset of these columns (`commodity_hectares`, `peatland_commodity_hectares`, `emissions_mt`, `peatland_occupation_emissions_mt`, `forest_emissions_mt`, `grassland_emissions_mt`, `peatland_conversion_emissions_mt`, and — statistical only — `production_mt` and the herd's heads at each MapSPAM snapshot, `herd_heads_2000` to `herd_heads_2020`); `trace` maps those names onto `commodity_id`, adding together the rows that share one, joins production and appends the ratio columns (`production_kg`, `yield_kg_per_ha`, `emissions_factor_kgco2e_per_kg`).
2 changes: 1 addition & 1 deletion docs/executive_summary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This document is aimed at readers assessing how the methodology maps to the LSRS

## Data output

The published deliverable is a single per-(jurisdiction, commodity) table — the **emissions-factors parquet** — indexed by `(admin_level, admin_id, commodity_name, methodology)`, where `admin_level` is `NATIONAL` or `PROVINCIAL`, `admin_id` is the ISO 3166 alpha-3 code nationally and the World Bank admin-1 code provincially, and `methodology` is `JURISDICTIONAL_DIRECT` or `STATISTICAL`. Each row carries the headline emissions factor and the quantities behind it; both attribution legs (jdLUC and sLUC) emit the same columns. The full schema is in [`data.md`](data.md#3-emissions-factors-parquet).
The published deliverable is a single per-(jurisdiction, commodity) table — the **emissions-factors parquet** — indexed by `(admin_level, admin_id, commodity_id, methodology)`, where `admin_level` is `NATIONAL` or `PROVINCIAL`, `admin_id` is the ISO 3166 alpha-3 code nationally and the World Bank admin-1 code provincially, `commodity_id` is the FAOSTAT item code (or a negative code of our own where FAOSTAT has none) with its name in `commodity_name`, and `methodology` is `JURISDICTIONAL_DIRECT` or `STATISTICAL`. Each row carries the headline emissions factor and the quantities behind it; both attribution legs (jdLUC and sLUC) emit the same columns. The full schema is in [`data.md`](data.md#3-emissions-factors-parquet).

| Column | Units | What it is |
| -------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------- |
Expand Down
4 changes: 2 additions & 2 deletions docs/methodology.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,13 +269,13 @@ Each pixel's emissions are split by the land class it became, cropland or pastur

### From rollups to emissions factors (`trace.py`)

`trace.py` takes the attribution rollup and derives the emissions factor identically for the jurisdictional-direct and statistical methodologies — the only methodology-specific step is where production comes from (NASS yield × area for direct; for statistical, MapSPAM production for crops and GPW heads × FAOSTAT carcass weight per head for livestock). The proxy leg has no production yet:
`trace.py` takes the attribution rollup and derives the emissions factor identically for the jurisdictional-direct and statistical methodologies — the only methodology-specific step is where production comes from (NASS yield × area for direct; for statistical, MapSPAM production for crops and GPW heads × FAOSTAT carcass weight per head for livestock). It then maps each leg's own commodity names onto FAOSTAT item codes (`jdluc/taxonomy.py`), so the legs share one vocabulary, and adds together the rows that share a code, such as MapSPAM's arabica and robusta coffee. The proxy leg has no production yet, and no codes:

```
emissions_factor_kgco2e_per_kg = Σ emissions / Σ production
```

Provincial (admin-1) rows are then summed up to national (admin-0) totals. The final output is a `pandas.DataFrame` indexed by `(admin_level, admin_id, commodity_name, methodology)` with crop hectares, peatland crop hectares, emissions (with the forest, grassland, peatland-conversion and peatland-occupation components broken out), production, and the emissions factor; see `data.md` for the full column schema. The full stage-by-stage pipeline that produces this table, and how its outputs are cached, is described in `architecture.md`.
Provincial (admin-1) rows are then summed up to national (admin-0) totals. The final output is a `pandas.DataFrame` indexed by `(admin_level, admin_id, commodity_id, methodology)` with the commodity's name, crop hectares, peatland crop hectares, emissions (with the forest, grassland, peatland-conversion and peatland-occupation components broken out), production, and the emissions factor; see `data.md` for the full column schema. The full stage-by-stage pipeline that produces this table, and how its outputs are cached, is described in `architecture.md`.

______________________________________________________________________

Expand Down
Loading
Loading