feat: compute emissions factors for JD proxy allocation

docs/architecture.md

@@ -44,7 +44,7 @@ flowchart LR
 - **Harmonize** — align the ingested sources onto a common grid, one 10° tile at a time, yielding one lazy `xarray.Dataset` per tile backed by a zarr.
 - **Emit** — compute per-pixel land conversion events and emissions as a lazy dask/xarray graph and persist to zarr.
 - **Attribute** — clip per-pixel emissions to jurisdiction polygons and crop masks, rolling up to per-(jurisdiction, crop) totals via the jurisdictional-direct, statistical or jurisdictional-direct proxy leg.
-- **Trace** — join the attribution rollups to production (NASS yields for the direct leg, MapSPAM for the statistical leg) to produce the final emissions-factor table. The proxy leg has no production yet.
+- **Trace** — join the attribution rollups to production (NASS yields for the direct leg, MapSPAM for the statistical leg, FAOSTAT's national production split by CROPGRIDS crop area for the proxy leg) to produce the final emissions-factor table.
 
 ## Compute platform and libraries
 

docs/coverage.md

@@ -303,3 +303,30 @@ The jurisdictional-direct pipeline (USA only) covers the following 11 crops, map
 | SUGARBEET | SUGARBEETS                              | 157 (Sugar beet)            |
 | SUGARCANE | SUGARCANE                               | 156 (Sugar cane)            |
 | WHEAT     | DURUM_WHEAT, SPRING_WHEAT, WINTER_WHEAT | 15 (Wheat)                  |
+
+### Jurisdictional direct proxy
+
+The proxy pipeline covers all 173 CROPGRIDS crops, each published under a FAOSTAT item with FAOSTAT's name for it. Most follow the CROPGRIDS publisher's crop table (`CROPS_matching_v17.xlsx`). Where that table names no item, or a retired one, FAOSTAT's current list gives flax (771), green broad beans (420) and jute-like fibres (782), and our judgement gives pimento (689, dry chillies and peppers) and popcorn (56, with maize). Several crops may share an item: maize and popcorn are both 56, and kapok fibre and kapok seed both 310. The full crosswalk is the proxy leg's entry in `jdluc/taxonomy.py`. Livestock is as in the statistical leg.
+
+FAOSTAT has no item for the 16 fodder crops, so each takes a code of our own and the name the CROPGRIDS publisher gives it. They carry emissions and area, but no production and so no emissions factor.
+
+| Crop         | Published as                           |
+| ------------ | -------------------------------------- |
+| ALFALFA      | -101 (Alfalfa)                         |
+| BEETFOR      | -102 (Beets for fodder)                |
+| CABBAGEFOR   | -103 (Cabbage for fodder)              |
+| CARROTFOR    | -104 (Carrots for fodder)              |
+| CLOVER       | -105 (Clover)                          |
+| FORNES       | -106 (Forage products, other)          |
+| GRASSNES     | -107 (Grasses, other)                  |
+| LEGUMENES    | -108 (Legumes, other)                  |
+| MAIZEFOR     | -109 (Maize for forage and silage)     |
+| MIXEDGRASS   | -110 (Mixed grasses and legumes)       |
+| OILSEEDFOR   | -111 (Green oilseeds for fodder)       |
+| RYEFOR       | -112 (Rye grass for forage and silage) |
+| SORGHUMFOR   | -113 (Sorghum for forage and silage)   |
+| SWEDEFOR     | -114 (Swedes for fodder)               |
+| TURNIPFOR    | -115 (Turnips for fodder)              |
+| VEGETABLENES | -116 (Vegetables and roots for fodder) |
+
+Eleven FAOSTAT crop items have no CROPGRIDS crop. Seven are co-products counted in the crop they come from: palm kernels, kapokseed, kapok fibre, cotton seed, cotton lint, cassava leaves and coir. Four are crops CROPGRIDS does not map, whose production has no area to be split by and is left out: jojoba seeds, tallowtree seeds, leeks and other alliaceous vegetables, and other pome fruits.

docs/data.md

@@ -83,31 +83,31 @@ emissions = xarray.open_zarr("<scratch-root>/<cache-key>.zarr", consolidated=Fal
 
 Deposited per data version as `emissions-factors.parquet` (see [Access](#access)).
 
-The final per-(jurisdiction, commodity) table produced by the `trace` stage. Indexed by `(admin_level, admin_id, commodity_id, methodology)`, where `admin_level` is `PROVINCIAL` (World Bank admin-1) or `NATIONAL` (admin-0, summed from provincial rows), `methodology` is `JURISDICTIONAL_DIRECT` or `STATISTICAL`, and `commodity_id` is an integer: the FAOSTAT item code where FAOSTAT has an item, and a negative code of our own where it does not. `commodity_name` holds FAOSTAT's item name (`Maize (corn)`, `Soya beans`), or our own name for a negative code. Both legs use the same codes, so the same crop has the same `commodity_id` in each. Where MapSPAM splits one FAOSTAT item in two, the statistical leg publishes one row: arabica and robusta coffee add together as 656 (`Coffee, green`), and pearl and small millet as 79 (`Millet`). The statistical leg's `OTHER_OILCROPS` (-201) and `OTHER_PULSES` (-202) are MapSPAM groups of several FAOSTAT items, so they take codes of our own. [`coverage.md`](coverage.md#crops) lists each leg's crops with their codes.
-
-Four rows are not crops. Beef, 867 (`Meat of cattle with the bone, fresh or chilled`), is a livestock commodity: it carries its share of pasture's emissions and area, and production like a crop's. `PASTURE` (-3) keeps the rest of pasture's share, with emissions and area but no production. `DROPPED` (-1) and `UNATTRIBUTED` (-2) appear in both legs and are not commodities at all. `UNATTRIBUTED` holds emissions that count toward the jurisdiction's total but that none of the commodities we model takes, so the commodity rows plus `UNATTRIBUTED` add up to that total. `DROPPED` holds carbon released where, as far as our maps show, the land became neither cropland nor pasture; it does not count toward the total and comes on top of it. `PASTURE`, `DROPPED` and `UNATTRIBUTED` have no production figure, so none of them has an emissions factor. See "Rows that name no commodity" in [`methodology.md`](methodology.md).
-
-| Column                             | Type    | Units        | Description                                                                                                                      |
-| ---------------------------------- | ------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------- |
-| `jurisdiction_name`                | string  | —            | Display name (e.g. `United States of America \| Iowa`)                                                                           |
-| `commodity_name`                   | string  | —            | FAOSTAT's item name for a positive `commodity_id`, our own name for a negative one                                               |
-| `commodity_hectares`               | float64 | ha           | Area the commodity occupies (land occupation), on production's discounted 2000–2020 window                                       |
-| `peatland_commodity_hectares`      | float64 | ha           | Commodity area on peatland                                                                                                       |
-| `emissions_mt`                     | float64 | t CO₂e       | Total allocated LUC emissions                                                                                                    |
-| `peatland_occupation_emissions_mt` | float64 | t CO₂e       | Annual peatland-occupation emissions on the commodity's pixels                                                                   |
-| `forest_emissions_mt`              | float64 | t CO₂e       | Forest-conversion emissions                                                                                                      |
-| `grassland_emissions_mt`           | float64 | t CO₂e       | Grassland-conversion emissions                                                                                                   |
-| `peatland_conversion_emissions_mt` | float64 | t CO₂e       | Peatland-conversion (drainage-pulse) emissions                                                                                   |
-| `production_kg`                    | float64 | kg           | Production: crops from NASS yield × area (direct leg) or MapSPAM (statistical leg); livestock as FAOSTAT carcass weight, bone in |
-| `yield_kg_per_ha`                  | float64 | kg/ha        | `production_kg / commodity_hectares`                                                                                             |
-| `emissions_factor_kgco2e_per_kg`   | float64 | kg CO₂e / kg | `1000 × emissions_mt / production_kg` — the headline factor                                                                      |
-
-Both attribution legs emit the same columns.
+The final per-(jurisdiction, commodity) table produced by the `trace` stage. Indexed by `(admin_level, admin_id, commodity_id, methodology)`, where `admin_level` is `PROVINCIAL` (World Bank admin-1) or `NATIONAL` (admin-0, summed from provincial rows), `methodology` is `JURISDICTIONAL_DIRECT`, `STATISTICAL` or `JURISDICTIONAL_DIRECT_PROXY`, and `commodity_id` is an integer: the FAOSTAT item code where FAOSTAT has an item, and a negative code of our own where it does not. `commodity_name` holds FAOSTAT's item name (`Maize (corn)`, `Soya beans`), or our own name for a negative code. All three legs use the same codes, so the same crop has the same `commodity_id` in each. Where MapSPAM splits one FAOSTAT item in two, the statistical leg publishes one row: arabica and robusta coffee add together as 656 (`Coffee, green`), and pearl and small millet as 79 (`Millet`). The statistical leg's `OTHER_OILCROPS` (-201) and `OTHER_PULSES` (-202) are MapSPAM groups of several FAOSTAT items, so they take codes of our own, as do the proxy leg's 16 fodder crops (-101 to -116), which FAOSTAT has no item for. [`coverage.md`](coverage.md#crops) lists each leg's crops with their codes.
+
+Four rows are not crops. Beef, 867 (`Meat of cattle with the bone, fresh or chilled`), is a livestock commodity: it carries its share of pasture's emissions and area, and production like a crop's. `PASTURE` (-3) keeps the rest of pasture's share, with emissions and area but no production. `DROPPED` (-1) and `UNATTRIBUTED` (-2) appear in every leg and are not commodities at all. `UNATTRIBUTED` holds emissions that count toward the jurisdiction's total but that none of the commodities we model takes, so the commodity rows plus `UNATTRIBUTED` add up to that total. `DROPPED` holds carbon released where, as far as our maps show, the land became neither cropland nor pasture; it does not count toward the total and comes on top of it. `PASTURE`, `DROPPED`, `UNATTRIBUTED` and the fodder crops have no production figure, so none of them has an emissions factor. See "Rows that name no commodity" in [`methodology.md`](methodology.md).
+
+| Column                             | Type    | Units        | Description                                                                                                                                                                   |
+| ---------------------------------- | ------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `jurisdiction_name`                | string  | —            | Display name (e.g. `United States of America \| Iowa`)                                                                                                                        |
+| `commodity_name`                   | string  | —            | FAOSTAT's item name for a positive `commodity_id`, our own name for a negative one                                                                                            |
+| `commodity_hectares`               | float64 | ha           | Area the commodity occupies (land occupation), on production's discounted 2000–2020 window; CROPGRIDS' 2020 area for the proxy leg                                            |
+| `peatland_commodity_hectares`      | float64 | ha           | Commodity area on peatland                                                                                                                                                    |
+| `emissions_mt`                     | float64 | t CO₂e       | Total allocated LUC emissions                                                                                                                                                 |
+| `peatland_occupation_emissions_mt` | float64 | t CO₂e       | Annual peatland-occupation emissions on the commodity's pixels                                                                                                                |
+| `forest_emissions_mt`              | float64 | t CO₂e       | Forest-conversion emissions                                                                                                                                                   |
+| `grassland_emissions_mt`           | float64 | t CO₂e       | Grassland-conversion emissions                                                                                                                                                |
+| `peatland_conversion_emissions_mt` | float64 | t CO₂e       | Peatland-conversion (drainage-pulse) emissions                                                                                                                                |
+| `production_kg`                    | float64 | kg           | Production: crops from NASS yield × area (direct leg), MapSPAM (statistical leg) or FAOSTAT split by CROPGRIDS area (proxy leg); livestock as FAOSTAT carcass weight, bone in |
+| `yield_kg_per_ha`                  | float64 | kg/ha        | `production_kg / commodity_hectares`                                                                                                                                          |
+| `emissions_factor_kgco2e_per_kg`   | float64 | kg CO₂e / kg | `1000 × emissions_mt / production_kg` — the headline factor                                                                                                                   |
+
+All three legs emit the same columns.
 
 ```python
 import pandas
 
 emission_factors = pandas.read_parquet("emissions-factors.parquet")
 ```
 
-The `attribute` stage writes an intermediate rollup parquet indexed by `(admin_level, admin_id, commodity_name, methodology)`, where `commodity_name` is the leg's own name for the commodity (a CDL crop for the direct leg, a MapSPAM crop for the statistical leg, `BEEF_CATTLE` for beef), with the non-derived subset of these columns (`commodity_hectares`, `peatland_commodity_hectares`, `emissions_mt`, `peatland_occupation_emissions_mt`, `forest_emissions_mt`, `grassland_emissions_mt`, `peatland_conversion_emissions_mt`, and — statistical only — `production_mt` and the herd's heads at each MapSPAM snapshot, `herd_heads_2000` to `herd_heads_2020`); `trace` maps those names onto `commodity_id`, adding together the rows that share one, joins production and appends the ratio columns (`production_kg`, `yield_kg_per_ha`, `emissions_factor_kgco2e_per_kg`).
+The `attribute` stage writes an intermediate rollup parquet indexed by `(admin_level, admin_id, commodity_name, methodology)`, where `commodity_name` is the leg's own name for the commodity (a CDL crop for the direct leg, a MapSPAM crop for the statistical leg, a CROPGRIDS crop for the proxy leg, `BEEF_CATTLE` for beef), with the non-derived subset of these columns (`commodity_hectares`, `peatland_commodity_hectares`, `emissions_mt`, `peatland_occupation_emissions_mt`, `forest_emissions_mt`, `grassland_emissions_mt`, `peatland_conversion_emissions_mt`, and the herd's heads: for the statistical leg `production_mt` and the heads at each MapSPAM snapshot, `herd_heads_2000` to `herd_heads_2020`, and for the proxy leg the heads in 2020, `herd_heads_2020`); `trace` maps those names onto `commodity_id`, adding together the rows that share one, joins production and appends the ratio columns (`production_kg`, `yield_kg_per_ha`, `emissions_factor_kgco2e_per_kg`).

docs/executive_summary.md

@@ -19,15 +19,15 @@ This document is aimed at readers assessing how the methodology maps to the LSRS
 
 ## Data output
 
-The published deliverable is a single per-(jurisdiction, commodity) table — the **emissions-factors parquet** — indexed by `(admin_level, admin_id, commodity_id, methodology)`, where `admin_level` is `NATIONAL` or `PROVINCIAL`, `admin_id` is the ISO 3166 alpha-3 code nationally and the World Bank admin-1 code provincially, `commodity_id` is the FAOSTAT item code (or a negative code of our own where FAOSTAT has none) with its name in `commodity_name`, and `methodology` is `JURISDICTIONAL_DIRECT` or `STATISTICAL`. Each row carries the headline emissions factor and the quantities behind it; both attribution legs (jdLUC and sLUC) emit the same columns. The full schema is in [`data.md`](data.md#3-emissions-factors-parquet).
-
-| Column                           | Units        | What it is                                                                                                 |
-| -------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------- |
-| `emissions_factor_kgco2e_per_kg` | kg CO₂e / kg | The headline factor — `1000 × emissions_mt / production_kg`                                                |
-| `emissions_mt`                   | t CO₂e       | Total allocated LUC emissions                                                                              |
-| `production_kg`                  | kg           | Production: crops from NASS × area (jdLUC) or MapSPAM (sLUC); livestock as FAOSTAT carcass weight, bone in |
-| `commodity_hectares`             | ha           | Commodity area in the jurisdiction, over 2000–2020                                                         |
-| `yield_kg_per_ha`                | kg/ha        | `production_kg / commodity_hectares`                                                                       |
+The published deliverable is a single per-(jurisdiction, commodity) table — the **emissions-factors parquet** — indexed by `(admin_level, admin_id, commodity_id, methodology)`, where `admin_level` is `NATIONAL` or `PROVINCIAL`, `admin_id` is the ISO 3166 alpha-3 code nationally and the World Bank admin-1 code provincially, `commodity_id` is the FAOSTAT item code (or a negative code of our own where FAOSTAT has none) with its name in `commodity_name`, and `methodology` is `JURISDICTIONAL_DIRECT`, `STATISTICAL` or `JURISDICTIONAL_DIRECT_PROXY`. Each row carries the headline emissions factor and the quantities behind it; all three legs emit the same columns. The full schema is in [`data.md`](data.md#3-emissions-factors-parquet).
+
+| Column                           | Units        | What it is                                                                                                                                          |
+| -------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `emissions_factor_kgco2e_per_kg` | kg CO₂e / kg | The headline factor — `1000 × emissions_mt / production_kg`                                                                                         |
+| `emissions_mt`                   | t CO₂e       | Total allocated LUC emissions                                                                                                                       |
+| `production_kg`                  | kg           | Production: crops from NASS × area (jdLUC), MapSPAM (sLUC) or FAOSTAT split by CROPGRIDS area (proxy); livestock as FAOSTAT carcass weight, bone in |
+| `commodity_hectares`             | ha           | Commodity area in the jurisdiction, over 2000–2020                                                                                                  |
+| `yield_kg_per_ha`                | kg/ha        | `production_kg / commodity_hectares`                                                                                                                |
 
 ## Known deviations and open items