[jeffseif]

Background

This provides the second and third supplements for our global crop mask, extending GACED30's coverage over woody perennials -- Descals oil palm brought oil palm and this brings cocoa and coffee. The Forest Data Partnership (FDaP) 2026a release provides probabilities at 10m resolution in one-degree EPSG:4326 tiles for 2018, 2020, 2022, and 2024. We employ several steps of pre-processing, both to streamline downstream usage and to keep the dataset consistent with other datasets:

1. all 1-degree tiles covered by a 10-degree tile are mosaicked at their native resolution, with any overlaps handled
2. thresholds are applied so that each country's mask covers an area consistent with FAOSTAT's area harvested for that year; countries without a threshold use the crop's floor, the model's precision-recall crossover
3. countries are zeroed unless FAOSTAT lists them as producing the crop, and for countries without a threshold, the crop covers at least 1% of their cropland; jurisdictions shown to be primarily false positives are also removed by an exclusion list
4. the 2020 and 2022 masks are smoothed temporally so that a given pixel is included if its value exceeds its threshold or if both the preceding and following year do

The thresholds and exclusions come from extensive manual validation by Ad Astra. That work is expected to be published in the open soon. The thresholds here are final; the exclusions are still pending, and the branch has a TODO to cite the work.

Steps before merging

• aecade2 thresholds for coffee
• d1c4be2 thresholds for cocoa
• cc86a3f producers
• d0ca852 query to regenerate the producer lists
• e5d9e53 use fitted thresholds as given, with the floor only as a fallback
• 0f44a1d provisional exclusions for cocoa and coffee
• Document runtime on a VM within the GCS bucket's region
• final producing countries
• final exclusion countries

Notes

1. This ingest presents heavy pre-processing logic, which can incur significant costs if infrastructure isn't configured properly. For example, to avoid paying egress costs, we plan on backfilling within a VM in the same GCP region as the source data and which writes to a GCS bucket that also resides in that region. Read from outside the region, a single tile costs around ~$20.
2. That pre-processing also depends heavily on external work which hasn't yet been cited and which may not be open-source. Although this presents a potential provenance wall, we may bring the calibration logic behind the thresholds into this repo in the future.
3. The thresholds are final (7 countries for cocoa, 13 for coffee). The exclusion lists are provisional, from Ad Astra's draft: 13 provinces for cocoa and 8 for coffee
4. The producer lists are fixed in code (28 countries for cocoa, 39 for coffee), so ingest never reads FAOSTAT. tools/fdap-producers.sql regenerates them from FAOSTAT's bulk files with DuckDB.
5. The FAOSTAT production dataset was extended to cover coffee and bumped from v1 to v2.
6. Nothing reads the masks yet: harmonizing them and using them in emit follow in the next stacked PR
7. I also split tests by dataset since the original module was feeling crowded
8. The source GeoTIFFs compress really poorly. For the next release, FDaP should consider quantizing (e.g., round(p * 100)::uint8) to a lower cardinality or lowering precision to float16
9. On an n2-standard-8 in us-central1, 10N_010W (57 source cells) took 40 minutes -- 30m for the downloading and mosaicking and 10m for the conversion to COG and uploading. Although n2-standard-16's seems to have higher total bandwidth so that multiple tiles and can ingested concurrently.

Manual validation

• FAOSTAT crop table: v2 is identical to v1 for all existing rows, with the addition of COFF for 78-79 countries (depending on the year), with reasonable data
• Ingest of one on CIV, 10N_010W: uv run python -m jdluc.ingest FDAP_COCOA CIV
• Screenshots of ^^ (~1 GiB versus the ~110 GiB of source data):

[TenaciousThee]

As discussed with Jeff, comments will be addressed then PR will be merged

[TenaciousThee]

As discussed with Jeff, comments will be addressed and PR merged