Skip to content

examples: fail loudly when GCP_PROJECT_ID is unset - #131

Merged
fas89 merged 1 commit into
mainfrom
fix/env-fallback-guards
Sep 17, 2026
Merged

fas89 merged 1 commit into
mainfrom
fix/env-fallback-guards

Conversation

@fas89

@fas89 fas89 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Four places in examples/bitcoin-tracker silently fell back to a placeholder project id instead of stopping. The placeholder cannot reach a real Google Cloud project, so nothing was writing to a stranger's dataset. The shape is what matters: it is the same one that appeared in docs/walkthrough/gcp.md, where the default was a real-looking id feeding straight into bigquery.Client(project=...).

examples/bitcoin-tracker/ingest_bitcoin_prices.py was already remediated. The other three scripts now match it: no default, a message naming GCP_PROJECT_ID, and a non-zero exit. None of these are Cloud Function handlers, so the 500-response shape did not apply; all four are scripts or a DAG.

What changed

File Was Now
load_bitcoin_price_batch.py os.getenv(..., "<<YOUR_PROJECT_HERE>>") guard in __main__, plus the missing sys import
runtime/ingest_bitcoin_prices.py same, resolved after the CoinGecko fetch guard moved to the top of main(), before any work
airflow-quickstart.sh ${GCP_PROJECT_ID:-<<YOUR_PROJECT_HERE>>} fails with a usage message
AIRFLOW_INTEGRATION.md silent fallback in 3 spots mirrors the shipped DAG's parse-time required read

Two of these are worth calling out. The runtime script resolved the project id after the network round trip, so the old code always hit CoinGecko before it could notice the variable was missing. And airflow-quickstart.sh exported the placeholder into the environment, which would have defeated the parse-time check in airflow/dags/bitcoin_tracker_enhanced.py by setting the variable to garbage rather than leaving it unset.

Verification

Each script was run under stubbed google.cloud.bigquery and requests that log every network call and client construction to a sentinel file.

  • Unset: exit 1, correct message, zero logged events. No network call, no client constructed.
  • Set (and via argv[1]): exit 0, proceeds, targets the given project.
  • Control: the same harness against the pre-fix code exits 0, logs NETWORK_CALL, then BQ_CLIENT_CONSTRUCTED and BQ_INSERT against <<YOUR_PROJECT_HERE>>. The check can go red, so the green above means something.

Gates, all exit 0: scripts/check_cli_docs.py, scripts/check_providers.py, npm run docs:build, scripts/check-dist-links.mjs (107,738 references across 219 pages clean).

Swept but not changed

grep -rn 'os.getenv(\|os.environ.get(' examples/ turned up three more two-argument defaults that are real-looking identifiers rather than obvious placeholders. I left them, because they resolve inside the reader's own authenticated account and cannot route data to a third party, which is the harm that motivated this change. Flagging them for a maintainer rather than deciding unilaterally:

  • bitcoin-price-tracker-0.7.1-snowflake/runtime/ingest.py:171 defaults SNOWFLAKE_ROLE to SYSADMIN. Silently running as a high-privilege role is the one I would most consider changing.
  • Same file line 170 defaults SNOWFLAKE_WAREHOUSE to COMPUTE_WH, the Snowflake trial default. Already fails loudly at USE WAREHOUSE.
  • bitcoin-price-tracker-0.7.1-aws-athena/runtime/ingest.py:77 defaults AWS_DEFAULT_REGION to eu-central-1. Worth a look for data-residency reasons.

Also noted, outside the fix criterion: ingest_iceberg.py:66 returns the unresolved {{ env.VAR }} template as a bucket name, where its sibling ingest.py logs a warning first. It fails at AWS rather than silently, but the sibling is stricter.

Nothing outside examples/ was touched. docs/walkthrough/gcp.md already has no default.

Four places in examples/bitcoin-tracker silently fell back to a placeholder
project id instead of stopping. The placeholder cannot reach a real Google
Cloud project, so nothing was writing to a stranger's dataset, but the shape
is the one that bit us in docs/walkthrough/gcp.md, where the default was a
real-looking id feeding straight into bigquery.Client(project=...).

ingest_bitcoin_prices.py was already remediated. The other three scripts now
match it: no default, a message naming GCP_PROJECT_ID, and exit 1.

- load_bitcoin_price_batch.py: guard in __main__, plus the missing sys import.
- runtime/ingest_bitcoin_prices.py: the project id was resolved after the
  CoinGecko fetch, so the old code did a network round trip before it could
  notice. The guard now runs first, before any work.
- airflow-quickstart.sh: it exported the placeholder into the environment,
  which would have defeated the parse-time check in the DAG below by setting
  the variable to garbage rather than leaving it unset.
- AIRFLOW_INTEGRATION.md: the documented DAG kept the silent fallback in three
  spots while the shipped airflow/dags/bitcoin_tracker_enhanced.py already
  required the variable. The snippet now mirrors the shipped DAG.

Verified by running each script under stubbed google.cloud.bigquery and
requests that log every network call and client construction. Unset: exit 1,
zero logged events. Set: proceeds and targets the given project. The same
harness run against the pre-fix code exits 0 and reaches BQ_INSERT against
the placeholder, so the check can go red.
@fas89
fas89 merged commit 4c74fb7 into main Sep 17, 2026
5 checks passed
@fas89
fas89 deleted the fix/env-fallback-guards branch September 17, 2026 17:48

This branch was successfully deployed

1 active deployment
github-pages — 43761e59 Deployed Sep 17, 2026 by fas89 via deploy #165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant