examples: fail loudly when GCP_PROJECT_ID is unset - #131
Merged
Merged
Conversation
Four places in examples/bitcoin-tracker silently fell back to a placeholder project id instead of stopping. The placeholder cannot reach a real Google Cloud project, so nothing was writing to a stranger's dataset, but the shape is the one that bit us in docs/walkthrough/gcp.md, where the default was a real-looking id feeding straight into bigquery.Client(project=...). ingest_bitcoin_prices.py was already remediated. The other three scripts now match it: no default, a message naming GCP_PROJECT_ID, and exit 1. - load_bitcoin_price_batch.py: guard in __main__, plus the missing sys import. - runtime/ingest_bitcoin_prices.py: the project id was resolved after the CoinGecko fetch, so the old code did a network round trip before it could notice. The guard now runs first, before any work. - airflow-quickstart.sh: it exported the placeholder into the environment, which would have defeated the parse-time check in the DAG below by setting the variable to garbage rather than leaving it unset. - AIRFLOW_INTEGRATION.md: the documented DAG kept the silent fallback in three spots while the shipped airflow/dags/bitcoin_tracker_enhanced.py already required the variable. The snippet now mirrors the shipped DAG. Verified by running each script under stubbed google.cloud.bigquery and requests that log every network call and client construction. Unset: exit 1, zero logged events. Set: proceeds and targets the given project. The same harness run against the pre-fix code exits 0 and reaches BQ_INSERT against the placeholder, so the check can go red.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four places in
examples/bitcoin-trackersilently fell back to a placeholder project id instead of stopping. The placeholder cannot reach a real Google Cloud project, so nothing was writing to a stranger's dataset. The shape is what matters: it is the same one that appeared indocs/walkthrough/gcp.md, where the default was a real-looking id feeding straight intobigquery.Client(project=...).examples/bitcoin-tracker/ingest_bitcoin_prices.pywas already remediated. The other three scripts now match it: no default, a message namingGCP_PROJECT_ID, and a non-zero exit. None of these are Cloud Function handlers, so the 500-response shape did not apply; all four are scripts or a DAG.What changed
load_bitcoin_price_batch.pyos.getenv(..., "<<YOUR_PROJECT_HERE>>")__main__, plus the missingsysimportruntime/ingest_bitcoin_prices.pymain(), before any workairflow-quickstart.sh${GCP_PROJECT_ID:-<<YOUR_PROJECT_HERE>>}AIRFLOW_INTEGRATION.mdTwo of these are worth calling out. The runtime script resolved the project id after the network round trip, so the old code always hit CoinGecko before it could notice the variable was missing. And
airflow-quickstart.shexported the placeholder into the environment, which would have defeated the parse-time check inairflow/dags/bitcoin_tracker_enhanced.pyby setting the variable to garbage rather than leaving it unset.Verification
Each script was run under stubbed
google.cloud.bigqueryandrequeststhat log every network call and client construction to a sentinel file.argv[1]): exit 0, proceeds, targets the given project.NETWORK_CALL, thenBQ_CLIENT_CONSTRUCTEDandBQ_INSERTagainst<<YOUR_PROJECT_HERE>>. The check can go red, so the green above means something.Gates, all exit 0:
scripts/check_cli_docs.py,scripts/check_providers.py,npm run docs:build,scripts/check-dist-links.mjs(107,738 references across 219 pages clean).Swept but not changed
grep -rn 'os.getenv(\|os.environ.get(' examples/turned up three more two-argument defaults that are real-looking identifiers rather than obvious placeholders. I left them, because they resolve inside the reader's own authenticated account and cannot route data to a third party, which is the harm that motivated this change. Flagging them for a maintainer rather than deciding unilaterally:bitcoin-price-tracker-0.7.1-snowflake/runtime/ingest.py:171defaultsSNOWFLAKE_ROLEtoSYSADMIN. Silently running as a high-privilege role is the one I would most consider changing.SNOWFLAKE_WAREHOUSEtoCOMPUTE_WH, the Snowflake trial default. Already fails loudly atUSE WAREHOUSE.bitcoin-price-tracker-0.7.1-aws-athena/runtime/ingest.py:77defaultsAWS_DEFAULT_REGIONtoeu-central-1. Worth a look for data-residency reasons.Also noted, outside the fix criterion:
ingest_iceberg.py:66returns the unresolved{{ env.VAR }}template as a bucket name, where its siblingingest.pylogs a warning first. It fails at AWS rather than silently, but the sibling is stricter.Nothing outside
examples/was touched.docs/walkthrough/gcp.mdalready has no default.