[Mahalaxmibejugam]

Summary (follow-up to #1054)

Follow-up to #1054. It extends caller-framework telemetry to ExtendedGcsFileSystem (the class gcsfs.GCSFileSystem exports by default) and to ZonalFile. It also strengthens the telemetry tests and adds real end-to-end telemetry tests to the Cloud Build e2e pipeline.

Changes

Telemetry coverage

• mirror_gcs_methods(ExtendedGcsFileSystem) wraps Extended's own sync and async overrides (mv, rm, mkdir, makedirs, rmdir, merge, _cat_file, _find, _pipe_file, _put_file, …). These previously went through plain asyn.sync_wrapper, so their requests lacked the framework token and ran stack detection on the fsspecIO thread for every request.
• _sync_lookup_bucket_layout and sync_process_limits_to_offset_and_length now use _gcs_sync_wrapper.
• wrap_file_methods(ZonalFile) wraps ZonalFile's own overrides (_fetch_range, _async_fetch_range, write, flush, commit, …).
• New with_telemetry_context decorator on ExtendedGcsFileSystem._open. The bucket-layout lookup, and for Rapid (zonal) reads the MRD pool's JSON API _info request in ZonalFile.__init__, run before GCSFile.__init__ caches caller_framework. Previously that _info request went out without the token, and detection ran on the fsspecIO thread. The framework is now detected once on the caller thread, and GCSFile.__init__ reuses it.
• core.py reads telemetry_manager.default_usage_tracker from the module, so tests only need to patch one place.

Tests

• Coverage test: it no longer pre-sets the telemetry context. Plain fsspec already carries a pre-set context into the loop thread, so the old version passed even without the wrappers. It now uses a detector that only finds a framework on the caller thread and runs on both GCSFileSystem and ExtendedGcsFileSystem. It also asserts detection never runs on the fsspec loop thread.
• Wrapped-methods test: it derives the methods to check from the class instead of a hardcoded list. Any sync function that wraps a coroutine must be _gcs_sync_wrapper, and every async method mirror_gcs_methods targets, along with its public sync counterpart, must be wrapped.
• ZonalFile tests:
  • every method defined on ZonalFile is wrapped;
  • _async_fetch_range and write carry the file's framework;
  • over a zonal file's full life (open, read, seek, read, info, close), detection runs exactly once on the caller thread, and every HTTP request and gRPC-side operation carries the framework.
• Test harness (mock_gcs_harness):
  • parametrized over core/extended;
  • restores the fsspec registry exactly;
  • mocks batch deletes;
  • skips credential refresh, since HTTP is mocked;
  • clears the instance cache on teardown.
• Real-framework tests (test_upstream_frameworks_real.py): they run on both classes, and Lightning writes to tmp_path.
• New test_telemetry_e2e.py: real pandas (to_csv/read_csv), torch DataLoader reads, and Lightning save_checkpoint against a GCS endpoint with no mocking. The User-Agent of every request to the GCS host is recorded at the aiohttp layer, and each one must carry the framework token.
  • Runs against real GCS in the Cloud Build standard (regional) and hns suites, which run gcsfs/, and against the fake emulator in GitHub CI.
  • Rapid/zonal buckets are intentionally not covered: their data path uses gRPC bidi APIs, which don't go through the HTTP User-Agent path.

CI / e2e

• environment_gcsfs.yaml: adds pandas, pytorch-cpu and lightning. Previously the real-framework tests were always skipped in CI; now they actually run.
• cloudbuild/setup_vm.sh: installs CPU-only torch from download.pytorch.org, plus pandas and lightning. The installs are non-fatal because the tests importorskip these packages.

Docs

• telemetry.rst now states the 64-frame detection cap, says detection runs on the caller thread, explains that the opt-out env vars are read on every call, and notes that gRPC (zonal) data-path requests don't carry the framework token yet.

Testing

• Full suite against the fake GCS emulator, rebased on main (after feat(telemetry): integrate upstream framework telemetry into User-Agent header (3/3) #1054 and refactor(extended_gcsfs): replace BucketType with orthogonal BucketLayout and routing helpers #1133):
  • GCSFS_EXPERIMENTAL_ZB_HNS_SUPPORT=false: 754 passed.
  • GCSFS_EXPERIMENTAL_ZB_HNS_SUPPORT=true: 1104 passed. The only failures are 3 test_hns_unit.py::test_get_control_plane_client_endpoint cases, a local mTLS environment issue that also fails on main.
• Cloud Build e2e (before the rebase): all suites passed, and test_telemetry_e2e.py passed against real GCS in both the standard and hns suites.
• Checked that the tests fail when the feature breaks:
  • removing mirror_gcs_methods(ExtendedGcsFileSystem), wrap_file_methods(ZonalFile) or @with_telemetry_context on ExtendedGcsFileSystem._open each makes the corresponding tests fail;
  • with GCSFS_NO_TELEMETRY=1, all e2e tests fail with N/N GCS requests missing fw/<framework>.
• pre-commit (black, isort, flake8) passes.

Notes for reviewers

• Not yet verified: conda resolving pytorch-cpu and lightning in GitHub CI. Expect a slower environment setup.
• Pre-existing CI issue (not changed here): ci.yml uses ${{ matrix.PY }}, but the matrix key is python-version, so every matrix job actually runs Python 3.11 from environment_gcsfs.yaml.
• Extended on the emulator: GitHub CI only runs with GCSFS_EXPERIMENTAL_ZB_HNS_SUPPORT=false. On the emulator, Extended is covered by the mocked harness, not by test_telemetry_e2e.py.
• Follow-up: adding the framework token to gRPC calls (per-call metadata).

[gemini-code-assist]

Code Review

This pull request introduces lightweight, automatic caller framework detection (telemetry) to identify upstream libraries like Pandas, PyTorch, and Lightning interacting with Google Cloud Storage via gcsfs. It includes documentation, environment updates, and extensive integration and end-to-end tests. The feedback highlights an issue in the VM setup script where a failure to install the CPU-only torch wheel would cause a subsequent lightning installation to pull down the massive CUDA-enabled torch wheel from PyPI, defeating the optimization. A conditional installation approach was suggested to resolve this.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.59%. Comparing base (76012a9) to head (59f5714).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1140      +/-   ##
==========================================
+ Coverage   92.51%   92.59%   +0.08%     
==========================================
  Files          20       20              
  Lines        4086     4091       +5     
==========================================
+ Hits         3780     3788       +8     
+ Misses        306      303       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Mahalaxmibejugam]

/gcbrun

[zhixiangli]

LGTM. One small docs nit inline.

[zhixiangli]

nit: gRPC requests without the token go beyond Rapid data-path ones. RCU reads also go to ZonalFile/MRD (_open, is_rcu). Storage Control calls (bucket layout lookup, HNS mkdir/rmdir/mv folder ops) use a fixed grpc.primary_user_agent too. Suggest:

Suggested change

	The framework token is attached to JSON API (HTTP) requests. gRPC data-path requests to Rapid (zonal) buckets
	use a ``User-Agent`` fixed when the gRPC client is created and do not yet carry the framework token.
	The framework token is attached to JSON API (HTTP) requests only. gRPC requests use a ``User-Agent`` fixed
	when the gRPC client is created and do not yet carry the framework token. This includes Rapid (zonal) reads
	and writes, Rapid Cache Ultra reads, and Storage Control API calls (bucket layout lookups and HNS folder
	operations such as ``mkdir``, ``rmdir`` and ``mv``).

[zhixiangli]

nit: same here. Rapid Cache Ultra reads also use the gRPC data path, so maybe say "Rapid (zonal) and Rapid Cache Ultra buckets are intentionally not covered".