[yuxin00j]

Summary

Detects Rapid Cache Ultra (RCU) buckets from the rapid_cache_info field returned by GetStorageLayout, and reads from them through the gRPC Multi-Range Downloader (MRD) path that zonal buckets already use. In this PR, writes to RCU buckets still go through GCSFile over HTTP. gRPC appendable writes will come in a separate PR.

Changes

• Layout detection (gcsfs/extended_gcsfs.py):
  • _get_bucket_layout maps rapid_cache_info.cache_type to a CachingTier: "rapid-cache-ultra" becomes RAPID_CACHE_ULTRA, "rapid-cache" becomes RAPID_CACHE, and a missing, empty, or unrecognized value becomes NONE.
• Read routing (gcsfs/extended_gcsfs.py):
  • _cat_file, _cat_ranges, _get_file_request, _get_file_concurrent and _get_threshold_for_disk_reads now check _uses_grpc_data_path instead of _is_zonal_bucket, so RCU buckets read through MRD. Buckets with a plain Rapid Cache (RAPID_CACHE) stay on HTTP.
  • _open returns a ZonalFile for "rb" on RCU buckets and keeps GCSFile for all other modes.
• Dependencies (pyproject.toml): require google-cloud-storage-control>=1.14.0, the first release with StorageLayout.rapid_cache_info.
• Tests:
  • gcsfs/tests/test_hns_unit.py: real-StorageLayout-proto detection cases for empty, unrecognized, rapid-cache and rapid-cache-ultra cache types on flat and HNS buckets, plus HNS routing coverage on RCU layouts.
  • gcsfs/tests/test_zonal_unit.py: runs the existing MRD read tests (cat_file, cat_ranges, get_file, the prefetcher and MRD error handling) against flat and HNS RCU layouts, and covers _get_threshold_for_disk_reads on gRPC and HTTP layouts, _open class routing across wb/ab/xb and HTTP/Rapid Cache/UNKNOWN layouts, cat_ranges fallback on non-gRPC layouts, and +-mode rejection without checking out an MRD pool.
  • gcsfs/tests/test_zonal.py: tests that force the MRD read path now mock _uses_grpc_data_path instead of _is_zonal_bucket.
  • gcsfs/tests/conftest.py, gcsfs/tests/test_core.py: a new skip_if_grpc_read fixture skips the test_core.py tests that check HTTP read internals when TEST_BUCKET is a zonal or RCU bucket.

[gemini-code-assist]

Code Review

This pull request refactors bucket type detection by replacing the BucketType enum with a detailed BucketLayout dataclass, which tracks storage class, caching tier, and namespace layout. This allows Rapid Cache Ultra (RCU) buckets to utilize the gRPC data path (ZonalFile) for read operations. The review feedback correctly identifies that read-write modes (such as r+) should be excluded from using ZonalFile for RCU buckets, as writes are not yet supported over the gRPC path.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.25%. Comparing base (c783299) to head (600085a).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1134      +/-   ##
==========================================
+ Coverage   92.24%   92.25%   +0.01%     
==========================================
  Files          20       20              
  Lines        4047     4055       +8     
==========================================
+ Hits         3733     3741       +8     
  Misses        314      314              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[zhixiangli]

Thanks! I ran the read path on real buckets with google-cloud-storage-control==1.14.0:

• yuxinj-us-central1-standard-rcu → STANDARD_RCU
• yuxinj-us-central1-hns-rcu → HIERARCHICAL_RCU
• chandrasiri-benchmarks-rb → plain RAPID_CACHE

RCU open("rb"), cat_file, cat_ranges and the 5 MiB disk threshold all go through MRD and return correct data. wb/ab/xb on RCU still return GCSFile and round-trip over HTTP. Plain Rapid Cache stays on HTTP.

One real issue: read-write modes on RCU buckets leak an MRD pool. Details and a repro are in a reply on the existing _open thread. The two inline comments here are test-coverage suggestions.

[zhixiangli]

Two more things from re-running against the real RCU buckets at aa46c4b. Layout detection still matches the server (rapid-cache-ultra / rapid-cache), and RCU reads over MRD return correct data.

test_core.py read-path tests vs RCU (non-blocking). skip_if_zonal checks is_zonal, so it doesn't skip RCU, but RCU reads no longer use HTTP. If TEST_BUCKET were an RCU bucket, these would fail (I replayed the assertions on the real bucket):

• test_cat_file_routing_and_thresholds: _cat_file_sequential called 0 times, expected 1
• test_cat_file_explicit_concurrency: _cat_file_concurrent called 0 times, expected 1
• test_user_agent_includes_cache_type_and_source_in_read: no cache_type/mmap:e User-Agent (its inline check also uses _is_zonal_bucket)

Suggest a skip_if_grpc_read fixture keyed on uses_grpc_data_path for these three. Keep skip_if_zonal for the write and precondition tests, since RCU writes stay on HTTP. CI doesn't run test_core.py on RCU today, so this is latent.

Nits:

• test_get_threshold_for_disk_reads_http_buckets_delegates_to_super is missing layouts.HIERARCHICAL_RAPID_CACHE and BucketLayout.UNKNOWN, which test_cat_ranges_non_zonal_delegates_to_fsspec already lists.
• test_fetch_range_split_concurrent_{success,exception} in test_zonal.py still patch _is_zonal_bucket, which isn't called there. The neighbouring tests moved to _uses_grpc_data_path.

[zhixiangli]

Inline details for the generation and "Short read" bullets in my earlier review summary. Both were reproduced on yuxinj-us-central1-standard-rcu at aa46c4b.

[yuxin00j]

Done for skip_if_grpc_read and both test nits; the zonal r+ pool leak is fixed in #1139.