[yuxin00j]

Summary

Writes to Rapid Cache Ultra (RCU) buckets now go through the gRPC appendable writer (AsyncAppendableObjectWriter) that zonal buckets already use, with storage_class="RAPID". An RCU bucket defaults new objects to STANDARD, which appendable writes reject; with RAPID the object is written to the zonal cache, and the server copies it to regional storage once it is finalized. For that reason finalize_on_close now defaults to True on RCU buckets, while zonal buckets keep False. Handling RCU being disabled while a process is running will come in a separate PR.

Depends on #1134. Only the last five commits are new here.

Changes

• Write routing (gcsfs/extended_gcsfs.py, gcsfs/core.py):
  • These check _uses_grpc_data_path instead of _is_zonal_bucket: _put_file, _pipe_file, and the initiate_upload and simple_upload helpers in both modules. Unknown layouts and buckets with a plain Rapid Cache stay on HTTP.
  • _open returns a ZonalFile for all supported read and write modes on RCU buckets ("+" not in mode).
  • The new ExtendedGcsFileSystem._init_aaow opens every appendable writer (for ZonalFile, put_file, pipe_file and the upload helpers). It passes storage_class="RAPID" only for RCU buckets. zb_hns_utils.init_aaow (gcsfs/zb_hns_utils.py) gets a storage_class argument, which it forwards only when it is set.
  • _cp_file and _merge still check _is_zonal_bucket, so on RCU buckets they keep using the JSON API.
• finalize_on_close (gcsfs/extended_gcsfs.py, gcsfs/zonal_file.py):
  • The filesystem and ZonalFile defaults change from False to None.
  • In open, ZonalFile, put_file, pipe_file and simple_upload, None resolves to True on RCU buckets and to False on zonal buckets.
  • Precedence: a per-call value first, then the filesystem attribute, then that default.
  • If put_file, pipe_file or simple_upload fails partway, the writer is closed without finalizing, so a partial object is never finalized.
• Write failures (gcsfs/extended_gcsfs.py, gcsfs/zb_hns_utils.py, gcsfs/zonal_file.py; these also apply to zonal buckets):
  • close_aaow re-raises a failed finalize instead of only logging it, so put_file, pipe_file, simple_upload, upload_chunk and closing a ZonalFile no longer report an unfinalized write as successful. Closes that do not finalize stay best-effort.
  • If the closing flush of a ZonalFile fails, the writer is still closed without finalizing, and the flush error is raised.
  • If the writer fails to open, the ZonalFile becomes close-only so close() does not retry opening the writer.
  • put_file opens the local file before the writer, so a missing or unreadable local file raises without touching the remote object, as on HTTP.
  • Once the writer is open, put_file and pipe_file update the dircache even when the write fails, and a ZonalFile whose closing flush fails drops its cached path and parent listing, because the object has already been replaced.
• Ignored upload parameters (gcsfs/zonal_file.py): the appendable writer ignores content_type, metadata, fixed_key_metadata and kms_key_name. ZonalFile write modes on zonal and RCU buckets now log a warning when any of them is set. put_file, pipe_file and the upload helpers already do this.
• Behavior changes on RCU buckets compared with writing over HTTP:
  • An overwrite replaces the existing object as soon as the writer opens (on the first write for open()), so a write that fails partway leaves the partially written, unfinalized object in place of the old one. Over HTTP, the old object stayed intact until the upload completed.
  • "ab" appends through the gRPC writer. It fails on a finalized or non-appendable object instead of overwriting it, so it can only extend objects that were left unfinalized, for example written with finalize_on_close=False.
  • autocommit=False and transactions no longer defer the write, since ZonalFile.discard() only logs a warning.
  • if_generation_match values above 0 raise NotImplementedError until the SDK writer accepts that argument. 0, mode="create" and "xb" still work.
• Dependencies (pyproject.toml): require google-cloud-storage>=3.15.0, the first release whose AsyncAppendableObjectWriter accepts storage_class.
• Tests:
  • gcsfs/tests/conftest.py: zonal_write_mocks takes a BucketLayout param (RAPID_WRITE_LAYOUTS: the zonal layout and both RCU layouts) and exposes the expected storage_class; it skips the RCU layouts on real GCS. Every open now gets an open writer that carries its bucket and object name.
  • gcsfs/tests/test_zonal.py, gcsfs/tests/test_zonal_file.py: the existing write, commit and empty-file tests also run on the RCU layouts and check storage_class. New tests cover every write mode on RCU, the ignored-parameter warning for each parameter, the full finalize_on_close precedence matrix, failed put_file, pipe_file and simple_upload calls closing the writer without finalizing, a failed closing flush still closing the writer, a failed writer open not being retried on close, a missing local file not opening the writer, the dircache update after a failed write, and a failed finalize propagating from put_file, pipe_file and simple_upload through the real close_aaow.
  • gcsfs/tests/test_zonal_unit.py: cp_file and merge on RCU buckets use the core implementation, Rapid Cache buckets keep writing over HTTP, and core.simple_upload and core.initiate_upload hand RCU buckets to the gRPC helpers.
  • gcsfs/tests/test_zb_hns_utils.py: init_aaow forwards storage_class only when it is set, and close_aaow re-raises a failed finalize.
  • The RCU cases use mocks; the live end-to-end suites do not have an RCU bucket yet.

[gemini-code-assist]

Code Review

This pull request refactors bucket type detection in gcsfs by replacing the BucketType enum with a more granular BucketLayout dataclass that tracks StorageClass, CachingTier, and NamespaceLayout. This change enables support for Rapid Cache Ultra (RCU) buckets alongside Zonal buckets, allowing them to use the high-performance gRPC data path. The default finalization behavior on close is updated to default to False for Zonal buckets and True for RCU buckets. The review feedback highlights potential resource leaks in _put_file, _pipe_file, and simple_upload if a layout lookup fails after a successful write, suggesting that finalize_on_close be resolved inside the try blocks to ensure the writer is always safely closed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.37%. Comparing base (c783299) to head (0e47110).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1135      +/-   ##
==========================================
+ Coverage   92.24%   92.37%   +0.13%     
==========================================
  Files          20       20              
  Lines        4047     4094      +47     
==========================================
+ Hits         3733     3782      +49     
+ Misses        314      312       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[zhixiangli]

Checked against a real RCU bucket (STANDARD + RAPID_CACHE_ULTRA) and a Zonal bucket with google-cloud-storage 3.17.0. RCU writes do land as RAPID and are finalized by default, as described. Details are in the inline comments:

1. A failed overwrite on RCU leaves the partial upload as the live object (behavior change, not documented).
2. Finalize failures are swallowed, so the write is reported as successful.
3. ZonalFile leaks the write stream if the closing flush fails (now reachable for RCU writes).
4. Nit: a failed "xb" open calls init_aaow twice.
5. Test gaps.

I tested the fixes suggested for 2–4 against the real RCU bucket, and the existing unit tests still pass with them applied.

Two test gaps that can't be anchored inline (outside the diff):

• test_zonal_file_commit and test_zonal_file_lazy_init_creates_empty_file_on_close only run on Zonal. Running them on RAPID_WRITE_LAYOUTS would cover commit() turning finalize_on_close from True to False, and an empty file closing with RAPID and finalize.
• Every test mocks the writer, so the suite also passes on google-cloud-storage 3.12. A cheap guard: assert storage_class is in inspect.signature(AsyncAppendableObjectWriter.__init__).parameters.

[yuxin00j]

Done for the commit and empty-file tests; skipped the SDK signature test since CI runs pip install -e . (google-cloud-storage>=3.15.0) and older versions fail with TypeError on the first RCU write.

[zhixiangli]

Thanks for the PR. Routing RCU writes through the appendable writer with storage_class="RAPID" works as described. I checked the write paths against google-cloud-storage 3.17, with the gRPC transport replaced by an in-memory server that reproduces GCS appendable-write behavior. An overwrite replaces the object as soon as the writer opens, and only flushed bytes survive a broken stream.

Because the overwrite happens at writer open, a write that fails now changes the bucket. Before this PR, RCU wrote over HTTP and a failed write left the object unchanged. Two places don't account for that yet: the _put_file ordering and the cache maintenance on failure (inline comments 1 and 2). I'd ask for those to be fixed before merging. The other inline comments are minor.