[WIP] Added max_block and request_batch to pyfive.File and backend

[dwest77a]

Description

• Added max_block and request_batch as parameters to the pyfive.File declaration. This is relayed to the backend bulk_fsspec method which now uses the merge_requests utility from fsspec if max_block is defined. Request batch allows configuration via the fsspec cat_ranges method to restrict the number of simultaneous requests down from the default for remote connections of 1280.

Checklist

• This pull request has a descriptive title and labels
• This pull request has a minimal description (most was discussed in the issue, but a two-liner description is still desirable)
• Unit tests have been added (if codecov test fails)
• Any changed dependencies have been added or removed correctly (if need be)
• If you are working on the documentation, please ensure the current build passes
• All tests pass

[valeriupredoi]

thanks @dwest77a 🍺 Note that if you wish the PR to be truly draft, you can always convert it to a Draft PR. I approved the tests to run here, and if you don't mind, I'll also add you to PyActiveStorage 🍻

[dwest77a]

This relates to issue #257

[valeriupredoi]

@dwest77a the issue with pre-commit kicks in since you change the functional call and args:

131c139,141
<                 self._read_bulk_fsspec(fh, chunks, out, dtype)
---
>                 self._read_bulk_fsspec(
>                     fh, chunks, out, dtype, max_block=max_block, batch_size=batch_size
>                 )

posix is defined as attr only if not fh, and the same for _fh - not great in terms of coding, but also mypy is not catching that. I think you can safely ignore those, and to run the tests you can temporarily turn off the call to pre-commit in the Github Action workflow

[valeriupredoi]

@dwest77a I realized we've had quite a few of these [attr-defined] mypy barfs, that I had to manually allow - the way the code is written is a bit confusing to mypy, and having checked manually all is kosher, I told mypy to mind its own business in 702bf12 - note, though, it's still a good idea to run pre-commit install && pre-commit run -a when you done with code changes, so to catch any issues locally, and fix em 🍺

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.52%. Comparing base (667c468) to head (04164d0).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #256      +/-   ##
==========================================
+ Coverage   78.48%   78.52%   +0.04%     
==========================================
  Files          15       15              
  Lines        3416     3423       +7     
  Branches      546      547       +1     
==========================================
+ Hits         2681     2688       +7     
  Misses        593      593              
  Partials      142      142              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dwest77a]

@valeriupredoi thanks, I'll look into the test coverage now.

[valeriupredoi]

Good man Dan 🍺

[dwest77a]

@valeriupredoi happy to reconfigure this in some way that doesn't use a hardcoded path, it does need to be remote to test the block and batch sizes though.