Skip to content

fix(index): download the catalogue without a 60 second cap - #1

Open
atsyplenkov wants to merge 1 commit into
spatial-tools:mainfrom
atsyplenkov:fix/index-curl-download
Open

atsyplenkov wants to merge 1 commit into
spatial-tools:mainfrom
atsyplenkov:fix/index-curl-download

Conversation

@atsyplenkov

Copy link
Copy Markdown

Summary

index() downloads the AlphaEarth index once and keeps it in the cache.

.index_download
  if the index is already in the cache
    return that file
  curl_download into a temporary file
  .index_check_size
  move the file into the cache

download.file() stops after 60 seconds. The index is too large for that limit, so the download stopped before a complete file reached the cache. curl_download() has no total time limit. The file moves into the cache only after the download finishes and the size matches the server Content-Length. A response with no Content-Length is accepted. A short file is not stored in the cache.

Evidence

  • Before: download.file() used the 60 second timeout. A missing Content-Length made the size check stop with "missing value where TRUE/FALSE needed".
    After: pkgload::load_all() and testthat::test_file("tests/testthat/test-index.R") reported 7 passes and 0 failures. The new tests mock the server. One accepts a response with no Content-Length. One rejects a 4 byte file when the server reports 12 bytes. air format and jarl check passed on the edited R files.

Merge Danger

Door: two-way

Revert this commit. An index that is already in the cache is reused. The new download runs only when the cache is empty or overwrite = TRUE.

Blast Radius: the first download

People who already have the index see no change. A new download uses curl instead of download.file().

download.file() stops at options(timeout), which defaults to 60 seconds.
curl writes the index to a temporary file and moves it into the cache
only when the size matches Content-Length.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant