Download error: Error in send_request(method = "POST", data = data) : Server error!

[a-qvecell]

Hello :)
Thank you for this super useful package!

We have run into an issue while trying to download metadata (and sequences) in batches of 5000:

df_test <- query(
  credentials = credentials, 
  location = "Denmark", 
  from = "2019-07-26", 
  to = "2020-10-28",
 fast = TRUE
)

df_test <- df_test %>% 
  mutate(batchnumber = floor(row_number()/5000)+1)

for (b in unique(df_test$batchnumber)){
  print(b)
  this_batch <- df_test %>% 
    filter(batchnumber == b)
  
  full_df <- download(credentials = credentials, list_of_accession_ids = this_batch$accession_id, get_sequence=FALSE)

  write_csv(full_df, paste0("batchnummer_", b, ".csv"))
  }

It crashes during the download command with the output
Selecting entries...
Compressing data. Please wait...
Data ready.
Error in send_request(method = "POST", data = data) : Server error!
In addition: Warning message:
In send_request(method = "POST", data = data) :
An error occurred while trying to POST https://www.epicov.org/epi3/frontend?

I am using GISAIDR version 0.9.10.
Can you help? :)
Thank you in advance,
Cheers,
Amanda

[Wytamma]

Hi Amanda!

Thanks for the issue! I’ll look into it this week. In the meantime just wanted to confirm you can download batches smaller than 5000 E.g1000? The server error warning makes me think it’s something on the GIDAID side. Sometimes they change the download limits for different accounts. You could also try manually downloading a single batch of 5000 from GISAID to confirm your account limit.

All the best,

Wytamma

[a-qvecell]

Hi :)
Thank you for the speedy reply.

I tried changing to
df_test <- df_test %>%
mutate(batchnumber = floor(row_number()/400)+1)
and that works fine - and also for 1000 and 2000.

I I tried downloading 4,964 viruses from the GISAID GUI and that works just fine.
I tried
df_test <- df_test %>%
mutate(batchnumber = floor(row_number()/4000)+1)
but it fails, now with the message
Error in download.file(download_url, tmpTarFile, quiet = TRUE, method = "auto", :
download from 'https://www.epicov.org/epi3/entities/tmp/tmp_sd_2024_06_03_14_22_sdxpxm_qu6v57b3a3ba/gisaid_auspice_input_hcov-19_2024_06_03_06.tar' failed
In addition: Warning message:
In download.file(download_url, tmpTarFile, quiet = TRUE, method = "auto", :
URL 'https://www.epicov.org/epi3/entities/tmp/tmp_sd_2024_06_03_14_22_sdxpxm_qu6v57b3a3ba/gisaid_auspice_input_hcov-19_2024_06_03_06.tar': Timeout of 60 seconds was reached.

It would be nice to be able to download eg. up to 4000 or 5000 seqs but 2000 can do if it is not easily fixable :)
All the best,
Amanda

[Wytamma]

Hi Amanda, glad you found a temporary work around. It might be some sort of HTTP timeout with the large file size. I’ll do some testing and get back to you.