get eurostat freezing

[antagomir]

Since a week someone had problems with downloading the daily exchange rates file: ert_bil_eur_d It downloads the file, but then seems to freeze (I had it running for several hours but it looks like an infinite loop J ), This is the R code:

library(eurostat)
x  <- get_eurostat("ert_bil_eur_d", time_format="raw", cache=F)

The problem started last week and it could be the result of some package updates.

[antagomir]

For me this works fine. Could it be that eurostat service itself has ome issues, rather than the package?

[WietseDol]

Running the command I see this

 > theData=get_eurostat(code, time_format="raw",cache=F)
trying URL 'http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?sort=1&file=data%2Fert_bil_eur_d.tsv.gz'
Content type 'application/octet-stream;charset=UTF-8' length 663612 bytes (648 KB)
downloaded 648 KB

|=================================================================================| 100%    2 MB
                                                                  |  22%

It hangs after getting the data, in something I would guess it is readr or dplyr

[jhuovari]

It freeze for me also. Could be dplyr update.

[jhuovari]

The problem seems to be in tidy_eurostat:

    dat2 <- tidyr::gather_(dat2, cnames2, "values",
                           names(dat2)[!(names(dat2) %in% cnames1)],
                           convert = FALSE, na.rm = TRUE)

It could be that there are just too many columns, more than 11 000. I could not find quick solution.

[jhuovari]

It seems to work with tidyr::gather. So, i try to fix it with non-standard evaluation.

[WietseDol]

The new version works. Great job. Wietse From: Janne Huovari [mailto:[email protected]] Sent: Monday, September 11, 2017 12:14 To: rOpenGov/eurostat <[email protected]> Cc: WietseDol <[email protected]>; Comment <[email protected]> Subject: Re: [rOpenGov/eurostat] get eurostat freezing (#98) It seems to work with tidyr::gather. So, i try to fix it with non-standard evaluation. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#98 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASvVK5fZ_72fY-aSpr8crBcPGlfd8AB6ks5shQgEgaJpZM4PSu0a>.

[georgeblck]

I am having the same problem. It hangs after getting the data, so it must be the tidy::gather?

Here is the code to reproduce:

library(eurostat)
get_eurostat("migr_asyappctzm", type = "label", time_format = "num", stringsAsFactors = FALSE)

[WietseDol]

I just tried it, R 4.02 and the latest dplyr 1.0.2. It takes a few minutes but is succeeds, there is a warning...

get_eurostat("migr_asyappctzm", type = "label", time_format = "num", stringsAsFactors = FALSE)
trying URL 'https://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?sort=1&file=data%2Fmigr_asyappctzm.tsv.gz'
Content type 'application/octet-stream;charset=UTF-8' length 7808372 bytes (7.4 MB)
downloaded 7.4 MB

|========================================================================================================================================================| 100% 219 MB
Table migr_asyappctzm cached at C:\Users\Wiets\AppData\Local\Temp\RtmpcZNLwU/eurostat/migr_asyappctzm_num_label_FF.rds

A tibble: 63,798,248 x 8

unit citizen sex age asyl_app geo time values

1 Person Andorra Females Total Asylum applicant Belgium 2021. 0
2 Person Andorra Females Total Asylum applicant Bulgaria 2021. 0
3 Person Andorra Females Total Asylum applicant Switzerland 2021. 0
4 Person Andorra Females Total Asylum applicant Germany (until 1990 former territory of the FRG) 2021. 0
5 Person Andorra Females Total Asylum applicant Denmark 2021. 0
6 Person Andorra Females Total Asylum applicant Spain 2021. 0
7 Person Andorra Females Total Asylum applicant European Union - 27 countries (from 2020) 2021. 0
8 Person Andorra Females Total Asylum applicant Finland 2021. 0
9 Person Andorra Females Total Asylum applicant France 2021. 0
10 Person Andorra Females Total Asylum applicant Hungary 2021. 0

... with 63,798,238 more rows

Warning message:
In label_eurostat(y[[i]], i, eu_order = eu_order, lang = lang, countrycode = countrycode, :
All labels for citizen were not found.

[antagomir]

Thanks! This should be sorted out as soon as the situation allows. PRs also welcome.

[jhuovari]

I don't think there is a problem in package. The migr_asyappctzm is just really big dataset. The warning comes from NA codes in citizen. The label_eurostat doesn't seems to handle those.

[georgeblck]

Thanks for the quick feedback from all of you. Loading migr_asyappctzm does indeed work after more than five minutes.
However, migr_asydcfstq definitely crashes R for me with the following code:

library(eurostat)
accept <- get_eurostat("migr_asydcfstq", type = "label", time_format = "num", stringsAsFactors = FALSE)

For me personally, a small warning message would be very helpful that a chosen big dataset can take a long time to process, get_eurostat is already quite verbose anyways.
When using certain eurostat codes without filters, it is not immediately obvious that some are huge and some not.

[jhuovari]

That is even bigger. You can try to get it first without labels, and label afterwards, and turn off caching, cache = FALSE.

[georgeblck]

Leaving out the labels made it possible to download the data set without crashing R. Thank you!