Is there a way to use PATs for URL file_in()?

[shirdekel]

Prework

• Read and agree to the code of conduct and contributing guidelines.
• If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• For any problems you identify, write a minimal reproducible example so the maintainer can troubleshoot. A reproducible example is:
  • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
  • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
  • Readable: format your code according to the tidyverse style guide.

Question

I'd like to import a file from a private OSF project using osfr. When I do, I am prompted with an error. There seems to be some way around this using curl that I don't really understand (from #981). Is this the case, or does that not apply to URLs that need access through tokens?

Reproducible example

library(drake)
library(osfr)
#> Automatically registered OSF personal access token

## Public project
osf_retrieve_file("https://osf.io/btgx3/")
#> # A tibble: 1 x 3
#>   name                  id                       meta            
#>   <chr>                 <chr>                    <list>          
#> 1 Study_19_Figure_1.pdf 5751d71d9ad5a102079379d7 <named list [3]>

## Private project
osf_retrieve_file("https://osf.io/gnc7u/")
#> # A tibble: 1 x 3
#>   name       id                       meta            
#>   <chr>      <chr>                    <list>          
#> 1 mtcars.csv 609a31c35dfcb90142bfd6dc <named list [3]>

plan <- drake_plan(
    public = osf_retrieve_file(file_in("https://osf.io/btgx3/"))
)

make(plan)
#> ▶ target public
readd(public)
#> # A tibble: 1 x 3
#>   name                  id                       meta            
#>   <chr>                 <chr>                    <list>          
#> 1 Study_19_Figure_1.pdf 5751d71d9ad5a102079379d7 <named list [3]>

plan <- drake_plan(
  private = osf_retrieve_file(file_in("https://osf.io/gnc7u/"))
)

make(plan)
#> Error: no ETag or Last-Modified for url: https://osf.io/gnc7u/
readd(private)
#> Warning in structure(list(hash = hash, message = sprintf("hash '%s' not
#> found", : restarting interrupted promise evaluation
#> Error: key 'private' ('objects') not found

^{Created on 2021-05-12 by the reprex package (v1.0.0)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS Mojave 10.14.6        
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_AU.UTF-8                 
#>  ctype    en_AU.UTF-8                 
#>  tz       Australia/Sydney            
#>  date     2021-05-12                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date       lib source                            
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.0.0)                    
#>  backports     1.2.1      2020-12-09 [1] CRAN (R 4.0.2)                    
#>  base64url     1.4        2018-05-14 [1] CRAN (R 4.0.0)                    
#>  cachem        1.0.4      2021-02-13 [1] CRAN (R 4.0.2)                    
#>  cli           2.5.0      2021-04-26 [1] CRAN (R 4.0.2)                    
#>  crayon        1.4.1      2021-02-08 [1] CRAN (R 4.0.2)                    
#>  crul          1.1.0      2021-02-15 [1] CRAN (R 4.0.2)                    
#>  curl          4.3.1      2021-04-30 [1] CRAN (R 4.0.2)                    
#>  DBI           1.1.1      2021-01-15 [1] CRAN (R 4.0.2)                    
#>  digest        0.6.27     2020-10-24 [1] CRAN (R 4.0.2)                    
#>  dplyr         1.0.5.9000 2021-04-16 [1] Github (tidyverse/dplyr@8b036bc)  
#>  drake       * 7.13.1     2021-02-03 [1] CRAN (R 4.0.2)                    
#>  ellipsis      0.3.2      2021-04-29 [1] CRAN (R 4.0.2)                    
#>  evaluate      0.14       2019-05-28 [1] CRAN (R 4.0.0)                    
#>  fansi         0.4.2      2021-01-15 [1] CRAN (R 4.0.2)                    
#>  fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.0.2)                    
#>  filelock      1.0.2      2018-10-05 [1] CRAN (R 4.0.0)                    
#>  fs            1.5.0      2020-07-31 [1] CRAN (R 4.0.2)                    
#>  generics      0.1.0      2020-10-31 [1] CRAN (R 4.0.2)                    
#>  glue          1.4.2      2020-08-27 [1] CRAN (R 4.0.2)                    
#>  highr         0.9        2021-04-16 [1] CRAN (R 4.0.2)                    
#>  hms           1.0.0      2021-01-13 [1] CRAN (R 4.0.2)                    
#>  htmltools     0.5.1.1    2021-01-22 [1] CRAN (R 4.0.2)                    
#>  httpcode      0.3.0      2020-04-10 [1] CRAN (R 4.0.2)                    
#>  httr          1.4.2      2020-07-20 [1] CRAN (R 4.0.2)                    
#>  igraph        1.2.6      2020-10-06 [1] CRAN (R 4.0.2)                    
#>  jsonlite      1.7.2      2020-12-09 [1] CRAN (R 4.0.2)                    
#>  knitr         1.33       2021-04-24 [1] CRAN (R 4.0.2)                    
#>  lifecycle     1.0.0      2021-02-15 [1] CRAN (R 4.0.2)                    
#>  magrittr      2.0.1      2020-11-17 [1] CRAN (R 4.0.2)                    
#>  memoise       2.0.0      2021-01-26 [1] CRAN (R 4.0.2)                    
#>  osfr        * 0.2.8      2020-02-17 [1] CRAN (R 4.0.2)                    
#>  pillar        1.6.0      2021-04-13 [1] CRAN (R 4.0.2)                    
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.0.0)                    
#>  prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.0.0)                    
#>  progress      1.2.2      2019-05-16 [1] CRAN (R 4.0.0)                    
#>  ps            1.6.0      2021-02-28 [1] CRAN (R 4.0.2)                    
#>  purrr         0.3.4      2020-04-17 [1] CRAN (R 4.0.0)                    
#>  R6            2.5.0      2020-10-28 [1] CRAN (R 4.0.2)                    
#>  Rcpp          1.0.6      2021-01-15 [1] CRAN (R 4.0.2)                    
#>  reprex        1.0.0      2021-01-27 [1] CRAN (R 4.0.2)                    
#>  rlang         0.4.11     2021-04-30 [1] CRAN (R 4.0.2)                    
#>  rmarkdown     2.8.1      2021-05-11 [1] Github (rstudio/rmarkdown@e98207f)
#>  rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.0.2)                    
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.0.0)                    
#>  storr         1.2.5      2020-12-01 [1] CRAN (R 4.0.2)                    
#>  stringi       1.6.1      2021-05-10 [1] CRAN (R 4.0.2)                    
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 4.0.0)                    
#>  styler        1.3.2      2020-02-23 [1] CRAN (R 4.0.0)                    
#>  tibble        3.1.1      2021-04-18 [1] CRAN (R 4.0.2)                    
#>  tidyselect    1.1.0      2020-05-11 [1] CRAN (R 4.0.0)                    
#>  triebeard     0.3.0      2016-08-04 [1] CRAN (R 4.0.2)                    
#>  txtq          0.2.3      2020-06-23 [1] CRAN (R 4.0.2)                    
#>  urltools      1.7.3      2019-04-14 [1] CRAN (R 4.0.2)                    
#>  utf8          1.2.1      2021-03-12 [1] CRAN (R 4.0.2)                    
#>  vctrs         0.3.8      2021-04-29 [1] CRAN (R 4.0.2)                    
#>  withr         2.4.2      2021-04-18 [1] CRAN (R 4.0.2)                    
#>  xfun          0.22       2021-03-11 [1] CRAN (R 4.0.2)                    
#>  yaml          2.2.1      2020-02-01 [1] CRAN (R 4.0.2)                    
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

[wlandau]

drake interfaces through curl for general URLs, and PATs seem specific to particular services, so I think workarounds will probably be user-side. You could try setting the PAT as an environment variable (in a project-level .Renviron file) if OSF accepts it. Alternatively, some services allow PATs to be part of the URL or REST API request. GitHub is an example: https://github.com/rstudio/bookdown-demo/blob/eb2dd2a8d29e7eaffb5b69c1003f0e04172c3587/_deploy.sh#L11

[wlandau]

On second thought the PAT as part of a URL could be risky from a security point of view.

[shirdekel]

Setting the PAT as an environment variable is what I'm currently doing to get access outside of drake. Is there a way to have drake take it into account? That would be ideal because then I can just put it in .gitignore. However, setting a project-level .Renviron file doesn't seem to automatically fix this. Is there something I need to specifically set in drake?

[wlandau]

drake does not directly support PATs, that’s something you would need to configure manually depending on the needs of the target in question. If .Renviron does not work (surprised to learn that’s the case for you) then maybe the target needs some custom code to securely retrieve and set the env var, e.g. with {secret} and Sys.setenv().

[shirdekel]

Ok, thanks.