TEST - WIP: Full pak install

[gerbeldo]

Description

Just playing around, trying to restore all packages using pak. This uses renv::restore with the option renv.config.pak.enable=TRUE.

Locally, the pipeline builds in about 10 - 15 minutes (using 6 cores). In a github actions it cuts build time by half (from ~90 to ~45min)

• upgrades R to 4.3.1
• upgrade renv to 1.0.3
• upgrades several dependencies, due to conflicting versions which make pak install fail
  • harmony: fixes several bugs (in particular, uses the correct dim number, allowing simplification of integration code), but needs workaround due to licensing of RhpcBLASctl package
• updated snapshots (in particular, updates to scDblFinder change the scores of some cells, voiding the snapshots)
• fixed some tests (some functions called without the package declaration, in particular data.table::data.table; harmony failing on too few cells in a subset test; some incorrectly stubbed functions)

Due to the weird behaviours below, In Rstudio it might be better to not use pak to install the dependencies (using regular renv::restore installs quite quickly due to binaries available)

There are weird behaviours:

• Installing packages locally (macOS) without fixing versions works. After snapshot, the lockfile can be restored using pak on a new and clean local Rstudio project. BUT fails in a docker container (rocker/r-ver) due to dependency conflicts.

• The lockfile was generated by manually installing dependencies in a docker container and snapshotting from there (using renv::snapshot(type = "all"). This lockfile works in a docker container. But it brings issues on a local Rstudio project: it can be restored, but upon R restart, renv/pak return "subprocess could not start" errors when trying to install anything. The issue seems to be the lockfile, as renaming it to anything other than "renv.lock" (bypassing the automatic renv checks) fixes it, and allows regular package install using renv/pak.

• In addition, there's something broken with the .libPaths(), at least in my install, as some packages that are installed are not detected after re-starting R. They might be installed to a different library. But I don't have enough information yet.

• for some reason, some bioconductor/github packages are installed twice (see the action log). renv/pak understand that they need to be updated, but it installs the same version number. Might be related to the remote used (bioconductor/github/RSPM)

Details

URL to issue

N/A

Link to staging deployment URL (or set N/A)

N/A

Links to any PRs or resources related to this PR

N/A

Integration test branch

master

Merge checklist

Your changes will be ready for merging after all of the steps below have been completed.

Code updates

Have best practices and ongoing refactors being observed in this PR

• Migrated any selector / reducer used to the new format.

Manual/unit testing

• Tested changes using InfraMock locally or no tests required for change, e.g. Kubernetes chart updates.
• Validated that current unit tests for code work as expected and are sufficient for code coverage or no unit tests required for change, e.g. documentation update.
• Unit tests written or no unit tests required for change, e.g. documentation update.

Integration testing

You must check the box below to run integration tests on the latest commit on your PR branch.
Integration tests have to pass before the PR can be merged. Without checking the box, your PR
will not pass the required status checks for merging.

• Started end-to-end tests on the latest commit.

Documentation updates

• Relevant Github READMEs updated or no GitHub README updates required.
• Relevant Wiki pages created/updated or no Wiki updates required.

Optional

• Staging environment is unstaged before merging.
• Photo of a cute animal attached to this PR.

[codecov]

Codecov Report

Attention: Patch coverage is 60.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 79.24%. Comparing base (2d407d9) to head (0ad6674).
Report is 245 commits behind head on master.

Files with missing lines	Patch %	Lines
pipeline-runner/R/qc-6-integrate_scdata-harmony.R	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #341      +/-   ##
==========================================
- Coverage   79.65%   79.24%   -0.41%     
==========================================
  Files          29       29              
  Lines        2875     2877       +2     
==========================================
- Hits         2290     2280      -10     
- Misses        585      597      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.