[R] CRAN packaging checklist for version 13.0.0 #36802

thisisnic · 2023-07-21T09:21:45Z

Packaging checklist for CRAN release

For a high-level overview of the release process see the
Apache Arrow Release Management Guide.

Before the release candidate is cut

Create a GitHub issue entitled [R] CRAN packaging checklist for version X.X.X and copy this checklist to the issue.
Evaluate the status of any failing nightly tests and nightly packaging builds. These checks replicate most of the checks that CRAN runs, so we need them all to be passing or to understand that the failures may (though won't necessarily) result in a rejection from CRAN.
Check current CRAN check results
Ensure the contents of the README are accurate and up to date.
Run urlchecker::url_check() on the R directory at the release candidate.
commit. Ignore any errors with badges as they will be removed in the CRAN release branch.
Polish NEWS but do not update version numbers (this is done automatically later).
Run preliminary reverse dependency checks using archery docker run r-revdepcheck.
For major releases, prepare tweet thread highlighting new features.

Wait for the release candidate to be cut:

After release candidate has been cut

Create a CRAN-release branch from the release candidate commit

Optional: PRs for autobrew and rtools (release candidate)

Make pull requests into the autobrew and
rtools-packages repositories
used by the configure script on MacOS and Windows. These pull requests will
use the release candidate as the source.

Pull request to modify
the apache-arrow autobrew formula
to update the release version, SHA256 checksum of the release source file (which can be found in the same directory as the release source file), and any changes to dependencies and build steps that have changed in the
copy of the formula we have of that formula in the Arrow repo
- apache-arrow 13.0.0 autobrew/homebrew-core#126
Pull request to modify
the apache-arrow-static autobrew formula
to update the version, SHA, and any changes to dependencies and build steps that have changed in the
copy of the formula we have of that formula in the Arrow repo
apache-arrow 13.0.0 autobrew/homebrew-cran#86
Pull request to modify the
autobrew script
to include any additions made to
r/tools/autobrew.
- apache-arrow 13.0.0 autobrew/scripts#17
Pull request to modify the
RTools PKGBUILD script
to reflect changes in
ci/PKGBUILD,
uncommenting the line that says "uncomment to test the rc".
- apache-arrow 13.0.0 r-windows/rtools-packages#292

Prepare and check the .tar.gz that will be released to CRAN.

git fetch upstream && git checkout release-X.X.X-rcXX && git clean -f -d
Run make build. This copies Arrow C++ into tools/cpp, prunes some
unnecessary components, and runs R CMD build to generate the source tarball.
Because this will install the package, you will need to ensure that the version
of Arrow C++ available to the configure script is the same as the version
that is vendored into the R package (e.g., you may need to unset ARROW_HOME).
devtools::check_built("arrow_X.X.X.tar.gz") locally
Run reverse dependency checks using archery docker run r-revdepcheck.

Release vote

Release vote passed!

PRs for autobrew and rtools (official release)

Create new autobrew and r-windows PRs such that they use the release
instead of the release candidate:

PR into autobrew/homebrew-core (apache-arrow autobrew formula)
PR into autobrew/homebrew-core (apache-arrow-static autobrew formula)
PR into autobrew/scripts
PR into r-windows/rtools-packages

Generate R package to submit to CRAN

If the release candidate commit updated, rebase the CRAN release branch
on that commit.
Pick any commits that were made to main since the release commit that
were needed to fix CRAN-related submission issues identified in the above
steps.
Remove badges from README.md
Run urlchecker::url_check() on the R directory
Create a PR entitled WIP: [R] Verify CRAN release-10.0.1-rc0. Add
a comment @github-actions crossbow submit --group r to run all R crossbow
jobs against the CRAN-specific release branch.
- WIP: [R] Verify CRAN release-13.0.0-rc3 #37336
Regenerate arrow_X.X.X.tar.gz (i.e., make build)

Ensure linux binary packages are available:

Ensure linux binaries are available in the artifactory:
https://apache.jfrog.io/ui/repos/tree/General/arrow/r

Check binary Arrow C++ distributions specific to the R package

Upload the .tar.gz to win-builder (r-devel only)
and confirm (with Nic, who will automatically receive an email about the results) that the check is clean.
This step cannot be completed before Jeroen has put the binaries in the MinGW repository, i.e. here, here, and here.
Upload the .tar.gz to MacBuilder
and confirm that the check is clean
https://mac.r-project.org/macbuilder/results/1693354381-aed8fd0fff067774/
Check install.packages("arrow_X.X.X.tar.gz") on Ubuntu and ensure that the
hosted binaries are used
devtools::check_built("arrow_X.X.X.tar.gz") locally one more time (for luck)

CRAN submission

Upload arrow_X.X.X.tar.gz to the
CRAN submit page
Confirm the submission email

Wait for CRAN...

Accepted!
Tag the tip of the CRAN-specific release branch
Add a new line to the matrix in the backwards compatability job
(patch releases only) Update the package version in ci/scripts/PKGBUILD, dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb, r/DESCRIPTION, and r/NEWS.md
(CRAN-only releases) Rebuild the docs with pkgdown::build_site(examples = FALSE, lazy = TRUE, install = FALSE) and submit a PR to the asf-site branch of the docs site with the contents of r/docs/news/index.html.
(CRAN-only releases) Bump the version number in r/pkgdown/assets/versions.json, and update this on the the asf-site branch of the docs site too.
Update the packaging checklist template to reflect any new realities of the
packaging process.
Wait for CRAN-hosted binaries on the
CRAN package page to reflect the
new version
Tweet!

The text was updated successfully, but these errors were encountered:

thisisnic · 2023-07-21T09:36:25Z

TODO: follow-up on discussions in #36551 to see if we need to do anything difference for this release

thisisnic · 2023-07-21T10:00:16Z

We'll need to cherry-pick the NEWS update into here: #36806

paleolimbot · 2023-07-24T14:47:52Z

I ran the revdepchecks using archery and got one failure for nanoarrow (will be fixed by and I will cherry pick + submit ASAP). I also ran them more manually and got failures for dataversionr and pins related to the class of the output of read_parquet().

dataversionr

With CRAN arrow

-- R CMD check results --------------------------------- dataversionr 0.9.0 ----
Duration: 2m 20.5s

0 errors v | 0 warnings v | 0 notes v

With local arrow

-- R CMD check results --------------------------------- dataversionr 0.9.0 ----
Duration: 2m 28.9s

> checking tests ...
  See below...

-- Test failures ------------------------------------------------- testthat ----

> library(testthat)
> library(dataversionr)
> 
> test_check("dataversionr")

Attaching package: 'dplyr'

The following object is masked from 'package:testthat':

    matches

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union


Attaching package: 'tidyr'

The following object is masked from 'package:testthat':

    matches

Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.

Attaching package: 'arrow'

The following object is masked from 'package:testthat':

    matches

The following object is masked from 'package:utils':

    timestamp


Attaching package: 'lubridate'

The following object is masked from 'package:arrow':

    duration

The following objects are masked from 'package:base':

    date, intersect, setdiff, union

Committing diff to dataset...
Verifying diff can be retrieved from dataset...
A parquet file can be read from the target path...
Remote diff is identical to local diff.
Committing diff to dataset...
Verifying diff can be retrieved from dataset...
A parquet file can be read from the target path...
Remote diff is identical to local diff.
Checking that new_df can be diffed...
Diff test passed.
Checking that new_df can be diffed...
Diff test passed.
Checking that new_df can be diffed...
Checking that new_df can be diffed...
Diff test passed.
Checking that new_df can be diffed...
Diff test passed.
destination parameter is a string. Coercing to local FileSystem.
destination parameter is a string. Coercing to local FileSystem.
[ FAIL 6 | WARN 19 | SKIP 0 | PASS 43 ]

══ Failed tests ════════════════════════════════════════════════════════════════
── Failure ('test-backup_ops.R:27:13'): local get backup ───────────────────────
{
    ...
} (`actual`) not equal to `old_df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"
── Failure ('test-backup_ops.R:161:13'): local read_dv_backup ──────────────────
{
    ...
} (`actual`) not equal to `old_df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"
── Failure ('test-diff_ops.R:67:3'): local retrieve diff works ─────────────────
... %>% select(-diff_timestamp) (`actual`) not equal to `diff_df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"
── Failure ('test-dv_ops.R:107:13'): local read_dv ─────────────────────────────
{
    ...
} (`actual`) not equal to `newer_df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"
── Failure ('test-dv_ops.R:124:13'): local read_dv via backup ──────────────────
{
    ...
} (`actual`) not equal to new_df %>% select(sort(colnames(.))) (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"
── Failure ('test-latest_ops.R:29:13'): local get latest ───────────────────────
{
    ...
} (`actual`) not equal to `new_df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"

[ FAIL 6 | WARN 19 | SKIP 0 | PASS 43 ]
Error: Test failures
Execution halted

1 error x | 0 warnings v | 0 notes v

gbifdb

With CRAN arrow

-- R CMD check results --------------------------------------- gbifdb 0.1.2 ----
Duration: 15.3s

> checking tests ...
  See below...

-- Test failures ------------------------------------------------- testthat ----

> library(testthat)
> library(gbifdb)
> 
> test_check("gbifdb")
[ FAIL 0 | WARN 0 | SKIP 2 | PASS 9 ]

══ Skipped tests (2) ═══════════════════════════════════════════════════════════
• On CRAN (2): 'test_gbifdb.R:53:3', 'test_gbifdb.R:66:3'

[ FAIL 0 | WARN 0 | SKIP 2 | PASS 9 ]
> 
> proc.time()
   user  system elapsed 
  0.711   0.067   0.808 
Warning messages:
1: Connection is garbage-collected, use dbDisconnect() to avoid this. 
2: Database is garbage-collected, use dbDisconnect(con, shutdown=TRUE) or duckdb::duckdb_shutdown(drv) to avoid this. 
Error: bad value
Execution halted

1 error x | 0 warnings v | 0 notes v

With local arrow

-- R CMD check results --------------------------------------- gbifdb 0.1.2 ----
Duration: 15.9s

0 errors v | 0 warnings v | 0 notes v

nanoarrow

With CRAN arrow

-- R CMD check results ---------------------------------- nanoarrow 0.2.0.1 ----
Duration: 18.7s

0 errors v | 0 warnings v | 0 notes v

With local arrow

-- R CMD check results ---------------------------------- nanoarrow 0.2.0.1 ----
Duration: 19.4s

> checking Rd cross-references ... WARNING
  Missing link or links in documentation object 'as_nanoarrow_array.Rd':
    ‘[arrow:array]{arrow::Array}’ ‘[arrow:array]{arrow::StructArray}’
  
  See section 'Cross-references' in the 'Writing R Extensions' manual.

0 errors v | 1 warning x | 0 notes v

pins

With CRAN arrow

-- R CMD check results ----------------------------------------- pins 1.2.0 ----
Duration: 42.4s

0 errors v | 0 warnings v | 0 notes v

With local arrow

-- R CMD check results ----------------------------------------- pins 1.2.0 ----
Duration: 43.1s

> checking tests ...
  See below...

-- Test failures ------------------------------------------------- testthat ----

> library(testthat)
> library(pins)
> 
> test_check("pins")
Guessing `type = 'rds'`
> <http://127.0.0.1:49432/data.txt> is not cacheable
> <http://127.0.0.1:49432/x.rds> is not cacheable
> <http://127.0.0.1:49436/x/20230724T142915Z-c3943/data.txt> is not cacheable
> <http://127.0.0.1:49436/x/20230724T142915Z-c3943/x.json> is not cacheable
> <http://127.0.0.1:49440/y/20230724T142915Z-cba09/data.txt> is not cacheable
> <http://127.0.0.1:49440/y/20230724T142915Z-5026d/data.txt> is not cacheable
[ FAIL 3 | WARN 0 | SKIP 66 | PASS 183 ]

══ Skipped tests (66) ══════════════════════════════════════════════════════════
• On CRAN (52): 'test-board_connect_bundle.R:36:3',
  'test-board_connect_bundle.R:41:3', 'test-board_connect_server.R:22:3',
  'test-board_connect_server.R:31:3', 'test-board_connect_server.R:51:3',
  'test-board_connect_server.R:60:3', ???, ???, 'test-board_folder.R:9:3',
  'test-board_folder.R:34:3', 'test-board_folder.R:42:3',
  'test-board_folder.R:88:3', 'test-board_url.R:54:3',
  'test-board_url.R:154:3', 'test-board_url.R:168:3', 'test-board_url.R:180:3',
  'test-board_url.R:199:3', 'test-board_url.R:218:3', 'test-board_url.R:237:3',
  'test-legacy_board.R:2:3', 'test-legacy_datatxt.R:4:3',
  'test-legacy_datatxt.R:16:3', 'test-legacy_datatxt.R:25:3',
  'test-legacy_datatxt.R:35:3', 'test-legacy_local.R:43:3',
  'test-legacy_packages.R:23:3', 'test-legacy_registry.R:30:3',
  'test-meta.R:13:3', 'test-meta.R:20:3', 'test-meta.R:24:3',
  'test-pin-delete.R:11:3', 'test-pin-meta.R:2:3',
  'test-pin-read-write.R:35:3', 'test-pin-read-write.R:41:3',
  'test-pin-read-write.R:58:3', 'test-pin-read-write.R:80:3',
  'test-pin-read-write.R:88:3', 'test-pin-upload-download.R:17:3',
  'test-pin-upload-download.R:46:3', 'test-pin-upload-download.R:61:3',
  'test-pin.R:76:3', 'test-pin.R:108:3', 'test-pin_info.R:5:3',
  'test-pin_info.R:23:3', 'test-pin_info.R:33:3', 'test-pin_search.R:27:3',
  'test-pin_versions.R:8:3', 'test-pin_versions.R:18:3',
  'test-pin_versions.R:26:3', 'test-pin_versions.R:41:3',
  'test-pin_versions.R:67:3', 'test-pin_versions.R:91:3'
• board_azure() tests require PINS_AZURE_KEY (3):
  'test-board_azure_adls2.R:1:1', 'test-board_azure_blob.R:1:1',
  'test-board_azure_file.R:1:1'
• board_connect() tests requires `creds.rds` (4): 'test-board_connect.R:2:1',
  'test-board_connect_url.R:4:3', 'test-board_connect_url.R:22:3',
  'test-board_connect_url.R:33:3'
• board_gcs() tests require PINS_GCS_PASSWORD (1): 'test-board_gcs.R:1:1'
• board_ms365() tests require PINS_MS365_TEST_DRIVE (1):
  'test-board_ms365.R:1:1'
• board_s3() tests require PINS_AWS_ACCESS_KEY, PINS_AWS_SECRET_ACCESS_KEY (1):
  'test-board_s3.R:1:1'
• legacy_azure() tests require TEST_AZURE_CONTAINER, TEST_AZURE_ACCOUNT,
  TEST_AZURE_KEY (1): 'test-legacy_azure.R:16:1'
• legacy_gcloud() tests require TEST_GOOGLE_BUCKET (1):
  'test-legacy_gcloud.R:12:1'
• legacy_github() tests require TEST_GITHUB_REPO, TEST_GITHUB_BRANCH (1):
  'test-legacy_github.R:1:1'
• legacy_s3() tests require TEST_AWS_BUCKET, TEST_AWS_KEY, TEST_AWS_SECRET,
  TEST_AWS_REGION (1): 'test-legacy_s3.R:14:1'

══ Failed tests ════════════════════════════════════════════════════════════════
── Failure ('test-pin-read-write.R:12:3'): can round trip all types ────────────
pin_read(board, "df-2") (`actual`) not equal to `df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"
── Failure ('test-pin-read-write.R:15:3'): can round trip all types ────────────
pin_read(board, "df-2") (`actual`) not equal to `df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"
── Failure ('test-pin-read-write.R:18:3'): can round trip all types ────────────
pin_read(board, "df-3") (`actual`) not equal to `df` (`expected`).

`class(actual)`:   "tbl_df" "tbl" "data.frame"
`class(expected)`:                "data.frame"

[ FAIL 3 | WARN 0 | SKIP 66 | PASS 183 ]
Error: Test failures
Execution halted

1 error x | 0 warnings v | 0 notes v

thisisnic · 2023-07-26T08:07:28Z

We also want to cherry pick #36884
We need this docs update too: #37020

thisisnic · 2023-08-03T12:09:34Z

We should flag #34775 as a breaking change

thisisnic · 2023-08-03T13:49:56Z

revdepcheck failures:

pins - arrow 13.0.0 compatibility rstudio/pins-r#761
dataversionr - arrow 13.0.0 compatibility riazarbi/dataversionr#1

I've mentioned that we are planning to release to CRAN in 2 weeks from now (17th August)

thisisnic · 2023-08-23T08:33:31Z

We need to also grab the missing NEWS items documentation: #37326

thisisnic · 2023-08-25T10:33:19Z

And this one so we don't fail on CRAN: #37387

thisisnic · 2023-08-30T09:13:21Z

I've also cherry-picked this into this branch as we were failing on R-devel: #37457

thisisnic · 2023-09-06T14:06:28Z

Done!

thisisnic self-assigned this Jul 21, 2023

thisisnic added the Component: R label Jul 24, 2023

thisisnic closed this as completed Sep 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[R] CRAN packaging checklist for version 13.0.0 #36802

[R] CRAN packaging checklist for version 13.0.0 #36802

thisisnic commented Jul 21, 2023 •

edited

Loading

thisisnic commented Jul 21, 2023

thisisnic commented Jul 21, 2023

paleolimbot commented Jul 24, 2023

dataversionr

With CRAN arrow

With local arrow

gbifdb

With CRAN arrow

With local arrow

nanoarrow

With CRAN arrow

With local arrow

pins

With CRAN arrow

With local arrow

thisisnic commented Jul 26, 2023 •

edited

Loading

thisisnic commented Aug 3, 2023

thisisnic commented Aug 3, 2023 •

edited

Loading

thisisnic commented Aug 23, 2023

thisisnic commented Aug 25, 2023

thisisnic commented Aug 30, 2023

thisisnic commented Sep 6, 2023

[R] CRAN packaging checklist for version 13.0.0 #36802

[R] CRAN packaging checklist for version 13.0.0 #36802

Comments

thisisnic commented Jul 21, 2023 • edited Loading

Packaging checklist for CRAN release

Before the release candidate is cut

After release candidate has been cut

Optional: PRs for autobrew and rtools (release candidate)

Prepare and check the .tar.gz that will be released to CRAN.

Release vote

PRs for autobrew and rtools (official release)

Generate R package to submit to CRAN

Check binary Arrow C++ distributions specific to the R package

CRAN submission

thisisnic commented Jul 21, 2023

thisisnic commented Jul 21, 2023

paleolimbot commented Jul 24, 2023

dataversionr

With CRAN arrow

With local arrow

gbifdb

With CRAN arrow

With local arrow

nanoarrow

With CRAN arrow

With local arrow

pins

With CRAN arrow

With local arrow

thisisnic commented Jul 26, 2023 • edited Loading

thisisnic commented Aug 3, 2023

thisisnic commented Aug 3, 2023 • edited Loading

thisisnic commented Aug 23, 2023

thisisnic commented Aug 25, 2023

thisisnic commented Aug 30, 2023

thisisnic commented Sep 6, 2023

thisisnic commented Jul 21, 2023 •

edited

Loading

thisisnic commented Jul 26, 2023 •

edited

Loading

thisisnic commented Aug 3, 2023 •

edited

Loading