ARROW-13199: [R] add ubuntu 21.04 to nightly builds #10611

jonkeane · 2021-06-28T20:09:01Z

No description provided.

github-actions · 2021-06-28T20:09:19Z

https://issues.apache.org/jira/browse/ARROW-13199

jonkeane · 2021-06-28T20:09:45Z

@github-actions crossbow submit test-r-rocker-r-edge-latest

github-actions · 2021-06-28T20:10:35Z

Revision: e342c87

Submitted crossbow builds: ursacomputing/crossbow @ actions-521

Task	Status
test-r-rocker-r-edge-latest

kszucs · 2021-06-29T10:27:29Z

@jonkeane the change itself looks good to me, though the build is failing.

jonkeane · 2021-06-29T13:20:47Z

Thanks for taking a look at this. I actually think that I might change the implementation and manage a docker file that does something similar in our repo (while this one is published to rocker, it looks like quite a bit of a one-off).

The failure is accurate: we are not currently able to build on gcc11. I've tried bumping both aws-c-common and aws-sdk-cpp versions. The first resolves one error, though the second one doesn't build due to changes in the way that aws-sdk-cpp handles dependencies. I'm still trying to get it, but it looks like they've transitioned to a setup where they require using git (+ submodules) to grab the dependencies.

jonkeane · 2021-06-29T20:28:40Z

@github-actions crossbow submit test-r-ubuntu-21.04 test-r-gcc-11

github-actions · 2021-06-29T20:29:36Z

Revision: 910cd0f

Submitted crossbow builds: ursacomputing/crossbow @ actions-535

Task	Status
test-r-gcc-11
test-r-ubuntu-21.04

jonkeane · 2021-06-29T20:29:51Z

Ok, I've split these out into two different jobs:

one that is 21.04 directly (which fails when building R with a segfault on loading)
one that is testing gcc 11 and that shows issues building the aws-sdk

jonkeane · 2021-06-29T20:32:29Z

I'm going to try and resolve the 21.04 issue now, but we might need to punt on gcc11 support for now, especially in light of https://issues.apache.org/jira/browse/ARROW-13134 which is pinning to a version of the aws-sdk that won't build cleanly on gcc11

pitrou · 2021-06-30T15:01:49Z

Or you can disable S3 on that build for now?

jonkeane · 2021-06-30T23:32:29Z

@github-actions crossbow submit -g r

github-actions · 2021-06-30T23:33:23Z

Revision: 42b54cf

Submitted crossbow builds: ursacomputing/crossbow @ actions-539

Task	Status
conda-linux-gcc-py36-cpu-r40
conda-linux-gcc-py37-cpu-r41
conda-osx-clang-py36-r40
conda-osx-clang-py37-r41
conda-win-vs2017-py36-r40
conda-win-vs2017-py37-r41
homebrew-r-autobrew
test-r-devdocs
test-r-gcc-11
test-r-install-local
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-rhub-ubuntu-gcc-release-latest
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos7-devtoolset-8
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42
test-r-rtools-35
test-r-ubuntu-21.04
test-r-version-compatibility
test-r-versions
test-r-without-arrow
test-ubuntu-18.04-r-sanitizer

jonkeane · 2021-07-01T02:11:54Z

@github-actions crossbow submit -g r

github-actions · 2021-07-01T02:16:15Z

Revision: 9e932c0

Submitted crossbow builds: ursacomputing/crossbow @ actions-541

Task	Status
conda-linux-gcc-py36-cpu-r40
conda-linux-gcc-py37-cpu-r41
conda-osx-clang-py36-r40
conda-osx-clang-py37-r41
conda-win-vs2017-py36-r40
conda-win-vs2017-py37-r41
homebrew-r-autobrew
test-r-devdocs
test-r-gcc-11
test-r-install-local
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-rhub-ubuntu-gcc-release-latest
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos7-devtoolset-8
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42
test-r-rtools-35
test-r-ubuntu-21.04
test-r-version-compatibility
test-r-versions
test-r-without-arrow
test-ubuntu-18.04-r-sanitizer

… display failure

jonkeane · 2021-07-01T15:51:35Z

@github-actions crossbow submit test-r-ubuntu-21.04 test-r-gcc-11

github-actions · 2021-07-01T15:52:32Z

Revision: 973f4b0

Submitted crossbow builds: ursacomputing/crossbow @ actions-548

Task	Status
test-r-gcc-11
test-r-ubuntu-21.04

kszucs · 2021-07-02T10:43:38Z

ci/docker/linux-apt-r.dockerfile

@@ -19,6 +19,9 @@ ARG base
 FROM ${base}
 ARG arch

+ARG arrow_build_static="OFF"


Please remove these.

kszucs · 2021-07-02T10:43:58Z

ci/docker/linux-apt-r.dockerfile

@@ -84,7 +97,7 @@ COPY python/requirements-build.txt /arrow/python/
 RUN pip install -r arrow/python/requirements-build.txt

 ENV \
-    ARROW_BUILD_STATIC=OFF \
+    ARROW_BUILD_STATIC=${arrow_build_static} \


Restore the default values instead of using a build argument.

kszucs · 2021-07-02T10:44:05Z

ci/docker/linux-apt-r.dockerfile

@@ -95,7 +108,7 @@ ENV \
    ARROW_PARQUET=ON \
    ARROW_PLASMA=OFF \
    ARROW_PYTHON=ON \
-    ARROW_S3=ON \
+    ARROW_S3=${arrow_s3} \


Same as above.

kszucs · 2021-07-02T10:45:31Z

dev/tasks/tasks.yml

+        UBUNTU: 21.04
+        CLANG_TOOLS: 9 # can remove this when >=9 is the default
+        GCC_VERSION: 11
+        ARROW_S3: OFF # S3 support is not buildable with gcc11 right now


Add:

env: ... flags: "-e ARROW_S3=OFF"

kszucs · 2021-07-02T10:46:01Z

docker-compose.yml

@@ -1006,6 +1008,8 @@ services:
        arch: ${ARCH}
        r: ${R}
        base: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp
+        gcc_version: ${GCC_VERSION}
+        arrow_s3: ${ARROW_S3}


Remove arrow_s3.

kszucs · 2021-07-02T10:49:06Z

docker-compose.yml

@@ -1019,6 +1023,38 @@ services:
        /arrow/ci/scripts/python_build.sh /arrow /build &&
        /arrow/ci/scripts/r_test.sh /arrow"

+  ubuntu-r-static:


ubuntu-r-static ideally should be removed and called with:

archery docker run -e ARROW_BUILD_STATIC=ON -e ANOTHER_ENV_VAR=ETC ubuntu-r

See an example of a static C++ build using the ubuntu-cpp image.

Ah, ha! That's what I was missing. Thanks for that pointer

kszucs

Build time variables

The variables in the .env files are build time variables used during the docker build commands, in other words used to produce docker images. We pass these arguments as environment variables to the archery command itself:

UBUNTU=21.04 archery docker run ubuntu-cpp

Runtime variables

Whereas the "runtime" variables passed to the docker container to alter the behavior of the command we run inside the image:

/bin/bash -c "
        /arrow/ci/scripts/cpp_build.sh /arrow /build &&
        /arrow/ci/scripts/python_build.sh /arrow /build &&
        /arrow/ci/scripts/r_test.sh /arrow"

We typically list these runtime variables at the and of the dockerfiles and in the docker-compose.yml as ENV and environment variables (not build-arguments). We can pass custom environment variables to the containers using:

archery docker run --env ARROW_PARQUET=OFF -e ARROW_BUILD_TESTS=ON ubuntu-cpp

jonkeane · 2021-07-02T15:14:07Z

@github-actions crossbow submit test-r-ubuntu-21.04 test-r-gcc-11

github-actions · 2021-07-02T15:18:41Z

Revision: b209a48

Submitted crossbow builds: ursacomputing/crossbow @ actions-561

Task	Status
test-r-gcc-11
test-r-ubuntu-21.04

jonkeane · 2021-07-02T17:02:38Z

Thanks for the help @kszucs , I've made the changes that implement what you suggested (I'll push them in a bit).

One difference using the same ubuntu-r docker layer for both non-static and static is that the non-static one will build a version of the cpp that we then ignore later when R is being installed. Since we're running the same command:

    command: >
      /bin/bash -c "
        /arrow/ci/scripts/cpp_build.sh /arrow /build &&
        /arrow/ci/scripts/python_build.sh /arrow /build &&
        /arrow/ci/scripts/r_test.sh /arrow"

That wouldn't be a huge deal, but I'm running into an issue with the GCC11 build where if we do run /arrow/ci/scripts/cpp_build.sh /arrow /build first, the r package fails to build the cpp when it needs to: https://github.com/ursacomputing/crossbow/runs/2973848414#step:7:6488 (note: it's possible this is resolveable with a different change, I haven't dug too deeply into it — and don't want to delay the other fixes in this ticket tracking down this particular failure)

If I change this command to be:

    command: >
      /bin/bash -c "
        /arrow/ci/scripts/r_test.sh /arrow"

it passes just fine. Is there a way with archery to override the docker command (e.g. from the tasks.yml file) (like I can do with the docker-compose commands below)? I've looked and tried a few things, but nothing seemed to work.

Another alternative would be to create another docker-compose service like the following (and we might even move some of the content in flags here too). Though this is similar to what I had before that we've changed

  ubuntu-r-static:
    extends: ubuntu-r
    command: /bin/bash -c "/arrow/ci/scripts/r_test.sh /arrow"

jonkeane · 2021-07-02T17:33:39Z

Hmm, it looks like that supplying a custom command should work out of the box now that I read the source, but this is what I'm getting when I try with this command:

UBUNTU=21.04 ARROW_R_DEV=TRUE CLANG_TOOLS=9 GCC_VERSION=11 docker-compose run -e ARROW_SOURCE_HOME="/arrow" -e FORCE_BUNDLED_BUILD=TRUE -e LIBARROW_BUILD=TRUE -e ARROW_S3=OFF ubuntu-r "/bin/bash -c '/arrow/ci/scripts/r_test.sh /arrow'"
Creating arrow_ubuntu-r_run ... done
Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: exec: "/bin/bash -c '/arrow/ci/scripts/r_test.sh /arrow'": stat /bin/bash -c '/arrow/ci/scripts/r_test.sh /arrow': no such file or directory: unknown
ERROR: 1

jonkeane · 2021-07-02T18:31:18Z

@github-actions crossbow submit test-r-ubuntu-21.04 test-r-gcc-11

github-actions · 2021-07-02T18:32:24Z

Revision: 5ad97c6

Submitted crossbow builds: ursacomputing/crossbow @ actions-563

Task	Status
test-r-gcc-11
test-r-ubuntu-21.04

pachadotdev · 2021-07-03T02:54:44Z

I moved to 21.04 on my laptop to avoid drivers problems... and now I'm solving problems to install Arrow, like moving to a better circle in Dante's Inferno

Here are the installation steps I used

git clone -b ARROW-13199-r-ubuntu-21.04 [email protected]:jonkeane/arrow.git ARROW-13199-r-ubuntu-21.04

cd ARROW-13199-r-ubuntu-21.04/r
ln -s '/home/pacha/github/arrow-setup/02-install-arrow.sh'
bash 02-install-arrow.sh

02-install-arrow.sh corresponds to https://github.com/pachadotdev/arrow-setup/blob/main/02-install-arrow.sh, and I installed with ninja configures as in https://github.com/pachadotdev/arrow-setup/blob/main/01-install-dependencies.sh

This is the output after building successfully from this branch:

> arrow_info()
Arrow package version: 4.0.1.9000

Capabilities:
               
dataset    TRUE
parquet    TRUE
s3         TRUE
utf8proc   TRUE
re2        TRUE
snappy     TRUE
gzip       TRUE
brotli     TRUE
zstd       TRUE
lz4        TRUE
lz4_frame  TRUE
lzo       FALSE
bz2        TRUE
jemalloc   TRUE
mimalloc  FALSE

Memory:
                  
Allocator jemalloc
Current    0 bytes
Max        0 bytes

Runtime:
                        
SIMD Level          avx2
Detected SIMD Level avx2

Build:
                                                             
C++ Library Version                            5.0.0-SNAPSHOT
C++ Compiler                                              GNU
C++ Compiler Version                                   10.3.0
Git ID               5ad97c6c43b936a64c7124f0407799a768a5acae

jonkeane · 2021-07-05T19:58:26Z

I've created https://issues.apache.org/jira/browse/ARROW-13261 to remove the additional service once GCC 11 building is fixed (either via https://issues.apache.org/jira/browse/ARROW-13241 or another ticket)

kszucs · 2021-07-06T11:09:09Z

@jonkeane Sorry, just seen the notification. Just wanted to confirm that your solution looks good to me.

jonkeane · 2021-07-06T14:19:28Z

No problem, thanks for the confirmation!

kou · 2021-07-06T23:49:34Z

@jonkeane It seems that we can't use extends in docker-compose.yml.

$ archery docker run python-wheel-manylinux-2014 bash
...
ValueError: Found errors with docker-compose:
 - The Compose file '/home/kou/work/cpp/arrow.kou/docker-compose.yml' is invalid because:
 - Unsupported config option for services.ubuntu-r-only-r: 'extends'

jonkeane · 2021-07-07T00:02:14Z

Out of curiosity, what version of docker-compose are you running? It looks like it was removed from the v3 specification, but then added back in. I'm running version 1.29.1 and it seems to work for me.

Regardless of it being supported in new versions, we might want to revert this to the prior full but more repetitive specification I had earlier in the PR process (or we could figure out / fix overriding the command issue I ran into).

kou · 2021-07-07T00:09:30Z

Thanks. I'm using 1.25.0 installed by apt: https://packages.debian.org/search?keywords=docker-compose

I'll use newer docker-compose.

We need it for "extends". See also: * https://issues.apache.org/jira/browse/ARROW-13199 * apache#10611 * docker/compose#7588

We need it for "extends". See also: * https://issues.apache.org/jira/browse/ARROW-13199 * #10611 * docker/compose#7588 Closes #10681 from kou/require-docker-compose-1.27.0-or-later Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

github-actions bot added the Component: C++ label Jun 28, 2021

github-actions bot added the Component: R label Jun 30, 2021

jonkeane added 9 commits June 30, 2021 21:11

use r-edge for ubuntu 20.10 / bleeding edge installs

6860242

bump aws_c_common version

9c45c13

bump

3eac8ed

re-use arrow CI for new gcc version

cd4e7cf

remove rocker edge image

d729cb6

oops, take out a silly exit from testing

71892cf

undo aws-sdk changes

c07ba8f

bikeshedding a comment

63ec5ee

Another round

9e932c0

jonkeane force-pushed the ARROW-13199-r-ubuntu-21.04 branch from 42b54cf to 9e932c0 Compare July 1, 2021 02:11

add changes for the R builds to selectively turn off ARROW_S3, and to…

973f4b0

… display failure

oops quotes

c47a952

kszucs reviewed Jul 2, 2021

View reviewed changes

kszucs requested changes Jul 2, 2021

View reviewed changes

PR comments

b209a48

test something...

2f2d659

try with a separate service for now

5ad97c6

jonkeane closed this in 41c4143 Jul 5, 2021

kou added a commit to kou/arrow that referenced this pull request Jul 8, 2021

[CI] Require docker-compose 1.27.0 or later

ef3edc7

We need it for "extends". See also: * https://issues.apache.org/jira/browse/ARROW-13199 * apache#10611 * docker/compose#7588

kou mentioned this pull request Jul 8, 2021

ARROW-13286: [CI] Require docker-compose 1.27.0 or later #10681

Closed

This was referenced Jul 7, 2021

[R] add ubuntu 21.04 to nightly builds #28889

Closed

[CI] Require docker-compose 1.27.0 or later #18737

Closed

[R] Install fails with LTO flags #28586

Closed

ARROW-13199: [R] add ubuntu 21.04 to nightly builds #10611

ARROW-13199: [R] add ubuntu 21.04 to nightly builds #10611

Conversation

jonkeane commented Jun 28, 2021

github-actions bot commented Jun 28, 2021

jonkeane commented Jun 28, 2021

github-actions bot commented Jun 28, 2021

kszucs commented Jun 29, 2021

jonkeane commented Jun 29, 2021

jonkeane commented Jun 29, 2021

github-actions bot commented Jun 29, 2021

jonkeane commented Jun 29, 2021

jonkeane commented Jun 29, 2021

pitrou commented Jun 30, 2021

jonkeane commented Jun 30, 2021

github-actions bot commented Jun 30, 2021

jonkeane commented Jul 1, 2021

github-actions bot commented Jul 1, 2021

jonkeane commented Jul 1, 2021

github-actions bot commented Jul 1, 2021

kszucs Jul 2, 2021

Choose a reason for hiding this comment

kszucs Jul 2, 2021

Choose a reason for hiding this comment

kszucs Jul 2, 2021

Choose a reason for hiding this comment

kszucs Jul 2, 2021

Choose a reason for hiding this comment

kszucs Jul 2, 2021

Choose a reason for hiding this comment

kszucs Jul 2, 2021 • edited Loading

Choose a reason for hiding this comment

kszucs Jul 2, 2021

Choose a reason for hiding this comment

jonkeane Jul 2, 2021

Choose a reason for hiding this comment

kszucs left a comment

Choose a reason for hiding this comment

Build time variables

Runtime variables

jonkeane commented Jul 2, 2021

github-actions bot commented Jul 2, 2021

jonkeane commented Jul 2, 2021

jonkeane commented Jul 2, 2021

jonkeane commented Jul 2, 2021

github-actions bot commented Jul 2, 2021

pachadotdev commented Jul 3, 2021

jonkeane commented Jul 5, 2021

kszucs commented Jul 6, 2021

jonkeane commented Jul 6, 2021

kou commented Jul 6, 2021

jonkeane commented Jul 7, 2021

kou commented Jul 7, 2021

kszucs Jul 2, 2021 •

edited

Loading