Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-12958: [CI][Developer] Build + host the docs for PR branches #13913

Merged
merged 14 commits into from
Aug 19, 2022
5 changes: 5 additions & 0 deletions ci/scripts/java_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ if [[ "$(uname -s)" == "Linux" ]] && [[ "$(uname -m)" == "s390x" ]]; then
fi

mvn="mvn -B -DskipTests -Drat.skip=true -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"

if [ $ARROW_JAVA_SKIP_GIT_PLUGIN ]; then
mvn="${mvn} -Dmaven.gitcommitid.skip=true"
fi

# Use `2 * ncores` threads
mvn="${mvn} -T 2C"

Expand Down
1 change: 1 addition & 0 deletions dev/archery/archery/bot.py
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,7 @@ def submit(obj, tasks, groups, params, arrow_version):

# parse additional job parameters
params = dict([p.split("=") for p in params])
params['pr_number'] = pull_request.number

# instantiate the job object
job = Job.from_config(config=config, target=target, tasks=tasks,
Expand Down
9 changes: 2 additions & 7 deletions dev/tasks/docker-tests/github.linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,9 @@ jobs:
name: |
Docker Test {{ flags|default("") }} {{ image }} {{ command|default("") }}
runs-on: ubuntu-latest
{% if env is defined %}
env:
{% for key, value in env.items() %}
{{ key }}: "{{ value }}"
{% endfor %}
{% endif %}
{{ macros.github_set_env(env) }}
steps:
{{ macros.github_checkout_arrow(fetch_depth=fetch_depth if fetch_depth is defined else 1)|indent }}
{{ macros.github_checkout_arrow(fetch_depth=fetch_depth|default(1))|indent }}
{{ macros.github_install_archery()|indent }}

- name: Execute Docker Build
Expand Down
59 changes: 59 additions & 0 deletions dev/tasks/docs/github.linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

{% import 'macros.jinja' as macros with context %}
{% set pr_number = pr_number|replace("Unset", arrow.branch) %}
{{ macros.github_header() }}

jobs:
test:
name: Docs Preview
runs-on: ubuntu-latest
{{ macros.github_set_env(env) }}
steps:
{{ macros.github_checkout_arrow(fetch_depth=fetch_depth|default(1))|indent }}
{{ macros.github_install_archery()|indent }}

- name: Execute Docker Build
shell: bash
env:
ARROW_JAVA_SKIP_GIT_PLUGIN: true
run: |
archery docker run \
-e SETUPTOOLS_SCM_PRETEND_VERSION="{{ arrow.no_rc_version }}" \
{{ flags|default("") }} \
{{ image }} \
{{ command|default("") }}
- name: Checkout Crossbow
uses: actions/checkout@v3
with:
ref: {{ default_branch|default("master") }}
path: crossbow
fetch-depth: 1
- name: Move docs
run: |
sudo chown -R ${USER}: build
mkdir -p crossbow/docs/pr/{{ pr_number }}
rsync -a --delete build/docs/ crossbow/docs/pr/{{ pr_number }}
- name: Push changes
run: |
cd crossbow
git config --local user.name "Github Actions"
git config --local user.email "[email protected]"
git add docs/pr/{{ pr_number }}
git commit -m "Add docs preview for PR {{ pr_number }}"
git push
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe echo the URL where the docs are hosted here? Just thinking it would be nice to have a reminder where they appear in the CI output.

7 changes: 1 addition & 6 deletions dev/tasks/fuzz-tests/github.oss-fuzz.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,7 @@ jobs:
test:
name: OSS-Fuzz build test
runs-on: ubuntu-latest
{% if env is defined %}
env:
{% for key, value in env.items() %}
{{ key }}: "{{ value }}"
{% endfor %}
{% endif %}
{{ macros.github_set_env(env) }}
steps:
{{ macros.github_checkout_arrow()|indent }}

Expand Down
9 changes: 9 additions & 0 deletions dev/tasks/macros.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -352,3 +352,12 @@ on:

{% set r_release = '4.2' %}
{% set r_oldrel = '4.1' %}

{%- macro github_set_env(env) -%}
{% if env is defined %}
env:
{% for key, value in env.items() %}
{{ key }}: "{{ value }}"
{% endfor %}
{% endif %}
{%- endmacro -%}
7 changes: 1 addition & 6 deletions dev/tasks/python-minimal-build/github.linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,7 @@ jobs:
name: |
Docker Python Minimal Build {{ flags|default("") }} {{ image }} {{ command|default("") }}
runs-on: ubuntu-latest
{% if env is defined %}
env:
{% for key, value in env.items() %}
{{ key }}: "{{ value }}"
{% endfor %}
{% endif %}
{{ macros.github_set_env(env) }}
steps:
{{ macros.github_checkout_arrow(submodules=false)|indent }}

Expand Down
2 changes: 1 addition & 1 deletion dev/tasks/r/github.packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
# This allows us to set a custom version via param:
# crossbow submit --param custom_version=8.5.3 r-binary-packages
# if the param is unset defaults to the usual Ymd naming scheme
{% set package_version = custom_version|default("\\2.\'\"$(date +%Y%m%d)\"\'") %}
{% set package_version = custom_version|replace("Unset", "\\2.\'\"$(date +%Y%m%d)\"\'") %}
{% set is_fork = macros.is_fork %}

{{ macros.github_header() }}
Expand Down
12 changes: 12 additions & 0 deletions dev/tasks/tasks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -969,6 +969,8 @@ tasks:
r-binary-packages:
ci: github
template: r/github.packages.yml
params:
custom_version: Unset
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a minor fix for an issue I noticed in this PR, namely that for params to work they have to be listed in tasks.yml. This also necessitates the replace( thing in the templates themselves as even empty params in task.yml are None and default does not trigger on None.

artifacts:
- r-lib__libarrow__bin__windows__arrow-[0-9\.]+\.zip
- r-lib__libarrow__bin__centos-7__arrow-[0-9\.]+\.zip
Expand Down Expand Up @@ -1607,3 +1609,13 @@ tasks:
type: minimal_build
run: {{ kind }}
{% endfor %}

############################## Utility tasks ############################
preview-docs:
ci: github
template: docs/github.linux.yml
params:
pr_number: Unset
artifacts: "build/docs.tar.gz"
flags: "-v $PWD/build/:/build/"
image: ubuntu-docs
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image: ubuntu-docs
image: ubuntu-docs

1 change: 1 addition & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1642,6 +1642,7 @@ services:
base: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-3
environment:
<<: *ccache
ARROW_JAVA_SKIP_GIT_PLUGIN:
ARROW_CUDA: "ON"
BUILD_DOCS_C_GLIB: "ON"
BUILD_DOCS_CPP: "ON"
Expand Down
2 changes: 1 addition & 1 deletion java/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<version>2.2.2</version>
<version>4.0.5</version>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plugin was causing problems with shallow copies, it should have been able to handle it but it looks like our version was too old. The new version introduces the cli switch I added to java_build.sh. This is now the newest possible version we can use as 5.0.0 does not support java < 11.

<executions>
<execution>
<id>for-jars</id>
Expand Down