Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-12958: [CI][Developer] Build + host the docs for PR branches #13913

Merged
merged 14 commits into from
Aug 19, 2022

Conversation

assignUser
Copy link
Member

No description provided.

@github-actions
Copy link

@assignUser
Copy link
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions
Copy link

Failed to render template `docs/github.linux.yml` with UndefinedError: 'pr_number' is undefined
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2882330760

@assignUser
Copy link
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions
Copy link

Revision: 8d77f24

Submitted crossbow builds: ursacomputing/crossbow @ actions-76bab72639

Task Status
preview-docs Github Actions

@assignUser
Copy link
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions
Copy link

Revision: 03a8233

Submitted crossbow builds: ursacomputing/crossbow @ actions-cff97e19a8

Task Status
preview-docs Github Actions

@assignUser assignUser marked this pull request as ready for review August 19, 2022 12:12
@assignUser
Copy link
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions
Copy link

Revision: 4d0d4ee

Submitted crossbow builds: ursacomputing/crossbow @ actions-6e06515170

Task Status
preview-docs Github Actions

@@ -969,6 +969,8 @@ tasks:
r-binary-packages:
ci: github
template: r/github.packages.yml
params:
custom_version: Unset
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a minor fix for an issue I noticed in this PR, namely that for params to work they have to be listed in tasks.yml. This also necessitates the replace( thing in the templates themselves as even empty params in task.yml are None and default does not trigger on None.

pr_number: Unset
artifacts: "build/docs.tar.gz"
flags: "-v $PWD/build/:/build/"
image: ubuntu-docs
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image: ubuntu-docs
image: ubuntu-docs

@@ -251,7 +251,7 @@
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<version>2.2.2</version>
<version>4.0.5</version>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plugin was causing problems with shallow copies, it should have been able to handle it but it looks like our version was too old. The new version introduces the cli switch I added to java_build.sh. This is now the newest possible version we can use as 5.0.0 does not support java < 11.

@assignUser
Copy link
Member Author

We will of course also need to add a "clean up"-job on crossbow that removes previews older than n-days (We could use the api to check if the pr is merged/closed but we have an issue with the cb gh-api limit already so I'd rather not spend it on this).

When using archery crossbow submit you have to include the pr number as a parameter -p pr_number=xxx. For submissions via the bot this will be passed through automatically (but the change will only take affect once this is merged, so I had to use -p to test).

@wjones127 @jorisvandenbossche review please?

@assignUser
Copy link
Member Author

🥳 https://crossbow.voltrondata.com/pr/13913/

Copy link
Member

@rok rok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great @assignUser! I can't wait to use it.
Perhaps a follow up task (outside of this PR) could be also generating Go's docs.

@rok
Copy link
Member

rok commented Aug 19, 2022

We should probably document this in https://arrow.apache.org/docs/developers/documentation.html ?

A nit about the URL: perhaps /docs_preview/# or /pr_docs/# would be more idiomatic than /pr/#?

Copy link
Member

@wjones127 wjones127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! I'm impressed you got this done this quickly.

The only thing I think needs improving in some reminder in the CI where the output of the docs is living.

Also, I guess clean up will be a follow up for later?

git config --local user.email "[email protected]"
git add docs/pr/{{ pr_number }}
git commit -m "Add docs preview for PR {{ pr_number }}"
git push
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe echo the URL where the docs are hosted here? Just thinking it would be nice to have a reminder where they appear in the CI output.

@assignUser
Copy link
Member Author

assignUser commented Aug 19, 2022

@wjones127 How about something like this (open to text suggestions :D): https://github.com/assignUser/test-repo-a/actions/runs/2890950254
image

@assignUser
Copy link
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions
Copy link

Revision: 5aaa225

Submitted crossbow builds: ursacomputing/crossbow @ actions-0d5323889b

Task Status
preview-docs Github Actions

@assignUser
Copy link
Member Author

@rok https://issues.apache.org/jira/browse/ARROW-17477

@rok rok merged commit 28cf3f9 into apache:master Aug 19, 2022
@ursabot
Copy link

ursabot commented Aug 20, 2022

Benchmark runs are scheduled for baseline = 4af3df0 and contender = 28cf3f9. 28cf3f9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.1% ⬆️0.0%] test-mac-arm
[Failed ⬇️2.19% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.28% ⬆️0.14%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 28cf3f9f ec2-t3-xlarge-us-east-2
[Failed] 28cf3f9f test-mac-arm
[Failed] 28cf3f9f ursa-i9-9960x
[Finished] 28cf3f9f ursa-thinkcentre-m75q
[Finished] 4af3df0c ec2-t3-xlarge-us-east-2
[Failed] 4af3df0c test-mac-arm
[Failed] 4af3df0c ursa-i9-9960x
[Finished] 4af3df0c ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

zagto pushed a commit to zagto/arrow that referenced this pull request Oct 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants