Add GitHub Actions Workflows. #471

bdice · 2022-12-22T04:14:06Z

This PR adds GitHub Actions workflows to cucim.

Task list

Coverage required for this PR:

C++ build
~~C++ tests~~ (There are no C++ tests.)
Python build
Python tests
- Some tests have been marked as xfail on ARM following discussion with @jakirkham and @grlee77. Those will be addressed in follow-up work.
Style checks

codecov-commenter · 2022-12-22T22:39:56Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-23.02@dbbeaf9). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files

@@               Coverage Diff               @@
##             branch-23.02     #471   +/-   ##
===============================================
  Coverage                ?   92.94%           
===============================================
  Files                   ?      130           
  Lines                   ?     9773           
  Branches                ?        0           
===============================================
  Hits                    ?     9084           
  Misses                  ?      689           
  Partials                ?        0

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

ajschmidt8 · 2023-01-01T18:44:09Z

ci/test_notebooks.sh

@@ -0,0 +1,62 @@
+#!/bin/bash


@bdice, I see this test_notebooks.sh file, but I don't see a corresponding job in pr.yaml.

Also, it doesn't seem that cucim notebooks are currently tested in Jenkins PRs.

Therefore, should we punt notebooks testing for cucim for a separate PR to prevent scope creep?

I was going to make a quick attempt at running notebooks and defer to a later PR if it does not work.

In general, my strategy for all the tasks we want to do in this release is to try things aggressively, and reduce scope if an “optional” task fails after some small, fixed amount of effort. Some bets pay off, others do not, but I try to keep the downside loss (budget of effort for non-essential goals) small.

ajschmidt8 · 2023-01-01T20:41:22Z

Here's a summary of the latest pytest failures:

on amd64
- it seems like all of the amd64 tests only have a single failure (1,2,3) that says ImportError: libGL.so.1: cannot open shared object file: No such file or directory. I can't seem to figure out where this .so is available in our Jenkins CI images, so I'm not sure how to fix it for GHAs.
on arm64
- the arm64 test failures (link) have the same .so error as amd64, plus a few others. Just a reminder, we don't test arm64 in PRs on Jenkins, so these failures could be legit

bdice · 2023-01-02T23:16:31Z

I think the libGL.so.1 errors are from attempting to run code that requires a display (or virtual GL output device?) on a headless machine without GL libraries.

On my WSL Ubuntu 22.04 system, I have /usr/lib/x86_64-linux-gnu/libGL.so.1 which is provided by the apt package libgl1. I am not sure if that's sufficient here. We might be able to do something like export an environment variable to tell the package to run in "headless" mode. I see there are some channels like fastai offering a package named opencv-python-headless, and a PyPI package: https://pypi.org/project/opencv-python-headless/ but no conda-forge package labeled "headless."

Some related StackOverflow answers are here: https://stackoverflow.com/questions/55313610/importerror-libgl-so-1-cannot-open-shared-object-file-no-such-file-or-directo

bdice · 2023-01-03T00:05:28Z

I think we have a few potential courses of action:

Install libgl1 in Ubuntu images and mesa-libGL in CentOS/Rocky images. I am convinced that this is necessary to use the conda-forge packages because of this comment
Use the fastai::opencv-python-headless package and skip OpenCV tests on aarch64 (package not available for aarch64)
Try the PyPI opencv-python-headless by adding it as a pip dependency in the conda environment -- not sure if this would work but it might be worth trying
Skip these tests entirely (probably not a sufficient solution)

bdice · 2023-01-03T19:07:54Z

@ajschmidt8 I implemented the first proposed solution here: rapidsai/ci-imgs#31

If you're satisfied with that, we can update the images and retry this PR.

edit: We decided against that approach and used a "headless" version of opencv from pip in 6045729. See rapidsai/ci-imgs#31 (comment)

jakirkham · 2023-01-06T09:15:54Z

python/cucim/src/cucim/skimage/exposure/tests/test_exposure.py

+# TODO: Some tests fail unexpectedly on ARM.
+ON_AARCH64 = platform.machine() == "aarch64"
+ON_AARCH64_REASON = "TODO: Test fails unexpectedly on ARM."


We decided to skip these after discussing with @grlee77 offline. Noting this here for clarity. Also for reference in a follow up issue ( #473 )

ajschmidt8 · 2023-01-06T20:59:16Z

/merge

jakirkham

Thanks all! 🙏

Sorry spotted a couple things below that I had questions on. Just want to make sure I'm not missing something

jakirkham · 2023-01-06T21:04:20Z

python/cucim/.pre-commit-config.yaml

@@ -1,20 +0,0 @@
-# To install the git pre-commit hook run:


Was this meant to be dropped?

This shouldn't exist as far as I understand. The only config that should matter is in the repo root.

Interestingly it seems we are including this one in MANIFEST.in though

In any event, agree we should only have this in one place. If we stick to dropping this one, the MANIFEST.in needs an update

Good catch! Removed from MANIFEST.in in 93c2c1e.

Thanks Bradley! 🙏

Given the file is still here (just in another location), it is not really being dropped. If it turns out we need to move it for whatever reason, we can do that in a follow up.

conda/environments/all_cuda-115_arch-x86_64.yaml

dependencies.yaml

jakirkham · 2023-01-07T01:11:19Z

/merge

This PR re-enables pre-commit checks for normalizing whitespace (trimming trailing spaces) that I disabled in #471. These pre-commit hooks existed prior to the GitHub Actions migration but they were not being enforced by CI style checks. I disabled them in #471 because enforcing them would have made the PR diff large. The primary change is in `.pre-commit-config.yaml`, and everything else was automatically fixed by `pre-commit`. ``` # TODO: re-enable this after GitHub Actions migration, it currently makes a # large number of changes that shouldn't go in the GitHub Actions PR. #- repo: https://github.com/pre-commit/pre-commit-hooks # rev: v4.4.0 # hooks: # - id: trailing-whitespace # - id: end-of-file-fixer # - id: debug-statements ``` Authors: - Bradley Dice (https://github.com/bdice) - https://github.com/jakirkham Approvers: - Gregory Lee (https://github.com/grlee77) - Ray Douglass (https://github.com/raydouglass) - https://github.com/jakirkham URL: #474

bdice added 15 commits December 21, 2022 22:10

Standardizing conda recipes.

e334c78

Draft of GitHub Actions.

f0612f8

Update pre-commit configuration.

9136ebb

Update test scripts and style.

27a1629

Add flake8 config path.

6aace33

Fix style checks.

65d1571

Update dependencies.yaml.

43cc750

Update dependencies.

4080dde

Retry CI.

0943a58

Update opencv dependency.

33277eb

Skip notebook tests.

afcad76

Fix pr.yaml, add workarounds for Jenkins compat, sccache vars.

b0f9292

Require cupy 10 to avoid issues with latest numpy.

4e6a72b

Use same testing script as other repos; enable pytest-xdist.

99b57b6

Minimize dependencies.yaml. [skip ci]

82ec7f7

bdice added 3 commits December 22, 2022 16:56

Rerun CI.

d5ab7a9

Update dependencies.

2fbb855

Exit with proper code.

0bb6559

ajschmidt8 added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Jan 1, 2023

ajschmidt8 reviewed Jan 1, 2023

View reviewed changes

ajschmidt8 added 4 commits January 1, 2023 13:49

fix/reorder mamba install pkgs

0193e8b

sort recipe lists

b4d69bd

rm extraneous YAML anchors in dependencies.yaml

83752e9

enable recently_updated plugin

df15e1a

update nightly workflows [skip ci]

ec8af4a

bdice mentioned this pull request Jan 3, 2023

Add libGL to image dependencies. rapidsai/ci-imgs#31

Closed

use opencv-python-headless for tests

6045729

bdice changed the title ~~Add GitHub Actions Workflows. [skip gpuci]~~ Add GitHub Actions Workflows. Jan 4, 2023

bdice added 3 commits January 3, 2023 22:04

Remove notebook testing.

97b48a4

Mark tests as xfail on ARM until they can be fixed.

eba8628

Merge remote-tracking branch 'upstream/branch-23.02' into gha

772e580

bdice marked this pull request as ready for review January 6, 2023 04:42

bdice requested review from a team as code owners January 6, 2023 04:42

jakirkham reviewed Jan 6, 2023

View reviewed changes

jakirkham mentioned this pull request Jan 6, 2023

Failing tests on ARM #473

Open

5 tasks

ajschmidt8 approved these changes Jan 6, 2023

View reviewed changes

jakirkham reviewed Jan 6, 2023

View reviewed changes

bdice commented Jan 6, 2023

View reviewed changes

dependencies.yaml Outdated Show resolved Hide resolved

bdice added 2 commits January 6, 2023 16:09

Add yasm to build dependencies.

b91daf5

Remove .pre-commit-config.yaml from MANIFEST.in.

93c2c1e

jakirkham approved these changes Jan 7, 2023

View reviewed changes

rapids-bot bot merged commit a2c19e9 into rapidsai:branch-23.02 Jan 7, 2023

bdice mentioned this pull request Jan 10, 2023

Normalize whitespace. #474

Merged

jakirkham added this to the v23.02.00 milestone Feb 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GitHub Actions Workflows. #471

Add GitHub Actions Workflows. #471

bdice commented Dec 22, 2022 •

edited

Loading

codecov-commenter commented Dec 22, 2022 •

edited

Loading

ajschmidt8 Jan 1, 2023

bdice Jan 1, 2023

ajschmidt8 commented Jan 1, 2023

bdice commented Jan 2, 2023 •

edited

Loading

bdice commented Jan 3, 2023

bdice commented Jan 3, 2023 •

edited

Loading

jakirkham Jan 6, 2023 •

edited

Loading

ajschmidt8 commented Jan 6, 2023

jakirkham left a comment

jakirkham Jan 6, 2023

bdice Jan 6, 2023

jakirkham Jan 7, 2023

bdice Jan 7, 2023

jakirkham Jan 7, 2023

jakirkham commented Jan 7, 2023

Add GitHub Actions Workflows. #471

Add GitHub Actions Workflows. #471

Conversation

bdice commented Dec 22, 2022 • edited Loading

Task list

codecov-commenter commented Dec 22, 2022 • edited Loading

Codecov Report

ajschmidt8 Jan 1, 2023

Choose a reason for hiding this comment

bdice Jan 1, 2023

Choose a reason for hiding this comment

ajschmidt8 commented Jan 1, 2023

bdice commented Jan 2, 2023 • edited Loading

bdice commented Jan 3, 2023

bdice commented Jan 3, 2023 • edited Loading

jakirkham Jan 6, 2023 • edited Loading

Choose a reason for hiding this comment

ajschmidt8 commented Jan 6, 2023

jakirkham left a comment

Choose a reason for hiding this comment

jakirkham Jan 6, 2023

Choose a reason for hiding this comment

bdice Jan 6, 2023

Choose a reason for hiding this comment

jakirkham Jan 7, 2023

Choose a reason for hiding this comment

bdice Jan 7, 2023

Choose a reason for hiding this comment

jakirkham Jan 7, 2023

Choose a reason for hiding this comment

jakirkham commented Jan 7, 2023

bdice commented Dec 22, 2022 •

edited

Loading

codecov-commenter commented Dec 22, 2022 •

edited

Loading

bdice commented Jan 2, 2023 •

edited

Loading

bdice commented Jan 3, 2023 •

edited

Loading

jakirkham Jan 6, 2023 •

edited

Loading