Rebuild for CUDA 12 #1079

conda-forge-admin · 2023-06-04T20:27:43Z

Hi! This is the friendly automated conda-forge-webservice.

I've started rerendering the recipe as instructed in #1078.

If I find any needed changes to the recipe, I'll push them to this PR shortly. Thank you for waiting!

Here's a checklist to do before merging.

Address Rebuild for CUDA 12 #1079 (comment)
Bump the build number.

Fixes #1080
Fixes #1078
Fixes #974
Fixes #942

conda-forge-webservices · 2023-06-04T20:27:49Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

jakirkham · 2023-06-04T20:32:28Z

@conda-forge-admin, please re-render

github-actions · 2023-06-04T20:35:45Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/arrow-cpp-feedstock/actions/runs/5171072276.

jakirkham · 2023-06-04T20:48:10Z

@conda-forge-admin, please re-render

github-actions · 2023-06-04T20:52:10Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/arrow-cpp-feedstock/actions/runs/5171125185.

jakirkham · 2023-06-04T20:54:46Z

This logic may also need to be limited to CUDA 11 or earlier

arrow-cpp-feedstock/recipe/build-arrow.sh

Lines 30 to 48 in 5079e41

    
           # Enable CUDA support 
        
           if [[ ! -z "${cuda_compiler_version+x}" && "${cuda_compiler_version}" != "None" ]] 
        
           then 
        
               if [[ -z "${CUDA_HOME+x}" ]] 
        
               then 
        
                   echo "cuda_compiler_version=${cuda_compiler_version} CUDA_HOME=$CUDA_HOME" 
        
                   CUDA_GDB_EXECUTABLE=$(which cuda-gdb || exit 0) 
        
                   if [[ -n "$CUDA_GDB_EXECUTABLE" ]] 
        
                   then 
        
                       CUDA_HOME=$(dirname $(dirname $CUDA_GDB_EXECUTABLE)) 
        
                   else 
        
                       echo "Cannot determine CUDA_HOME: cuda-gdb not in PATH" 
        
                       return 1 
        
                   fi 
        
               fi 
        
               EXTRA_CMAKE_ARGS=" ${EXTRA_CMAKE_ARGS} -DARROW_CUDA=ON -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME} -DCMAKE_LIBRARY_PATH=${CONDA_BUILD_SYSROOT}/lib" 
        
           else 
        
               EXTRA_CMAKE_ARGS=" ${EXTRA_CMAKE_ARGS} -DARROW_CUDA=OFF" 
        
           fi

Here's how another maintainer approached their case

Edit: It's also possible this workaround is no longer needed and so can simply be dropped

h-vetinari · 2023-06-05T06:45:49Z

This logic may also need to be limited to CUDA 11 or earlier

Feel free to update this. AFAIU, we only need to build with one cuda version anyway, so we can just remove those workarounds.

h-vetinari

Thanks for the PR! Pleasantly surprised to see this passing¹ right off the bat - great work!

Aside from removing the workarounds (which would be great), I just have a comment about the run-constraints.

the aarch+CUDA failure is Skip some more crashy tests on aarch+CUDA #1081, and the windows failure is infra flakiness ↩

recipe/meta.yaml

jakirkham · 2023-06-07T21:07:56Z

@conda-forge-admin, please re-render

jakirkham · 2023-06-07T21:27:28Z

Hmm...it looks like the CUDA builds are being skipped on CI altogether?

Skipped: apache-arrow from /home/conda/recipe_root defines build/skip for this configuration ({'cuda_compiler_version': '10.2', 'target_platform': 'linux-64', 'channel_targets': 'conda-forge main'}).

Also re-rendering removed cuda_compiler_version_min for some reason

Makes a few changes that will generalize better over CUDA 11 / 12 like use `cuda-version` and referring to the `compiler("cuda")` package directly (since there are differences in what is in its `run_exports`).

Only the CUDA driver version matters for Arrow CUDA components. Co-authored-by: h-vetinari <[email protected]> Co-authored-by: Keith Kraus <[email protected]>

This looks for `$CUDA_HOME` when the `nvcc` wrapper script is installed by searching for `cuda-gdb`, which is in the CUDA 11 Docker images. However this isn't available on CUDA 12 (unless `cuda-gdb` is installed). That said, this logic was added to the `nvcc` wrapper script a while ago. So this can just be dropped in favor of relying on the `nvcc` wrapper sript to set this correctly.

…nda-forge-pinning 2023.06.08.00.22.40

h-vetinari · 2023-06-08T04:33:26Z

Also please feel free to push changes directly

I took you at your word on this, hope that's OK. :)

Rather than asking you to do the rebasing work I'd like to have (to clean up the back-and-forth here which messes up the git history), I did it myself. Let me know if you want to do it differently.

(sorry for the force-pushes)

h-vetinari

LGTM from my POV now!

jakirkham · 2023-06-08T04:40:46Z

Yep that's ok. Thanks for taking the lead! 🙏

Would just suggest wrapping up the discussion in this thread ( #1079 (comment) ). Perhaps that is just acknowledging there that users need new enough Conda & Mamba versions for this to work

jakirkham · 2023-06-08T04:53:37Z

@kkraus14 , anything else or is this good to go?

kkraus14 · 2023-06-08T18:48:57Z

~~I think we only want to build a CUDA variant with the minimum cuda compiler version that we support. It gives compatibility for newer versions and reduces the CI matrix a bit.~~

EDIT: This is already happening, apologies.

jakirkham · 2023-06-08T19:33:47Z

All good. It is important that we get this right. So thanks for looking more closely and confirming 😄

jakirkham · 2023-06-08T19:35:42Z

Asking the bot to merge when it is done, since it seems we are good with the changes here

Do we want to start backporting this now or are there other changes we want to combine this with on the older versions?

h-vetinari · 2023-06-08T21:46:59Z

Asking the bot to merge when it is done, since it seems we are good with the changes here

Aarch+cuda is inexplicably broken on some tests. I'm debugging this in #1081

jakirkham · 2023-06-08T21:53:38Z

Ah ok. No worries. Had tried restarting, but I guess the same issues came back

jakirkham · 2023-06-09T01:04:09Z

Do we want to start backporting this now or are there other changes we want to combine this with on the older versions?

Any thoughts on this question?

h-vetinari · 2023-06-09T01:34:46Z

Yes, we want to backport. I usually do this for the maintenance branches whenever some relevant changes accumulate (or the next migrator drops by), but feel free to open PRs already if you want!

jakirkham · 2023-06-09T06:24:22Z

Submitted PRs to backport these changes to the respective version branches that appear to still be supported:

h-vetinari · 2023-06-09T06:27:23Z

Submitted PRs to backport these changes to the respective version branches

Thanks! It's a bit weird to me that the render commit is ordered before adding the migration, but whatever...

that appear to still be supported

I't not just appearance. Policy is: support for last 4 released arrow versions.

jakirkham · 2023-06-09T06:31:53Z

Ah was trying to have a re-render in the beginning that refreshed things before applying changes. Re-rendering at the end was a no-op locally

Makes sense. Was just noting that in case I missed something

h-vetinari · 2023-06-09T07:54:06Z

While highly appreciated, there's no point in restarting the merge of this PR on main. There hasn't been a passing run on aarch+CUDA for ~20 runs across various PRs & merges, but I seem to finally have cracked which tests are responsible in #1081

conda-forge-admin requested review from cpcloud, h-vetinari, jakirkham, jreback and kou as code owners June 4, 2023 20:27

conda-forge-webservices bot mentioned this pull request Jun 4, 2023

@conda-forge-admin, please re-render #1078

Closed

conda-forge-admin requested review from kszucs, leifwalsh, nealrichardson, pcmoritz, pearu, pitrou, robertnishihara, siddharthteotia, wesm and xhochy as code owners June 4, 2023 20:27

jakirkham changed the title ~~MNT: rerender~~ Rebuild for CUDA 12 Jun 4, 2023

h-vetinari reviewed Jun 5, 2023

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

h-vetinari mentioned this pull request Jun 5, 2023

Workaround on Windows for CUDA_HOME & CUDA_PATH? #1080

Closed

kkraus14 reviewed Jun 5, 2023

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

kkraus14 reviewed Jun 5, 2023

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

jakirkham and others added 6 commits June 8, 2023 15:32

Refresh CUDA usage to align with CUDA 12

70bf53a

Makes a few changes that will generalize better over CUDA 11 / 12 like use `cuda-version` and referring to the `compiler("cuda")` package directly (since there are differences in what is in its `run_exports`).

Constrain __cuda only

0544e87

Only the CUDA driver version matters for Arrow CUDA components. Co-authored-by: h-vetinari <[email protected]> Co-authored-by: Keith Kraus <[email protected]>

Drop CUDA_* path workaround on Windows

e876b2e

Bump build/number to 7

b943d32

MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.23.1, and co…

06c46c8

…nda-forge-pinning 2023.06.08.00.22.40

h-vetinari force-pushed the conda_forge_admin_1078 branch from 565a87b to 06c46c8 Compare June 8, 2023 04:33

h-vetinari approved these changes Jun 8, 2023

View reviewed changes

jakirkham requested review from kkraus14 and removed request for kkraus14 June 8, 2023 17:26

kkraus14 approved these changes Jun 8, 2023

View reviewed changes

jakirkham added the automerge Merge the PR when CI passes label Jun 8, 2023

h-vetinari merged commit f18400b into conda-forge:main Jun 8, 2023

This was referenced Jun 9, 2023

Rebuild for CUDA 12 on 11.0.x #1083

Merged

Rebuild for CUDA 12 on 10.0.x #1084

Merged

Rebuild for CUDA 12 on 9.0.x #1085

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebuild for CUDA 12 #1079

Rebuild for CUDA 12 #1079

conda-forge-admin commented Jun 4, 2023 •

edited by jakirkham

Loading

conda-forge-webservices bot commented Jun 4, 2023

jakirkham commented Jun 4, 2023

github-actions bot commented Jun 4, 2023

jakirkham commented Jun 4, 2023

github-actions bot commented Jun 4, 2023

jakirkham commented Jun 4, 2023 •

edited

Loading

h-vetinari commented Jun 5, 2023

h-vetinari left a comment

jakirkham commented Jun 7, 2023

jakirkham commented Jun 7, 2023

h-vetinari commented Jun 8, 2023

h-vetinari left a comment

jakirkham commented Jun 8, 2023

jakirkham commented Jun 8, 2023 •

edited

Loading

kkraus14 commented Jun 8, 2023 •

edited

Loading

jakirkham commented Jun 8, 2023

jakirkham commented Jun 8, 2023

h-vetinari commented Jun 8, 2023

jakirkham commented Jun 8, 2023

jakirkham commented Jun 9, 2023

h-vetinari commented Jun 9, 2023

jakirkham commented Jun 9, 2023

h-vetinari commented Jun 9, 2023

jakirkham commented Jun 9, 2023

h-vetinari commented Jun 9, 2023

Rebuild for CUDA 12 #1079

Rebuild for CUDA 12 #1079

Conversation

conda-forge-admin commented Jun 4, 2023 • edited by jakirkham Loading

conda-forge-webservices bot commented Jun 4, 2023

jakirkham commented Jun 4, 2023

github-actions bot commented Jun 4, 2023

jakirkham commented Jun 4, 2023

github-actions bot commented Jun 4, 2023

jakirkham commented Jun 4, 2023 • edited Loading

h-vetinari commented Jun 5, 2023

h-vetinari left a comment

Choose a reason for hiding this comment

Footnotes

jakirkham commented Jun 7, 2023

jakirkham commented Jun 7, 2023

h-vetinari commented Jun 8, 2023

h-vetinari left a comment

Choose a reason for hiding this comment

jakirkham commented Jun 8, 2023

jakirkham commented Jun 8, 2023 • edited Loading

kkraus14 commented Jun 8, 2023 • edited Loading

jakirkham commented Jun 8, 2023

jakirkham commented Jun 8, 2023

h-vetinari commented Jun 8, 2023

jakirkham commented Jun 8, 2023

jakirkham commented Jun 9, 2023

h-vetinari commented Jun 9, 2023

jakirkham commented Jun 9, 2023

h-vetinari commented Jun 9, 2023

jakirkham commented Jun 9, 2023

h-vetinari commented Jun 9, 2023

conda-forge-admin commented Jun 4, 2023 •

edited by jakirkham

Loading

jakirkham commented Jun 4, 2023 •

edited

Loading

jakirkham commented Jun 8, 2023 •

edited

Loading

kkraus14 commented Jun 8, 2023 •

edited

Loading