Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEST: arrow v18.0.0rc0 #1432

Draft
wants to merge 34 commits into
base: main
Choose a base branch
from
Draft

TEST: arrow v18.0.0rc0 #1432

wants to merge 34 commits into from

Conversation

h-vetinari
Copy link
Member

Some changes require testing across both libarrow and pyarrow; since we split off the pyarrow builds in a66c7c6, we'd need to publish a changed libarrow, test the changes in pyarrow, and then potentially mark the builds as broken. In order to simplify this process, reintroduce some end-to-end testing here that can be used to do this all in one PR.

@conda-forge-webservices
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@h-vetinari
Copy link
Member Author

@conda-forge-admin, please rerender

@h-vetinari h-vetinari marked this pull request as draft May 27, 2024 00:49
@h-vetinari h-vetinari changed the title TEST: end-to-end testing with pyarrow TEST: end-to-end testing with pyarrow 17.0.0.rc0 Jul 9, 2024
@h-vetinari
Copy link
Member Author

@conda-forge/arrow-cpp, happy to report that 17.0.0-rc0 is looking good from the POV of conda-forge. :)

@raulcd
Copy link
Member

raulcd commented Jul 9, 2024

happy to report that 17.0.0-rc0 is looking good from the POV of conda-forge. :)

Cool! I had to create a second RC today due to a minor issue found

@h-vetinari h-vetinari changed the title TEST: end-to-end testing with pyarrow 17.0.0.rc0 TEST: end-to-end testing with llvm 17 Jul 28, 2024
@conda-forge-webservices
Copy link

conda-forge-webservices bot commented Jul 28, 2024

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@h-vetinari h-vetinari changed the title TEST: end-to-end testing with llvm 17 TEST: arrow v18.0.0rc0 Oct 16, 2024
pyarrow-{core,all} are not allowed for this feedstock, spuriously failing CI
kou added a commit to apache/arrow that referenced this pull request Oct 23, 2024
…cross-compiling (#44507)

### Rationale for this change

We can't use `find_package(gRPC)` and `gRPC::grpc_cpp_plugin` for cross-compiling because it's for host. We need `grpc_cpp_plugin` for target in cross-compiling.

### What changes are included in this PR?

Add `ARROW_GRPC_CPP_PLUGIN` CMake option that overwrites `gRPC::grpc_cpp_plugin` path found by `find_package(gRPC)`.

### Are these changes tested?

Yes.

conda-forge/arrow-cpp-feedstock#1432

### Are there any user-facing changes?

Yes.
* GitHub Issue: #44448

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@h-vetinari
Copy link
Member Author

h-vetinari commented Oct 24, 2024

@jaimergp, before I undid the windows-side of the recent rerender changes, this was failing reproducibly with a weird error:

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
ERROR:conda.core.link:An error occurred while installing package 'conda-forge::vc14_runtime-14.40.33810-hcc2c482_22'.
Rolling back transaction: ...working... done
Traceback (most recent call last):
  [...]
conda.CondaMultiError: [Errno 13] Permission denied: 'C:\\bld\\apache-arrow_1729721752938\\_h_env\\vcruntime140.dll'

which seemed to be due to some busy resources and lots of errors of the kind

WARNING:conda.gateways.disk.delete:Could not remove or rename %BUILD_PREFIX%\Library\bin\libssl-3-x64.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %BUILD_PREFIX%\Library\bin\zstd.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %BUILD_PREFIX%\msvcp140.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %BUILD_PREFIX%\vcruntime140.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %BUILD_PREFIX%\vcruntime140_1.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %BUILD_PREFIX%\zlib.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
Reloading output folder (local): ...working... done
Solving environment (_h_env): ...working... done
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\DLLs\select.pyd.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\DLLs\unicodedata.pyd.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\DLLs\_asyncio.pyd.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\DLLs\_bz2.pyd.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\DLLs\_hashlib.pyd.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\DLLs\_lzma.pyd.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\DLLs\_multiprocessing.pyd.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)

This doesn't really feel related to the windows drive changes (that ran into a hard "no space left" before e55b320) so I'm wondering if this might be related to the switch to micromamba? 🤔

(FWIW, the build using micromamba and without the pyarrow-* outputs is passing on main, so this may be a corner case caused by extreme circumstances? We have 30 outputs here for the end-to-end testing; in practice the python bindings get split off to https://github.com/conda-forge/pyarrow-feedstock)

CC @jjerphan @JohanMabille

jaimergp and others added 11 commits October 28, 2024 12:58
MNT: Re-rendered with conda-build 24.7.1, conda-smithy 3.43.0, and conda-forge-pinning 2024.10.27.23.07.45
MNT: Re-rendered with conda-build 24.7.1, conda-smithy 3.43.0, and conda-forge-pinning 2024.10.27.23.07.45
MNT: Re-rendered with conda-build 24.7.1, conda-smithy 3.43.0, and conda-forge-pinning 2024.10.27.23.07.45
Copy link
Member Author

@h-vetinari h-vetinari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jaimergp, looks like you found a way to fix things! I wanted to rerender because there's still a few things that are strange to me, mainly the contradictory definition of CONDA_BLD_PATH / MINIFORGE_HOME for windows, see below.

.azure-pipelines/azure-pipelines-win.yml Outdated Show resolved Hide resolved
.azure-pipelines/azure-pipelines-win.yml Outdated Show resolved Hide resolved
@h-vetinari
Copy link
Member Author

Ah, nevermind, CI never ran

image

I had run into the same issue previously... 😅

@jaimergp
Copy link
Member

Oops, thanks for that fix with variables. It's not a micromamba v1 vs v2 issue, so let's see if it has to do with the cache trick/hack.

MNT: Re-rendered with conda-build 24.9.0, conda-smithy 3.43.1.dev12+g92312b51, and conda-forge-pinning 2024.10.29.12.41.35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants