GH-35658: [Packaging] Sync conda recipes with feedstocks #35637

h-vetinari · 2023-05-17T13:39:31Z

Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65

Closes: [Packaging] Sync with conda-forge feedstocks #35658

github-actions · 2023-05-17T13:39:48Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

h-vetinari · 2023-05-17T13:40:23Z

@github-actions crossbow submit -g conda

github-actions · 2023-05-17T13:44:22Z

Revision: 8ef347b

Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-py3
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-py3
conda-win-x64-cpu-r41
conda-win-x64-cuda-py3

h-vetinari · 2023-05-17T20:05:44Z

Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:

E   pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

kou · 2023-05-17T20:29:57Z

I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh

We're using the script on Windows too.

github-actions · 2023-05-18T02:09:48Z

Closes: [Packaging] Sync with conda-forge feedstocks #35658

github-actions · 2023-05-18T02:09:50Z

⚠️ GitHub issue #35658 has been automatically assigned in GitHub to PR creator.

h-vetinari · 2023-05-18T02:11:08Z

I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:

Sounds like we should just add a dependency on the tzdata conda-forge package?

assignUser · 2023-05-18T02:52:10Z

Sounds like we should just add a dependency on the tzdata conda-forge package?

That sound's good. Thanks for taking care of the sync work, much appreciated!

kou · 2023-05-18T06:21:42Z

It may not work because it seems that tzdata doesn't include WindowsZones.xml.

h-vetinari · 2023-05-18T07:35:20Z

It may not work because it seems that tzdata doesn't include WindowsZones.xml.

Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not contain WindowsZones.xml).

The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?

raulcd · 2023-05-18T11:31:35Z

As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:

 E   pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee

kou · 2023-05-19T04:39:55Z

Ah, sorry. I misread this problem.

It seems that WindowsZones.xml isn't related to this problem.

But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265

So our code will not find tzdata installed by conda.

Could you open a new issue for tzdata related problem? We can work on this as a separated issue.

raulcd · 2023-05-19T09:07:28Z

The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files
And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.
It won't as this test was added on a different ticket that won't be added to 12.0.1 .

edit: strikethrough

h-vetinari · 2023-05-24T01:23:27Z

The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?

As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728

jorisvandenbossche · 2023-05-24T06:34:53Z

Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.

Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.

In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.

jorisvandenbossche · 2023-05-24T06:46:37Z

I opened #35735 to skip the test requiring a tz database on windows.

h-vetinari · 2023-05-24T06:58:19Z

Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.

I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO

h-vetinari · 2023-05-24T07:14:37Z

One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.

Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?

jorisvandenbossche · 2023-05-24T07:25:40Z

That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)

h-vetinari · 2023-07-24T16:47:16Z

@github-actions crossbow submit -g conda

h-vetinari · 2023-07-24T16:49:53Z

dev/tasks/conda-recipes/arrow-cpp/meta.yaml

@@ -313,7 +323,7 @@ outputs:
        - {{ pin_subpackage('pyarrow', exact=True) }}
        - clangdev {{ llvm_version }}
        - llvmdev {{ llvm_version }}
-        - cython
+        - cython <3


I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.

github-actions · 2023-07-24T16:50:09Z

Revision: e9e6605

Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-py3
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-py3
conda-win-x64-cpu-r41
conda-win-x64-cuda-py3

got removed in 0bb2d83

h-vetinari · 2023-07-24T20:58:20Z

@github-actions crossbow submit -g conda

github-actions · 2023-07-24T21:01:45Z

Revision: 17fc588

Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-py3
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-py3
conda-win-x64-cpu-r41
conda-win-x64-cuda-py3

h-vetinari · 2023-07-25T06:46:07Z

@kou @pitrou @raulcd @assignUser @jorisvandenbossche
This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.

assignUser

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections

kou

+1

kou · 2023-07-26T05:27:53Z

dev/tasks/conda-recipes/arrow-cpp/build-arrow.sh

    sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
-    sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
+    sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake


We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.

That sounds great, I'll change that on the feedstock and in the next sync

conbench-apache-arrow · 2023-08-02T05:16:45Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

…e#35637) Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65 * Closes: apache#35658 Authored-by: H. Vetinari <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

h-vetinari added 5 commits May 18, 2023 00:34

update arrow ci support files

85ffea1

sync current state of pyarrow recipe

5b8f4a9

add cross-compiled conda jobs for CUDA on aarch64/ppc64le

45a24c3

sync r-arrow recipe

03e6af5

sync ci support for r-arrow, reinstate R 4.1 builds

8ef347b

h-vetinari requested review from assignUser, kou and raulcd as code owners May 17, 2023 13:39

h-vetinari mentioned this pull request May 17, 2023

[main] Rebuild for aws_crt_cpp0201, google_cloud_cpp2101, re220230302 conda-forge/arrow-cpp-feedstock#1053

Merged

github-actions bot added the awaiting review Awaiting review label May 17, 2023

h-vetinari changed the title ~~Sync conda recipes with feedstocks~~ GH-35658: Sync conda recipes with feedstocks May 18, 2023

h-vetinari mentioned this pull request May 24, 2023

[Python] test_total_bytes_allocated failing on CI #35728

Closed

mattijn mentioned this pull request Jun 2, 2023

Sanitize timestamps in arrow tables vega/altair#3076

Merged

h-vetinari added 3 commits July 16, 2023 22:03

Merge remote-tracking branch 'up/main' into conda_update

95effab

resync arrow-cpp

1bdaefa

sync .ci_support files

e9e6605

h-vetinari commented Jul 24, 2023

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jul 24, 2023

h-vetinari added 2 commits July 24, 2023 22:50

free up disk space on linux before build

10e3ceb

fix test for expected acero header

17fc588

got removed in 0bb2d83

h-vetinari mentioned this pull request Jul 24, 2023

arrow 13.0.0 conda-forge/arrow-cpp-feedstock#1124

Merged

kou changed the title ~~GH-35658: Sync conda recipes with feedstocks~~ GH-35658: [Packaging] Sync conda recipes with feedstocks Jul 25, 2023

assignUser approved these changes Jul 26, 2023

View reviewed changes

kou approved these changes Jul 26, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Jul 26, 2023

raulcd approved these changes Jul 26, 2023

View reviewed changes

kou merged commit 8503c86 into apache:main Jul 26, 2023

kou removed the awaiting merge Awaiting merge label Jul 26, 2023

h-vetinari deleted the conda_update branch July 26, 2023 23:43

h-vetinari mentioned this pull request Jun 12, 2024

GH-42111: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42114

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-35658: [Packaging] Sync conda recipes with feedstocks #35637

GH-35658: [Packaging] Sync conda recipes with feedstocks #35637

h-vetinari commented May 17, 2023 •

edited by github-actions bot

Loading

github-actions bot commented May 17, 2023

h-vetinari commented May 17, 2023

github-actions bot commented May 17, 2023

h-vetinari commented May 17, 2023

kou commented May 17, 2023

github-actions bot commented May 18, 2023

github-actions bot commented May 18, 2023

h-vetinari commented May 18, 2023

assignUser commented May 18, 2023

kou commented May 18, 2023

h-vetinari commented May 18, 2023

raulcd commented May 18, 2023 •

edited

Loading

kou commented May 19, 2023

raulcd commented May 19, 2023 •

edited

Loading

h-vetinari commented May 24, 2023 •

edited

Loading

jorisvandenbossche commented May 24, 2023

jorisvandenbossche commented May 24, 2023

h-vetinari commented May 24, 2023 •

edited

Loading

h-vetinari commented May 24, 2023

jorisvandenbossche commented May 24, 2023

h-vetinari commented Jul 24, 2023

h-vetinari Jul 24, 2023

github-actions bot commented Jul 24, 2023

h-vetinari commented Jul 24, 2023

github-actions bot commented Jul 24, 2023

h-vetinari commented Jul 25, 2023 •

edited

Loading

assignUser left a comment

kou left a comment

kou Jul 26, 2023

h-vetinari Jul 26, 2023

conbench-apache-arrow bot commented Aug 2, 2023

GH-35658: [Packaging] Sync conda recipes with feedstocks #35637

GH-35658: [Packaging] Sync conda recipes with feedstocks #35637

Conversation

h-vetinari commented May 17, 2023 • edited by github-actions bot Loading

github-actions bot commented May 17, 2023

h-vetinari commented May 17, 2023

github-actions bot commented May 17, 2023

h-vetinari commented May 17, 2023

kou commented May 17, 2023

github-actions bot commented May 18, 2023

github-actions bot commented May 18, 2023

h-vetinari commented May 18, 2023

assignUser commented May 18, 2023

kou commented May 18, 2023

h-vetinari commented May 18, 2023

raulcd commented May 18, 2023 • edited Loading

kou commented May 19, 2023

raulcd commented May 19, 2023 • edited Loading

h-vetinari commented May 24, 2023 • edited Loading

jorisvandenbossche commented May 24, 2023

jorisvandenbossche commented May 24, 2023

h-vetinari commented May 24, 2023 • edited Loading

h-vetinari commented May 24, 2023

jorisvandenbossche commented May 24, 2023

h-vetinari commented Jul 24, 2023

h-vetinari Jul 24, 2023

Choose a reason for hiding this comment

github-actions bot commented Jul 24, 2023

h-vetinari commented Jul 24, 2023

github-actions bot commented Jul 24, 2023

h-vetinari commented Jul 25, 2023 • edited Loading

assignUser left a comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

kou Jul 26, 2023

Choose a reason for hiding this comment

h-vetinari Jul 26, 2023

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Aug 2, 2023

h-vetinari commented May 17, 2023 •

edited by github-actions bot

Loading

raulcd commented May 18, 2023 •

edited

Loading

raulcd commented May 19, 2023 •

edited

Loading

h-vetinari commented May 24, 2023 •

edited

Loading

h-vetinari commented May 24, 2023 •

edited

Loading

h-vetinari commented Jul 25, 2023 •

edited

Loading