Arrow 35901 #2

anjakefala · 2023-10-25T21:46:21Z

This was me trying to prep apache#36266 for merging.

I ran the linter, and added a comparison check: 0f39e8c. It is also a messy PR, bc I also updated the branch by merging arrow:main into it. =(

### Rationale for this change The CI job is failing on Windows after apache#38002. This was masked in review because the Windows job was failing for another reason (which is also now fixed). ### What changes are included in this PR? The error is fixed. ### Are these changes tested? Covered by existing tests ### Are there any user-facing changes? No Authored-by: Dewey Dunnington <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…pe information for categorical columns (apache#38065) ### Rationale for this change See: apache#38034 (comment) ### What changes are included in this PR? The `f_string` for the columns with categorical dtype is now corrected to reflect the type of the indices from the dictionary data type. Bit width has been correct before. From the spec: > For categoricals, the format string describes the type of the categorical in the data buffer. In case of a separate encoding of the categorical (e.g. an integer to string mapping), this can be derived from ``self.describe_categorical``. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38034 Authored-by: AlenkaF <[email protected]> Signed-off-by: AlenkaF <[email protected]>

…pache#38004) ### Rationale for this change It's caused by `backtrace()` and `backtrace_symbols_fd()` signatures are different on Linux and FreeBSD (`int` vs `size_t`). Linux: ```c extern int backtrace (void **__array, int __size) __nonnull ((1)); extern void backtrace_symbols_fd (void *const *__array, int __size, int __fd) __THROW __nonnull ((1)); ``` FreeBSD: ```c size_t backtrace(void **, size_t); int backtrace_symbols_fd(void *const *, size_t, int); ``` ### What changes are included in this PR? Use `auto` to ignore the difference. ### Are these changes tested? Yes on FreeBSD 12. ### Are there any user-facing changes? No. * Closes: apache#37735 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…ache#38162) Bumps [Google.Protobuf](https://github.com/protocolbuffers/protobuf) from 3.24.3 to 3.24.4. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/protocolbuffers/protobuf/commit/7789b3ac85248ad75631a1919071fa268e466210"><code>7789b3a</code></a> Updating version.json and repo version numbers to: 24.4</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/0006601261bdba91668fc454f782ffc8a3ca32e9"><code>0006601</code></a> Update upb dependency (<a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14277">#14277</a>)</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/b9cf19ba5ec4aa6ff108933fa4a1adb075f8c4fb"><code>b9cf19b</code></a> Automated rollback of commit 2a55706f9d36afd1742c97fd0d611ea8af0ac3f8. (<a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14246">#14246</a>)</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/e4baf96abe5a2d854ba1102ce1911eba467ef65a"><code>e4baf96</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14240">#14240</a> from vthib/backport-cython-fix</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/fe87c909399c45d54374817a52af40d6cfd8e063"><code>fe87c90</code></a> fix python code generation compatibility with Cython</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/b2b7a51158418f41cff0520894836c15b1738721"><code>b2b7a51</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14054">#14054</a> from mkruskal-google/iwyu-fix2</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/955d371eacedc37eeff8bf61a1046d694a5fbc8a"><code>955d371</code></a> <code>Printer</code>: Include Abseil's <code>str_cat.h</code> for <code>absl::AlphaNum</code></li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/0d22de520bf3fdb0978a962ae90b72db2f560cef"><code>0d22de5</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14042">#14042</a> from mkruskal-google/iwyu-fix</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/96c1a7d47702f6c82b96e2a0ff112b30ea85c330"><code>96c1a7d</code></a> Add dependency on absl::layout, and explicitly include absl layout includes</li> <li><a href="https://github.com/protocolbuffers/protobuf/commit/d3c44a42c218213d6d5a9fbd66afdd9e8946bfce"><code>d3c44a4</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/13898">#13898</a> from protocolbuffers/24.x-202309071538</li> <li>Additional commits viewable in <a href="https://github.com/protocolbuffers/protobuf/compare/v3.24.3...v3.24.4">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Google.Protobuf&package-manager=nuget&previous-version=3.24.3&new-version=3.24.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@ dependabot rebase` will rebase this PR - `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@ dependabot merge` will merge this PR after your CI passes on it - `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@ dependabot cancel merge` will cancel a previously requested merge and block automerging - `@ dependabot reopen` will reopen this PR if it is closed - `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Sutou Kouhei <[email protected]>

…pache#38075) ### Rationale for this change Prevent a segfault upon passing a non-(chunked_)array object into `Table.filter`. See apache#37650. ### What changes are included in this PR? 1. Check filter Datum kind to make sure that it is an array or a chunked array 2. test that attempting to filter a table with another table raises a not implemented error ### Are these changes tested? In PyArrow, yes ### Are there any user-facing changes? Raises an error if a non-array or non-chunked_array object is passed into `Table.filter` * Closes: apache#37650 Lead-authored-by: Patrick Clarke <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…mpty dataframes (apache#38037) ### Rationale for this change The implementation of the DataFrame Interchange Protocol does not currently support consumption of dataframes with 0 number of chunks (empty dataframes). ### What changes are included in this PR? Add a workaround to not error in this case. ### Are these changes tested? Yes, added `test_empty_dataframe` in `python/pyarrow/tests/interchange/test_conversion.py`. ### Are there any user-facing changes? No. * Closes: apache#37050 Authored-by: AlenkaF <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…arrow.fs.S3FileSystem and s3fs (apache#37768) ### What changes are included in this PR? Update `test_get_file_info_with_selector ` to check that the base directory is not included in the `S3FileSystem` or `s3fs` filesystem. Also remove old comments and flexible checks. * Closes: apache#36983 Lead-authored-by: AlenkaF <[email protected]> Co-authored-by: Joris Van den Bossche <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…thon install docs (apache#38146) ### What changes are included in this PR? The option to set custom path to timezone database through python API has been implemented in apache#35600 and is documented in `docs/source/python/install.rst` with this PR. * Closes: apache#38145 Authored-by: AlenkaF <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…he#38173) ### Rationale for this change It will reduce build time. ### What changes are included in this PR? Install GoogleTest by apt. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38172 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…es` (apache#36846) Revival of apache#12000 ### Rationale for this change It would be great to be able to do promotions when `concat`'ing a table, such as: ```python def test_concat_tables_with_promotion_int(): import pyarrow as pa t1 = pa.Table.from_arrays( [pa.array([1, 2], type=pa.int64())], ["int"]) t2 = pa.Table.from_arrays( [pa.array([3, 4], type=pa.int32())], ["int"]) result = pa.concat_tables([t1, t2], promote=True) assert result.equals(pa.Table.from_arrays([ pa.array([1, 2, 3, 4], type=pa.int64()) ], ["int"])) ``` ### What changes are included in this PR? ### Are these changes tested? ### Are there any user-facing changes? * Closes: apache#36845 Lead-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: David Li <[email protected]> Co-authored-by: Joris Van den Bossche <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…B Interface (apache#38020) ### Rationale for this change To unblock use cases that are not satisfied by the default Arrow -> MATLAB conversions (i.e. the `toMATLAB()` on `arrow.array.Array`), we would like expose the underlying Arrow data representation as a property on `arrow.array.Array`. One possible name for this property would be `DataLayout`, which would be an `arrow.array.DataLayout` object. Note, this class does not yet exist, so we would have to add it. For example, the `DataLayout` property for temporal array types would return an object of the following class type: ```matlab classdef TemporalDataLayout < arrow.array.DataLayout properties Values % an arrow.array.Int32Array or an arrow.array.Int64Array Valid % an arrow.buffer.Buffer end end ``` However, the `Valid` property on this class would need to be an `arrow.buffer.Buffer` object, which does not yet exist in the MATLAB interface. Therefore, it would be helpful to first add the `arrow.buffer.Buffer` class before adding the `DataLayout` property/class hierarchy. It's worth mentioning that adding `arrow.buffer.Buffer` will open up additional advanced use cases in the future. ### What changes are included in this PR? Added `arrow.buffer.Buffer` MATLAB class. *Properties of `arrow.buffer.Buffer`* 1. `NumBytes` - a scalar `int64` value representing the size of the buffer in bytes. *Methods of `arrow.buffer.Buffer`* 1. `toMATLAB` - returns the data in the buffer as `Nx1` `uint8` vector, where `N` is the number of bytes. 2. `fromMATLAB(data)` - Static method that creates an `arrow.buffer.Buffer` from a numeric array. **Example:** ```matlab >> dataIn = [1 2]; >> buffer = arrow.buffer.Buffer.fromMATLAB(dataIn) buffer = Buffer with properties: NumBytes: 16 >> dataOut = toMATLAB(buffer) dataOut = 16×1 uint8 column vector 0 0 0 0 0 0 240 63 0 0 0 0 0 0 0 64 % Reinterpret bit pattern as a double array >> toDouble = typecast(dataOut, "double") toDouble = 1 2 ``` ### Are these changes tested? Yes. Added a new test class called `tBuffer.m` ### Are there any user-facing changes? Yes. Users can now create `arrow.buffer.Buffer` objects via the `fromMATLAB` static method. However, there's not much users can do with this object as of now. We implemented this class to facilitate adding `DataLayout` property to `arrow.array.Array`, as described in the **Rational for this change** section. * Closes: apache#38015 Authored-by: Sarah Gilmore <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>

Fix typo: represenation -> representation. Authored-by: Alexander Grueneberg <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…cts (apache#37376) ### Rationale for this change See: apache#32439 ### What changes are included in this PR? During conversion from Python to Arrow, when a struct's child hits a capacity error and chunking is triggered, this can leave the Finish'd chunk in an invalid state since the struct's length does not match the length of its children. This change simply tries to Append the children first, and only if successful will Append the struct. This is safe because the order of Append'ing between the struct and its child is not specified. It is only specified that they must be consistent with each other. This is per: https://github.com/apache/arrow/blob/86b7a84c9317fa08222eb63f6930bbb54c2e6d0b/cpp/src/arrow/array/builder_nested.h#L507-L508 ### Are these changes tested? A unit test is added that would previously have an invalid data error. ``` > tab = pa.Table.from_pandas(df) pyarrow/tests/test_pandas.py:4970: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:3788: in pyarrow.lib.Table.from_pandas return cls.from_arrays(arrays, schema=schema) pyarrow/table.pxi:3890: in pyarrow.lib.Table.from_arrays result.validate() pyarrow/table.pxi:3170: in pyarrow.lib.Table.validate check_status(self.table.Validate()) # ... FAILED pyarrow/tests/test_pandas.py::test_nested_chunking_valid - pyarrow.lib.ArrowInvalid: Column 0: In chunk 0: Invalid: List child array invalid: Invalid: Struct child array #0 has length smaller than expected for struct array (2 < 3) ``` NOTE: This unit test uses about 7GB of memory (max RSS) on my macbook pro. This might make CI challenging; I'm open to suggestions to limit it. ### Are there any user-facing changes? No * Closes: apache#32439 Lead-authored-by: Mike Lui <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…hon scalar cast) (apache#38038) ### Rationale for this change Adding `string -> date32/date64` cast kernels, which then also fixes the pyarrow scalar cast method (which was earlier refactored to rely on the general cast kernels) * Closes: apache#37411 Authored-by: Joris Van den Bossche <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…en both data page and version is V2 (apache#38163) ### Rationale for this change Only use RLE as BOOLEAN default encoding when data page is V2. Previous patch ( apache#36955 ) set RLE encoding for Boolean type by default. However, parquet-cpp might write format v2 file with page v1 by default. This might cause parquet-cpp generating RLE encoding for boolean type by default. As https://issues.apache.org/jira/browse/PARQUET-2222 says, we still need some talks about that. So, we: 1. Still allow writing RLE on DataPage V2. This keeps same as parquet rust 2. If DataPage V1 is used, don't use RLE as default Boolean encoding. ### What changes are included in this PR? Only use RLE as BOOLEAN default encoding when both data page and version is V2. ### Are these changes tested? Yes ### Are there any user-facing changes? RLE encoding change for Boolean. * Closes: apache#36882 Lead-authored-by: mwish <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…E_LIST_VIEW array formats (apache#37468) ### Rationale for this change Go implementation of apache#35345. ### What changes are included in this PR? - [x] Add `LIST_VIEW` and `LARGE_LIST_VIEW` to datatype.go - [x] Add `ListView` and `LargeListView` to list.go - [x] Add `ListViewType` and `LargeListViewType` to datatype_nested.go - [x] Add list-view builders - [x] Implement list-view comparison in compare.go - [x] String conversion in both directions - [x] Validation of list-view arrays - [x] Generation of random list-view arrays - [x] Concatenation of list-view arrays in concat.go - [x] JSON serialization/deserialization - [x] Add data used for tests in `arrdata.go` - [x] Add Flatbuffer changes - [x] Add IPC support ### Are these changes tested? Yes. Existing tests are being changed to also cover list-view variations as well as new tests focused solely on the list-view format. ### Are there any user-facing changes? New structs and functions introduced. * Closes: apache#35344 Authored-by: Felipe Oliveira Carvalho <[email protected]> Signed-off-by: Matt Topol <[email protected]>

…che#38175) ### Rationale for this change It has g++ 13 support. ### What changes are included in this PR? Update version and checksum. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: apache#38174 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change We run "git describe --tag --dirty" implicitly in cpp/cmake_modules/DefineOptions.cmake. If we use "--dirty", .git/index's owner may be changed. Because "git describe" touches .git/index for "--dirty". We can avoid changing .git/index's owner by not using "--dirty". ### What changes are included in this PR? Remove "--dirty". ### Are these changes tested? Yes. I used "strace git describe ...". ### Are there any user-facing changes? No. * Closes: apache#37767 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Jacob Wujciak-Jens <[email protected]>

### Rationale for this change Add Jacob Wujciak-Jens as one of the R package authors ### What changes are included in this PR? Add Jacob Wujciak-Jens as one of the R package authors ### Are these changes tested? No ### Are there any user-facing changes? No Lead-authored-by: Nic Crane <[email protected]> Co-authored-by: Jacob Wujciak-Jens <[email protected]> Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…4 Java JNI" (apache#38194) ### Rationale for this change We don't have enough disk space. ### What changes are included in this PR? Remove unused pre-installed software. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38193 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…on test mode (apache#38177) ### Rationale for this change Unit tests for each implementations are done on unit test mode. ### What changes are included in this PR? Detect integration test only mode. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38159 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…mpiled binaries (apache#38115) ### Rationale for this change This change is to restore parity with the previous solution on macOS (brew does cs validation) and improve security for windows and linux. This also align with CRAN policy. ### What changes are included in this PR? This PR adds a script that can be run after the arrow release (once all files have been pushed to the artifactory) before the CRAN submission to download the checksum files for the pre-compiled binaries which are already added through the usual release. *libs.R have been extended to use these checksum files to validate the downloaded binaries. ### Are these changes tested? The r-binary-packages nightlies generate checksums and use them when building binary packages, this way the code path is tested. They do not modify the actual src package though. ### Are there any user-facing changes? no (outside of log messages) * Closes: apache#37941 Authored-by: Jacob Wujciak-Jens <[email protected]> Signed-off-by: Nic Crane <[email protected]>

…tion in the new Dataset API (apache#34616) ### Rationale for this change The purpose of this pull request is to support modular encryption in the new Dataset API. See [https://docs.google.com/document/d/13EysCNC6-Nu9wnJ8YpdzmD-aMLn4i2KXUJTNqIihy7A/edit#](url) for supporting document. ### What changes are included in this PR? I made improvements to the C++ and Python code to enable the Dataset API to have per-file settings for each file saved. Previously, the Dataset API applied the same encryption properties to all saved files, but now I've updated the code to allow for greater flexibility. In the Python code, I've added support for the changes by updating the ParquetFormat class to accept DatasetEncryptionConfiguration and DatasetDecryptionConfiguration structures. With these changes, you can pass the format object to the write_dataset function, giving you the ability to set unique encryption properties for each file in your Dataset. ### Are these changes tested? Yes, unit tests are included. I have also included a python sample project. ### Are there any user-facing changes? Yes, as stated above the ParquetFormat class has optional parameters for DatasetEncryptionConfiguration and DatasetDecryptionConfiguration through setters and getters. The Dataset now has the option using this to set different file encryption properties per file * Closes: apache#29238 Lead-authored-by: Don <[email protected]> Co-authored-by: Donald Tolley <[email protected]> Co-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: anjakefala <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Weston Pace <[email protected]> Co-authored-by: Gang Wu <[email protected]> Co-authored-by: scoder <[email protected]> Co-authored-by: Will Jones <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…#38154) ### Rationale for this change New C# drivers need the ability to concatenate array data, particularly for metadata calls. ### What changes are included in this PR? Converts a previously internal class and method to a public class and method. ### Are these changes tested? Yes ### Are there any user-facing changes? It exposes previously hidden functionality. Resolves apache#38153 * Closes: apache#38153 Lead-authored-by: David Coe <[email protected]> Co-authored-by: davidhcoe <[email protected]> Signed-off-by: David Li <[email protected]>

…ay` and `arrow.array.ChunkedArray` to `NumElements` (apache#38190) ### Rationale for this change We would like to rename the `Length` property on `arrow.array.Array` (and `arrow.array.ChunkedArray`) to `NumElements` because MATLAB has a function called `length(A)`, which returns the length of the largest array dimension in `A`. Because `arrow.array.Array` inherits from `matlab.mixin.Scalar`, it's size is always `1x1`, so the `length()` function always returns `1`. This may confuse users because the `Length` property of `arrow.array.Array` returns the number of elements within the array, which may not be `1`. ```matlab >> array = arrow.array([1 2 3 4 5]); % The length property returns the number of elements in the array >> array.Length ans = int64 5 % The length method returns the length of the largest dimension >> length(array) ans = 1 ``` I suspect this inconsistency will confuse users, so we should rename the `Length` property to `NumElements`. ### What changes are included in this PR? 1. Renamed `Length` property on `arrow.array.Array` to `NumElements` 2. Renamed `Length` property on `arrow.array.ChunkedArray` to `NumElements` ### Are these changes tested? Yes. I modified the existing `Length` test cases to now test the `NumElements` property. ### Are there any user-facing changes? Yes. Please note this is a breaking change because `Length` is no longer a property on either `arrow.array.Array` or `arrow.array.ChunkedArray`. However, we have not yet cut an initial release of the MATLAB interface, so we don't expect this change should affect too many users at this point. * Closes: apache#38164 Authored-by: Sarah Gilmore <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>

…mensions, implemented using ExtensionType (apache#37166) ### Rationale for this change For use cases where underlying datatype and number of dimensions in tensors are equal but not the actual shape we want to add a `VariableShapeTensorType`. See apache#24868 and huggingface/datasets#5272 ### What changes are included in this PR? This introduces definition of `arrow.variable_shape_tensor` extension and it's C++ implementation and a Python wrapper. ### Are these changes tested? Yes. ### Are there any user-facing changes? This introduces new extension type to the user. * Closes: apache#24868 Lead-authored-by: Rok Mihevc <[email protected]> Co-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…pache#37901) ### Rationale for this change Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release. ### What changes are included in this PR? This PR adds jobs to build pyarrow wheels for Python 3.12. ### Are these changes tested? They will be tested via archery tasks ### Are there any user-facing changes? No but users will be able to use pyarrow with Python 3.12 * Closes: apache#37880 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>

… proxies (apache#38199) ### Rationale for this change Fixes a bug in the Go Flight client library that makes using AuthenticateBasicToken unreliable behind proxies. ### What changes are included in this PR? Closes the sending side of the bi-directional stream for the Flight Handshake RPC call before trying to read the headers. This matches the C++ implementation. <img width="609" alt="Screenshot 2023-10-11 at 6 18 40 PM" src="https://github.com/apache/arrow/assets/879445/05e23c6a-0ff8-41fc-825b-8add7fe938bc"> ### Are these changes tested? I've tested these changes against my service deployed behind CloudFlare and verified the error listed in the linked issue disappears. ### Are there any user-facing changes? No * Closes: apache#38198 Authored-by: Phillip LeBlanc <[email protected]> Signed-off-by: Matt Topol <[email protected]>

…conan (apache#38202) ### Rationale for this change There is a conflict between the required Zlib version when using both thrift and GRPC. ### What changes are included in this PR? Pinning zlib when using thrifht. ### Are these changes tested? Via archery ### Are there any user-facing changes? No * Closes: apache#38201 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…he#38189) ### Rationale for this change In support of adding an [`arrow.array.ListArray`](apache#37815) MATLAB class, this pull request adds a new `arrow.type.ListType` MATLAB class. ### What changes are included in this PR? 1. New `arrow.list(<type>)` MATLAB construction function. 2. New `arrow.list.ListType` MATLAB class. `ListType` has a property named `Type` which indicates the inner type of the `List`. `Type` can be set to any subclass `arrow.type.Type` (including `arrow.type.ListType`, to support nested lists). 3. New `arrow.type.ID.List` type ID enumeration value. 4. New `arrow.type.traits.ListTraits` type traits class. Some of the properties, such as `ArrayConstructor` and `ArrayProxyClassName`, are set to `missing` because they are dependent on adding [`arrow.array.ListArray`](apache#37815) first. **Example** ```matlab % Create a simple List<String> type. >> stringListType = arrow.list(arrow.string()) stringListType = ListType with properties: ID: List Type: [1x1 arrow.type.StringType] % Create a nested List<List<Boolean>> type. >> nestedListType = arrow.list(arrow.list(arrow.boolean())) nestedListType = ListType with properties: ID: List Type: [1x1 arrow.type.ListType] % Extract the first-level, inner type, which is List<Boolean>. >> innerType = nestedListType.Type innerType = ListType with properties: ID: List Type: [1x1 arrow.type.BooleanType] % Extract the second-level, nested inner type, which is Boolean. >> innerType.Type ans = BooleanType with properties: ID: Boolean ``` ### Are these changes tested? Yes. 1. Added `tListType.m`. 2. Added `tListTraits.m`. 3. Updated `tField.m` to include `arrow.list`. 4. Updated `tID.m` to include `arrow.type.ID.List`. 6. Updated `tTypeDisplay.m` to include `arrow.type.ListType`. 7. Updated `ttraits.m` to include `arrow.type.traits.ListTraits`. ### Are there any user-facing changes? Yes. Client MATLAB code can now creates instances of `arrow.type.ListType` by using the `arrow.list(<type>)` construction function. ### Future Directions 1. apache#37815 * Closes: apache#37812 Authored-by: Kevin Gurney <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>

…ache#38685) Bumps [Grpc.AspNetCore](https://github.com/grpc/grpc-dotnet) from 2.58.0 to 2.59.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/grpc/grpc-dotnet/releases">Grpc.AspNetCore's releases</a>.</em></p> <blockquote> <h2>Release v2.59.0</h2> <h2>What's Changed</h2> <ul> <li>Fix HttpContext race condition by copying values to reader and writer by <a href="https://github.com/JamesNK"><code>@JamesNK</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2294">grpc/grpc-dotnet#2294</a></li> <li>Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2298">grpc/grpc-dotnet#2298</a></li> <li>Update grpc tools to 2.59 by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2303">grpc/grpc-dotnet#2303</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0">https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0</a></p> <h2>Release v2.59.0-pre1</h2> <h2>What's Changed</h2> <ul> <li>Fix HttpContext race condition by copying values to reader and writer by <a href="https://github.com/JamesNK"><code>@JamesNK</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2294">grpc/grpc-dotnet#2294</a></li> <li>Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2298">grpc/grpc-dotnet#2298</a></li> <li>Update grpc tools to 2.59 by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2303">grpc/grpc-dotnet#2303</a></li> <li>Update to version 2.59.0-pre1 on v2.59.x branch by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2307">grpc/grpc-dotnet#2307</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0-pre1">https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0-pre1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/grpc/grpc-dotnet/commit/9124a4cb8c210a27bab47cce3da694d0d0b049ea"><code>9124a4c</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2316">#2316</a> from apolcyn/bump_r</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/eb9f932d52597116625fc127e1ae463a0b5faa62"><code>eb9f932</code></a> bump to 2.59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/20a6e90ab43f47452446c49bab74c8839047b8d2"><code>20a6e90</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2307">#2307</a> from apolcyn/release_updates</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/2084009f8f375514dcad6b8b1d4571d266e8e6c8"><code>2084009</code></a> bump version to 2.59.0-pre1</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/e52049a94d69bf5105d07193b116800ddd62eae3"><code>e52049a</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2303">#2303</a> from apolcyn/update_59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/51ad5ab7573ce61c1eecf620dc26bd3687d85e07"><code>51ad5ab</code></a> update grpc tools to 2.59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/3c4abbada1a9794398c40e88b5e9de6c6f4641eb"><code>3c4abba</code></a> Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp...</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/91be3925c19ab297dda52ff0c201ee025eda0d6e"><code>91be392</code></a> Fix HttpContext race by copying values to reader and writer (<a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2294">#2294</a>)</li> <li>See full diff in <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Grpc.AspNetCore&package-manager=nuget&previous-version=2.58.0&new-version=2.59.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@ dependabot rebase` will rebase this PR - `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@ dependabot merge` will merge this PR after your CI passes on it - `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@ dependabot cancel merge` will cancel a previously requested merge and block automerging - `@ dependabot reopen` will reopen this PR if it is closed - `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Sutou Kouhei <[email protected]>

…ache#38686) Bumps [Grpc.Net.Client](https://github.com/grpc/grpc-dotnet) from 2.58.0 to 2.59.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/grpc/grpc-dotnet/releases">Grpc.Net.Client's releases</a>.</em></p> <blockquote> <h2>Release v2.59.0</h2> <h2>What's Changed</h2> <ul> <li>Fix HttpContext race condition by copying values to reader and writer by <a href="https://github.com/JamesNK"><code>@JamesNK</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2294">grpc/grpc-dotnet#2294</a></li> <li>Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2298">grpc/grpc-dotnet#2298</a></li> <li>Update grpc tools to 2.59 by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2303">grpc/grpc-dotnet#2303</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0">https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0</a></p> <h2>Release v2.59.0-pre1</h2> <h2>What's Changed</h2> <ul> <li>Fix HttpContext race condition by copying values to reader and writer by <a href="https://github.com/JamesNK"><code>@JamesNK</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2294">grpc/grpc-dotnet#2294</a></li> <li>Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2298">grpc/grpc-dotnet#2298</a></li> <li>Update grpc tools to 2.59 by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2303">grpc/grpc-dotnet#2303</a></li> <li>Update to version 2.59.0-pre1 on v2.59.x branch by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2307">grpc/grpc-dotnet#2307</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0-pre1">https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0-pre1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/grpc/grpc-dotnet/commit/9124a4cb8c210a27bab47cce3da694d0d0b049ea"><code>9124a4c</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2316">#2316</a> from apolcyn/bump_r</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/eb9f932d52597116625fc127e1ae463a0b5faa62"><code>eb9f932</code></a> bump to 2.59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/20a6e90ab43f47452446c49bab74c8839047b8d2"><code>20a6e90</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2307">#2307</a> from apolcyn/release_updates</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/2084009f8f375514dcad6b8b1d4571d266e8e6c8"><code>2084009</code></a> bump version to 2.59.0-pre1</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/e52049a94d69bf5105d07193b116800ddd62eae3"><code>e52049a</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2303">#2303</a> from apolcyn/update_59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/51ad5ab7573ce61c1eecf620dc26bd3687d85e07"><code>51ad5ab</code></a> update grpc tools to 2.59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/3c4abbada1a9794398c40e88b5e9de6c6f4641eb"><code>3c4abba</code></a> Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp...</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/91be3925c19ab297dda52ff0c201ee025eda0d6e"><code>91be392</code></a> Fix HttpContext race by copying values to reader and writer (<a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2294">#2294</a>)</li> <li>See full diff in <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Grpc.Net.Client&package-manager=nuget&previous-version=2.58.0&new-version=2.59.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@ dependabot rebase` will rebase this PR - `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@ dependabot merge` will merge this PR after your CI passes on it - `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@ dependabot cancel merge` will cancel a previously requested merge and block automerging - `@ dependabot reopen` will reopen this PR if it is closed - `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Sutou Kouhei <[email protected]>

…arp (apache#38688) Bumps [Grpc.AspNetCore.Server](https://github.com/grpc/grpc-dotnet) from 2.58.0 to 2.59.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/grpc/grpc-dotnet/releases">Grpc.AspNetCore.Server's releases</a>.</em></p> <blockquote> <h2>Release v2.59.0</h2> <h2>What's Changed</h2> <ul> <li>Fix HttpContext race condition by copying values to reader and writer by <a href="https://github.com/JamesNK"><code>@JamesNK</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2294">grpc/grpc-dotnet#2294</a></li> <li>Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2298">grpc/grpc-dotnet#2298</a></li> <li>Update grpc tools to 2.59 by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2303">grpc/grpc-dotnet#2303</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0">https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0</a></p> <h2>Release v2.59.0-pre1</h2> <h2>What's Changed</h2> <ul> <li>Fix HttpContext race condition by copying values to reader and writer by <a href="https://github.com/JamesNK"><code>@JamesNK</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2294">grpc/grpc-dotnet#2294</a></li> <li>Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2298">grpc/grpc-dotnet#2298</a></li> <li>Update grpc tools to 2.59 by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2303">grpc/grpc-dotnet#2303</a></li> <li>Update to version 2.59.0-pre1 on v2.59.x branch by <a href="https://github.com/apolcyn"><code>@apolcyn</code></a> in <a href="https://redirect.github.com/grpc/grpc-dotnet/pull/2307">grpc/grpc-dotnet#2307</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0-pre1">https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0-pre1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/grpc/grpc-dotnet/commit/9124a4cb8c210a27bab47cce3da694d0d0b049ea"><code>9124a4c</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2316">#2316</a> from apolcyn/bump_r</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/eb9f932d52597116625fc127e1ae463a0b5faa62"><code>eb9f932</code></a> bump to 2.59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/20a6e90ab43f47452446c49bab74c8839047b8d2"><code>20a6e90</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2307">#2307</a> from apolcyn/release_updates</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/2084009f8f375514dcad6b8b1d4571d266e8e6c8"><code>2084009</code></a> bump version to 2.59.0-pre1</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/e52049a94d69bf5105d07193b116800ddd62eae3"><code>e52049a</code></a> Merge pull request <a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2303">#2303</a> from apolcyn/update_59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/51ad5ab7573ce61c1eecf620dc26bd3687d85e07"><code>51ad5ab</code></a> update grpc tools to 2.59</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/3c4abbada1a9794398c40e88b5e9de6c6f4641eb"><code>3c4abba</code></a> Bump <code>@babel/traverse</code> from 7.17.3 to 7.23.2 in /examples/Spar/Server/ClientApp...</li> <li><a href="https://github.com/grpc/grpc-dotnet/commit/91be3925c19ab297dda52ff0c201ee025eda0d6e"><code>91be392</code></a> Fix HttpContext race by copying values to reader and writer (<a href="https://redirect.github.com/grpc/grpc-dotnet/issues/2294">#2294</a>)</li> <li>See full diff in <a href="https://github.com/grpc/grpc-dotnet/compare/v2.58.0...v2.59.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Grpc.AspNetCore.Server&package-manager=nuget&previous-version=2.58.0&new-version=2.59.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@ dependabot rebase` will rebase this PR - `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@ dependabot merge` will merge this PR after your CI passes on it - `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@ dependabot cancel merge` will cancel a previously requested merge and block automerging - `@ dependabot reopen` will reopen this PR if it is closed - `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Sutou Kouhei <[email protected]>

…OS version to 10.15 catalina to allow using new APIs in C++17 (apache#38677) ### Rationale for this change For macOS, some new APIs in C++17, e.g. `std::filesystem::path` is only visible when the compiling target is >= macOS 10.15 (Catalina). But currently the minimum macOS target is set to 10.14 (Mojave). * macOS 10.14 Mojave was released on 2018-09-24 [1] * macOS 10.15 Catalina was released on 2019-10-07 [2] * Both macOS 10.14 and 10.15 are end of life [3] There is a [similar issue](ClickHouse/ClickHouse#8541) [4] and [its fix](ClickHouse/ClickHouse#8600) [5] for ClickHouse on year 2020, which raised minimum macOS version from 10.14 to 10.15. ### What changes are included in this PR? Raise the minimum macOS version from 10.14 to 10.15 ### Are these changes tested? This should be tested and verified on CI. ### Are there any user-facing changes? * Users using macOS <= 10.14 may not able to run these CI jobs locally. * The new version of python wheels may not be able to be deployed to environments with macOS <= 10.14 (not completely sure how this affects the deployment) * Closes: apache#38653 ### References * [1] https://en.wikipedia.org/wiki/MacOS_Mojave * [2] https://en.wikipedia.org/wiki/MacOS_Catalina * [3] https://endoflife.date/macos * [4] ClickHouse/ClickHouse#8541 * [5] ClickHouse/ClickHouse#8600 Authored-by: Yue Ni <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change These are just some minor style tweaks which made the text and buttons on these cards read more naturally to me. Feel free to ignore or take only some of the changes here. ### Are these changes tested? Yes, I confirmed the docs still build as expected locally. ### Are there any user-facing changes? New language in docs, see above. Authored-by: Bryce Mecum <[email protected]> Signed-off-by: AlenkaF <[email protected]>

### Rationale for this change Add missing Default VectorValueComparators for some more types. ### What changes are included in this PR? Add comparators for: - FixedSizeBinaryVector - LargeListVector - FixedSizeListVector - NullVector ### Are these changes tested? Yes, unit tests added. ### Are there any user-facing changes? No * Closes: apache#38662 Authored-by: James Duong <[email protected]> Signed-off-by: David Li <[email protected]>

…tdown (apache#38637) ### Rationale for this change Some C++ destructors may be called after the Python interpreter has ceased to exist. If such a destructor tries to call back in the Python interpreter, for example by calling `Py_DECREF`, we get a crash. ### What changes are included in this PR? Protect `OwnedRef` and `OwneRefNoGIL` destructors against decref'ing a Python object after Python finalization. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38626 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

### Rationale for this change Fixing an off-by-one rounding issue with decimal128 by ensuring proper precision handling. ### Are these changes tested? The test case which reproduced the rounding issue has been added as a unit test. * Closes: apache#38477 Authored-by: Matt Topol <[email protected]> Signed-off-by: Matt Topol <[email protected]>

…arning (apache#38710) We renamed the new keyword in a final iteration of the PR, but apparently forgot to update the warning message. Authored-by: Joris Van den Bossche <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…o Go implementation (apache#35769) ### Rationale for this change See apache#35628 for the rationale and description of the StringView/BinaryView array types. This change is adding Go as a second implementation of it. ### What changes are included in this PR? Add Array Types for `StringView` and `BinaryView` along with `StringViewType` and `BinaryViewType` and necessary enums and builders. These arrays can be round tripped through JSON and IPC. ### Are these changes tested? Yes, unit tests have been added and integration tests run * Closes: [apache#38718](apache#38718) * Closes: apache#38718 Lead-authored-by: Matt Topol <[email protected]> Co-authored-by: Alex Shcherbakov <[email protected]> Signed-off-by: Benjamin Kietzman <[email protected]>

…38716) ### Rationale for this change The CRAN incoming check for 14.0.0 is failing with a NOTE about a possible bashism ### What changes are included in this PR? One `test -a` usage was replaced with `&&`. ### Are these changes tested? Yes (via crossbow, below) ### Are there any user-facing changes? No * Closes: apache#38715 Authored-by: Dewey Dunnington <[email protected]> Signed-off-by: Dewey Dunnington <[email protected]>

…he#38723) ### Rationale for this change Restored the Runner image to not delete `aws-cli` for the execution of the `preview-docs` command. ### Are these changes tested? No ### Are there any user-facing changes? No * Closes: apache#38711 Authored-by: Hyunseok Seo <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…he#38714) ### Rationale for this change It seems the object case in `_reconstruct_block` is a dead code and is not needed anymore so therefore could be removed. ### What changes are included in this PR? Removal of the object case in `_reconstruct_block` code. Was also looking at the `arrow_to_pandas.cc` code to see if there is any dead code present and I couldn't find any. ### Are these changes tested? The change in this PR should not make any of the existing tests fail. ### Are there any user-facing changes? There shouldn't be. * Closes: apache#38712 Authored-by: AlenkaF <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

### Rationale for this change Noticed wrong section headings on the web doc and proposing a fix. ### Are these changes tested? Yes. Built and verified the documentation locally. <img width="1049" alt="image" src="https://github.com/apache/arrow/assets/6668548/793d6222-2bc5-4f55-ab67-5abe5283add4"> ### Are there any user-facing changes? No. * Closes: apache#38599 Authored-by: Hyunseok Seo <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…apache#36073) ### Rationale for this change There is currently an active proposal to support half-float types in Parquet. For more details/discussion, see the links in this PR's accompanying issue. ### What changes are included in this PR? This PR implements basic support for a `Float16LogicalType` in accordance with the proposed spec. More specifically, this includes: - Changes to `parquet.thrift` and regenerated `parqet_types` files - Basic `LogicalType` class definition, method impls, and enums - Support for specialized comparisons and column statistics In the interest of scope, this PR does not currently deal with arrow integration and byte split encoding - although we will want both of these features resolved before the proposal is approved. ### Are these changes tested? Yes (tests are included) ### Are there any user-facing changes? Yes * Closes: apache#36036 Lead-authored-by: benibus <[email protected]> Co-authored-by: Ben Harkins <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…rs out (apache#38713) ### Rationale for this change A deadlock can happen in a C++ destructor in the following case: * the C++ destructor is called from Python, holding the GIL * the C++ destructor waits for a threaded task to finish * the threaded task has invoked some Python code which is waiting to acquire the GIL ### What changes are included in this PR? To reliably present such a deadlock, introduce `std::shared_ptr` and `std::unique_ptr` wrappers that release the GIL when deallocating the embedded pointer. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38676 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…Writer (apache#38581) ### Rationale for this change Currently, `ArrowColumnWriter` seems not having bug. But the usage is confusing. For nested type, `ArrowColumnWriter` should considering the logic below: ``` /// 0 foo.bar /// foo.bar.baz 0 /// foo.bar.baz2 1 /// foo.qux 2 /// 1 foo2 3 /// 2 foo3 4 ``` The left column is the column in root of `arrow::Schema`, the parquet itself only stores Leaf node, so, the column id for parquet is list at right. In the `ArrowColumnWriter`, the final argument is the LeafIdx in parquet, so, writer should considering using `leafIdx`. Also, it need a `LeafCount` API for getting the leaf-count here. ### What changes are included in this PR? Style enhancement for `LeafCount`, `leafIdx` and usage for `ArrowColumnWriter` ### Are these changes tested? no ### Are there any user-facing changes? no * Closes: apache#38503 Authored-by: mwish <[email protected]> Signed-off-by: Matt Topol <[email protected]>

…system::path in gandiva (apache#38698) ### Rationale for this change AlmaLinux 8 CI reported linker failure when `std::filesystem::path` is used, and This PR tries to it. ### What changes are included in this PR? Replace replace `std::filesystem::path` in gandiva with arrow's internal io util so that AlmaLinux 8 CI build can work. ### Are these changes tested? It should be covered by existing tests and CI. ### Are there any user-facing changes? No * Closes: apache#38697 Lead-authored-by: Yue Ni <[email protected]> Co-authored-by: Yue <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…pache#38727) This makes it so the Arrow column writer is not exported from the `pqarrow` package. This follows up on comments from apache#38581. * Closes: apache#38503 Authored-by: Tim Schaub <[email protected]> Signed-off-by: Matt Topol <[email protected]>

) ### Rationale for this change This enables tests that are currently disabled to improve coverage and help others build tests based on these. ### What changes are included in this PR? - Enable tests that were disabled due to flakey memory leaks - Explicitly close child allocators in these tests to match the behavior of FlightServerTestRule which does not leak. - Change TestBasicAuth to allocate only one server - Change TestBasicAuth2 to allocate only one server and client - Fix a bug in testBasucAuth2#asyncPut() not including credentials ### Are these changes tested? Tested locally. ### Are there any user-facing changes? No. * Closes: apache#38318 Authored-by: James Duong <[email protected]> Signed-off-by: David Li <[email protected]>

…s. (apache#38642) ### Rationale for this change some other interfaces lack equal options, suck as nan equal, so we add TestingEqualOptions for each function and add some testing. ### What changes are included in this PR? gtest_util related. ### Are these changes tested? gtest_util.cc ### Are there any user-facing changes? yes. * Closes: apache#38624 Authored-by: light-city <[email protected]> Signed-off-by: Benjamin Kietzman <[email protected]>

### Rationale for this change There is, at least, partial halffloat support in Arrow C++. A reason to not merge this is that there is still quite a bit casting work to be completed: * apache#20213 * apache#32802 Authored-by: anjakefala <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…tFileWriter (apache#38390) ### Rationale for this change Operations on closed ParquetFileWriter are not allowed, but should not segfault. Somehow, ParquetFileWriter::Close() also reset its pimpl, so after that, any operators, those need this pointer will lead to segfault ### What changes are included in this PR? Adding more checks for closed file. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#37969 Authored-by: Quang Hoang <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

### Rationale for this change The cache of SqlInfo properties that ArrowDatabaseMetaData maintains isn't populated in a thread-safe way. This can cause JDBC applications trying to retrieve several properties from DatabaseMetaData to encounter missing properties when they shouldn't. ### What changes are included in this PR? - Changed the checking for the cache being populated to be based on an AtomicBoolean marking that the cache is fully populated, rather than just checking if the cache is empty. - Avoid having multiple threads call getSqlInfo() if they see that the cache is empty concurrently. ### Are these changes tested? Verified existing unit tests. ### Are there any user-facing changes? No. * Closes: apache#38737 Authored-by: James Duong <[email protected]> Signed-off-by: David Li <[email protected]>

### Rationale for this change For some reason, the decimal files generated by the integration test suite have as many record batches as columns. This looks like a programming error rather than something deliberate. ### What changes are included in this PR? Use similar batch sizes as for other test files. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38745 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…when converting MATLAB duration to `arrow.array.Time32Array` and `arrow.array.Time64Array` (apache#38661) ### Rationale for this change The current conversion from MATLAB duration to `arrow.array.Time32Array` and `arrow.array.Time64Array` loses time precision, and there is no test to cover such limitation. It is best practice to have tests cover software design. In addition, such tests will be helpful to evaluate the impact in the future when we improve the design. ### What changes are included in this PR? I mainly added three test cases for each of `arrow.array.Time32Array` and `arrow.array.Time64Array`. - Updated the basic test case to verify both class and value. In the MATLAB interface tests, we would like to verify the value to make sure there is no precision loss. The basic test case will serve as a test example when people learn to write tests. Updating the basic test case will set a good example for contributors to learn. - Test the default value of "TimeUnit". - Test the functionality of "TimeUnit". ### Are these changes tested? No software change. The updated test files passed on my local machine. ### Are there any user-facing changes? No * Closes: apache#38652 Authored-by: Lei Hou <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>

paleolimbot and others added 30 commits October 10, 2023 11:26

MINOR: [Python] Fix typo: represenation -> representation (apache#38180)

9077236

Fix typo: represenation -> representation. Authored-by: Alexander Grueneberg <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

dependabot bot and others added 28 commits November 14, 2023 08:20

Merge branch 'main' into arrow-35901

1cc299f

github-actions bot added the awaiting review label Nov 16, 2023

anjakefala merged commit 1cc299f into vibhatha:arrow-35901 Nov 17, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arrow 35901 #2

Arrow 35901 #2

anjakefala commented Oct 25, 2023 •

edited

Loading

Arrow 35901 #2

Arrow 35901 #2

Conversation

anjakefala commented Oct 25, 2023 • edited Loading

anjakefala commented Oct 25, 2023 •

edited

Loading