### Rationale for this change

The CI job is failing on Windows after apache#38002. This was masked in review because the Windows job was failing for another reason (which is also now fixed).

### What changes are included in this PR?

The error is fixed.

### Are these changes tested?

Covered by existing tests

### Are there any user-facing changes?

No

Authored-by: Dewey Dunnington <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…pe information for categorical columns (apache#38065)

### Rationale for this change
See: apache#38034 (comment)

### What changes are included in this PR?

The `f_string` for the columns with categorical dtype is now corrected to reflect the type of the indices from the dictionary data type. Bit width has been correct before. From the spec:

> For categoricals, the format string describes the type of the
              categorical in the data buffer. In case of a separate encoding of
              the categorical (e.g. an integer to string mapping), this can
              be derived from ``self.describe_categorical``.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38034

Authored-by: AlenkaF <[email protected]>
Signed-off-by: AlenkaF <[email protected]>

…pache#38004)

### Rationale for this change

It's caused by `backtrace()` and `backtrace_symbols_fd()` signatures are different on Linux and FreeBSD (`int` vs `size_t`).

Linux:

```c
extern int backtrace (void **__array, int __size) __nonnull ((1));
extern void backtrace_symbols_fd (void *const *__array, int __size, int __fd)
     __THROW __nonnull ((1));
```

FreeBSD:

```c
size_t backtrace(void **, size_t);
int backtrace_symbols_fd(void *const *, size_t, int);
```

### What changes are included in this PR?

Use `auto` to ignore the difference.

### Are these changes tested?

Yes on FreeBSD 12.

### Are there any user-facing changes?

No.
* Closes: apache#37735

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…ache#38162)

Bumps [Google.Protobuf](https://github.com/protocolbuffers/protobuf) from 3.24.3 to 3.24.4.
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/7789b3ac85248ad75631a1919071fa268e466210"><code>7789b3a</code></a> Updating version.json and repo version numbers to: 24.4</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/0006601261bdba91668fc454f782ffc8a3ca32e9"><code>0006601</code></a> Update upb dependency (<a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14277">#14277</a>)</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/b9cf19ba5ec4aa6ff108933fa4a1adb075f8c4fb"><code>b9cf19b</code></a> Automated rollback of commit 2a55706f9d36afd1742c97fd0d611ea8af0ac3f8. (<a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14246">#14246</a>)</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/e4baf96abe5a2d854ba1102ce1911eba467ef65a"><code>e4baf96</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14240">#14240</a> from vthib/backport-cython-fix</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/fe87c909399c45d54374817a52af40d6cfd8e063"><code>fe87c90</code></a> fix python code generation compatibility with Cython</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/b2b7a51158418f41cff0520894836c15b1738721"><code>b2b7a51</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14054">#14054</a> from mkruskal-google/iwyu-fix2</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/955d371eacedc37eeff8bf61a1046d694a5fbc8a"><code>955d371</code></a> <code>Printer</code>: Include Abseil's <code>str_cat.h</code> for <code>absl::AlphaNum</code></li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/0d22de520bf3fdb0978a962ae90b72db2f560cef"><code>0d22de5</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/14042">#14042</a> from mkruskal-google/iwyu-fix</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/96c1a7d47702f6c82b96e2a0ff112b30ea85c330"><code>96c1a7d</code></a> Add dependency on absl::layout, and explicitly include absl layout includes</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/d3c44a42c218213d6d5a9fbd66afdd9e8946bfce"><code>d3c44a4</code></a> Merge pull request <a href="https://redirect.github.com/protocolbuffers/protobuf/issues/13898">#13898</a> from protocolbuffers/24.x-202309071538</li>
<li>Additional commits viewable in <a href="https://github.com/protocolbuffers/protobuf/compare/v3.24.3...v3.24.4">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Google.Protobuf&package-manager=nuget&previous-version=3.24.3&new-version=3.24.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@ dependabot rebase` will rebase this PR
- `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@ dependabot merge` will merge this PR after your CI passes on it
- `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@ dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@ dependabot reopen` will reopen this PR if it is closed
- `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <[email protected]>

…pache#38075)

### Rationale for this change
Prevent a segfault upon passing a non-(chunked_)array object into `Table.filter`. See apache#37650.

### What changes are included in this PR?
1. Check filter Datum kind to make sure that it is an array or a chunked array
2. test that attempting to filter a table with another table raises a not implemented error

### Are these changes tested?
In PyArrow, yes

### Are there any user-facing changes?
Raises an error if a non-array or non-chunked_array object is passed into `Table.filter`

* Closes: apache#37650

Lead-authored-by: Patrick Clarke <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…mpty dataframes (apache#38037)

### Rationale for this change

The implementation of the DataFrame Interchange Protocol does not currently support consumption of dataframes with 0 number of chunks (empty dataframes).

### What changes are included in this PR?

Add a workaround to not error in this case.

### Are these changes tested?

Yes, added `test_empty_dataframe` in `python/pyarrow/tests/interchange/test_conversion.py`.

### Are there any user-facing changes?
No.
* Closes: apache#37050

Authored-by: AlenkaF <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…arrow.fs.S3FileSystem and s3fs (apache#37768)

### What changes are included in this PR?

Update `test_get_file_info_with_selector ` to check that the base directory is not included in the `S3FileSystem` or `s3fs` filesystem. Also remove old comments and flexible checks.

* Closes: apache#36983

Lead-authored-by: AlenkaF <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…thon install docs (apache#38146)

### What changes are included in this PR?

The option to set custom path to timezone database through python API has been implemented in apache#35600 and is documented in `docs/source/python/install.rst` with this PR.

* Closes: apache#38145

Authored-by: AlenkaF <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…he#38173)

### Rationale for this change

It will reduce build time.

### What changes are included in this PR?

Install GoogleTest by apt.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38172

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…es` (apache#36846)

Revival of apache#12000

### Rationale for this change

It would be great to be able to do promotions when `concat`'ing a table, such as:

```python
def test_concat_tables_with_promotion_int():
    import pyarrow as pa
    t1 = pa.Table.from_arrays(
        [pa.array([1, 2], type=pa.int64())], ["int"])
    t2 = pa.Table.from_arrays(
        [pa.array([3, 4], type=pa.int32())], ["int"])

    result = pa.concat_tables([t1, t2], promote=True)

    assert result.equals(pa.Table.from_arrays([
        pa.array([1, 2, 3, 4], type=pa.int64())
    ], ["int"]))
```

### What changes are included in this PR?

### Are these changes tested?

### Are there any user-facing changes?

* Closes: apache#36845

Lead-authored-by: Fokko Driesprong <[email protected]>
Co-authored-by: David Li <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…B Interface (apache#38020)

### Rationale for this change

To unblock use cases that are not satisfied by the default Arrow -> MATLAB conversions (i.e. the `toMATLAB()` on `arrow.array.Array`), we would like expose the underlying Arrow data representation as a property on `arrow.array.Array`. One possible name for this property would be `DataLayout`, which would be an `arrow.array.DataLayout` object. Note, this class does not yet exist, so we would have to add it.

For example, the `DataLayout` property for temporal array types would return an object of the following class type: 

```matlab
classdef TemporalDataLayout < arrow.array.DataLayout
    properties
       Values % an arrow.array.Int32Array or an arrow.array.Int64Array
       Valid  % an arrow.buffer.Buffer 
    end
end
```

However, the `Valid` property on this class would need to be an `arrow.buffer.Buffer` object, which does not yet exist in the MATLAB interface.  Therefore, it would be helpful to first add the `arrow.buffer.Buffer` class before adding the `DataLayout` property/class hierarchy. It's worth mentioning that adding `arrow.buffer.Buffer` will open up additional advanced use cases in the future.

### What changes are included in this PR?

Added `arrow.buffer.Buffer` MATLAB class.

*Properties of `arrow.buffer.Buffer`*
 1. `NumBytes` - a  scalar `int64` value representing the size of the buffer in bytes. 

*Methods of `arrow.buffer.Buffer`*
1. `toMATLAB` - returns the data in the buffer as `Nx1` `uint8` vector, where `N` is the number of bytes.
2. `fromMATLAB(data)` - Static method that creates an `arrow.buffer.Buffer` from a numeric array. 

**Example:**
```matlab
>> dataIn = [1 2];
>> buffer = arrow.buffer.Buffer.fromMATLAB(dataIn)

buffer = 

  Buffer with properties:

    NumBytes: 16

>> dataOut = toMATLAB(buffer)

dataOut =

  16×1 uint8 column vector

     0
     0
     0
     0
     0
     0
   240
    63
     0
     0
     0
     0
     0
     0
     0
    64

% Reinterpret bit pattern as a double array 
>> toDouble = typecast(dataOut, "double")

toDouble =

     1
     2
```

### Are these changes tested?

Yes. Added a new test class called `tBuffer.m`

### Are there any user-facing changes?

Yes. Users can now create `arrow.buffer.Buffer` objects via the `fromMATLAB` static method. However, there's not much users can do with this object as of now. We implemented this class to facilitate adding `DataLayout` property to `arrow.array.Array`, as described in the **Rational for this change** section. 

* Closes: apache#38015

Authored-by: Sarah Gilmore <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>

Fix typo: represenation -> representation.

Authored-by: Alexander Grueneberg <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…cts (apache#37376)

### Rationale for this change

See: apache#32439 

### What changes are included in this PR?

During conversion from Python to Arrow, when a struct's child hits a capacity error and chunking is triggered, this can leave the Finish'd chunk in an invalid state since the struct's length does not match the length of its children.

This change simply tries to Append the children first, and only if successful will Append the struct. This is safe because the order of Append'ing between the struct and its child is not specified. It is only specified that they must be consistent with each other.

This is per: 

https://github.com/apache/arrow/blob/86b7a84c9317fa08222eb63f6930bbb54c2e6d0b/cpp/src/arrow/array/builder_nested.h#L507-L508

### Are these changes tested?

A unit test is added that would previously have an invalid data error.

```
>       tab = pa.Table.from_pandas(df)

pyarrow/tests/test_pandas.py:4970: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/table.pxi:3788: in pyarrow.lib.Table.from_pandas
    return cls.from_arrays(arrays, schema=schema)
pyarrow/table.pxi:3890: in pyarrow.lib.Table.from_arrays
    result.validate()
pyarrow/table.pxi:3170: in pyarrow.lib.Table.validate
    check_status(self.table.Validate())

# ...

FAILED pyarrow/tests/test_pandas.py::test_nested_chunking_valid - pyarrow.lib.ArrowInvalid: Column 0: In chunk 0: Invalid: List child array invalid: Invalid: Struct child array #0 has length smaller than expected for struct array (2 < 3)
```

NOTE: This unit test uses about 7GB of memory (max RSS) on my macbook pro. This might make CI challenging; I'm open to suggestions to limit it.

### Are there any user-facing changes?

No
* Closes: apache#32439

Lead-authored-by: Mike Lui <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…hon scalar cast) (apache#38038)

### Rationale for this change

Adding `string -> date32/date64` cast kernels, which then also fixes the pyarrow scalar cast method (which was earlier refactored to rely on the general cast kernels)

* Closes: apache#37411

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…en both data page and version is V2 (apache#38163)

### Rationale for this change

Only use RLE as BOOLEAN default encoding when data page is V2.

Previous patch ( apache#36955 ) set RLE encoding for Boolean type by default.  However, parquet-cpp might write format v2 file with page v1 by default. This might cause parquet-cpp generating RLE encoding for boolean type by default. As https://issues.apache.org/jira/browse/PARQUET-2222 says, we still need some talks about that. So, we:

1. Still allow writing RLE on DataPage V2. This keeps same as parquet rust
2. If DataPage V1 is used, don't use RLE as default Boolean encoding.

### What changes are included in this PR?

Only use RLE as BOOLEAN default encoding when both data page and version is V2.

### Are these changes tested?

Yes

### Are there any user-facing changes?

RLE encoding change for Boolean.

* Closes: apache#36882

Lead-authored-by: mwish <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…E_LIST_VIEW array formats (apache#37468)

### Rationale for this change

Go implementation of apache#35345.

### What changes are included in this PR?

- [x] Add `LIST_VIEW` and `LARGE_LIST_VIEW` to datatype.go
- [x] Add `ListView` and `LargeListView` to list.go
- [x] Add `ListViewType` and `LargeListViewType` to datatype_nested.go
- [x] Add list-view builders
- [x] Implement list-view comparison in compare.go
- [x] String conversion in both directions
- [x] Validation of list-view arrays
- [x] Generation of random list-view arrays
- [x] Concatenation of list-view arrays in concat.go
- [x] JSON serialization/deserialization
- [x] Add data used for tests in `arrdata.go`
- [x] Add Flatbuffer changes
- [x] Add IPC support

### Are these changes tested?

Yes. Existing tests are being changed to also cover list-view variations as well as new tests focused solely on the list-view format.

### Are there any user-facing changes?

New structs and functions introduced.
* Closes: apache#35344

Authored-by: Felipe Oliveira Carvalho <[email protected]>
Signed-off-by: Matt Topol <[email protected]>

…che#38175)

### Rationale for this change

It has g++ 13 support.

### What changes are included in this PR?

Update version and checksum.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: apache#38174

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change

We run "git describe --tag --dirty" implicitly in
cpp/cmake_modules/DefineOptions.cmake. If we use "--dirty", .git/index's owner may be changed. Because "git describe" touches .git/index for "--dirty".

We can avoid changing .git/index's owner by not using "--dirty".

### What changes are included in this PR?

Remove "--dirty".

### Are these changes tested?

Yes. I used "strace git describe ...".

### Are there any user-facing changes?

No.
* Closes: apache#37767

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

### Rationale for this change

Add Jacob Wujciak-Jens as one of the R package authors

### What changes are included in this PR?

Add Jacob Wujciak-Jens as one of the R package authors

### Are these changes tested?

No

### Are there any user-facing changes?

No

Lead-authored-by: Nic Crane <[email protected]>
Co-authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…4 Java JNI" (apache#38194)

### Rationale for this change

We don't have enough disk space.

### What changes are included in this PR?

Remove unused pre-installed software.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38193

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…on test mode (apache#38177)

### Rationale for this change

Unit tests for each implementations are done on unit test mode.

### What changes are included in this PR?

Detect integration test only mode.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38159

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…mpiled binaries (apache#38115)

### Rationale for this change

This change is to restore parity with the previous solution on macOS (brew does cs validation) and improve security for windows and linux. This also align with CRAN policy.

### What changes are included in this PR?

This PR adds a script that can be run after the arrow release (once all files have been pushed to the artifactory) before the CRAN submission to download the checksum files for the pre-compiled binaries which are already added through the usual release. *libs.R have been extended to use these checksum files to validate the downloaded binaries.

### Are these changes tested?

The r-binary-packages nightlies generate checksums and use them when building binary packages, this way the code path is tested. They do not modify the actual src package though.

### Are there any user-facing changes?
no (outside of log messages)
* Closes: apache#37941

Authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Nic Crane <[email protected]>

…tion in the new Dataset API (apache#34616)

### Rationale for this change

The purpose of this pull request is to support modular encryption in the new Dataset API.  See [https://docs.google.com/document/d/13EysCNC6-Nu9wnJ8YpdzmD-aMLn4i2KXUJTNqIihy7A/edit#](url) for supporting document.

### What changes are included in this PR?

I made improvements to the C++ and Python code to enable the Dataset API to have per-file settings for each file saved. Previously, the Dataset API applied the same encryption properties to all saved files, but now I've updated the code to allow for greater flexibility. In the Python code, I've added support for the changes by updating the ParquetFormat class to accept DatasetEncryptionConfiguration and DatasetDecryptionConfiguration structures. With these changes, you can pass the format object to the write_dataset function, giving you the ability to set unique encryption properties for each file in your Dataset.

### Are these changes tested?

Yes, unit tests are included. I have also included a python sample project.

### Are there any user-facing changes?

Yes,  as stated above the ParquetFormat class has optional parameters for DatasetEncryptionConfiguration and DatasetDecryptionConfiguration through setters and getters.  The Dataset now has the option using this to set different file encryption properties per file

* Closes: apache#29238

Lead-authored-by: Don <[email protected]>
Co-authored-by: Donald Tolley <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: anjakefala <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Weston Pace <[email protected]>
Co-authored-by: Gang Wu <[email protected]>
Co-authored-by: scoder <[email protected]>
Co-authored-by: Will Jones <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…#38154)

### Rationale for this change

New C# drivers need the ability to concatenate array data, particularly for metadata calls.

### What changes are included in this PR?

Converts a previously internal class and method to a public class and method.

### Are these changes tested?

Yes

### Are there any user-facing changes?

It exposes previously hidden functionality.

Resolves apache#38153
* Closes: apache#38153

Lead-authored-by: David Coe <[email protected]>
Co-authored-by: davidhcoe <[email protected]>
Signed-off-by: David Li <[email protected]>

…ay` and `arrow.array.ChunkedArray` to `NumElements` (apache#38190)

### Rationale for this change

We would like to rename the `Length` property on `arrow.array.Array` (and `arrow.array.ChunkedArray`) to `NumElements` because MATLAB has a function called `length(A)`, which returns the length of the largest array dimension in `A`.  
Because `arrow.array.Array` inherits from `matlab.mixin.Scalar`, it's size is always `1x1`, so the `length()` function always returns `1`. This may confuse users because the `Length` property of `arrow.array.Array` returns the number of elements within the array, which may not be `1`. 

```matlab
>> array = arrow.array([1 2 3 4 5]);

% The length property returns the number of elements in the array
>> array.Length

ans =

  int64

   5

% The length method returns the length of the largest dimension
>> length(array)

ans =

     1
```

I suspect this inconsistency will confuse users, so we should rename the `Length` property to `NumElements`. 

### What changes are included in this PR?

1. Renamed `Length` property on `arrow.array.Array` to `NumElements`
2. Renamed `Length` property on `arrow.array.ChunkedArray` to `NumElements`

### Are these changes tested?

Yes. I modified the existing `Length` test cases to now test the `NumElements` property.

### Are there any user-facing changes?

Yes. Please note this is a breaking change because `Length` is no longer a property on either `arrow.array.Array` or `arrow.array.ChunkedArray`. However, we have not yet cut an initial release of the MATLAB interface, so we don't expect this change should affect too many users at this point. 

* Closes: apache#38164

Authored-by: Sarah Gilmore <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>

…mensions, implemented using ExtensionType (apache#37166)

### Rationale for this change

For use cases where underlying datatype and number of dimensions in tensors are equal but not the actual shape we want to add a `VariableShapeTensorType`.
See apache#24868 and huggingface/datasets#5272

### What changes are included in this PR?

This introduces definition of `arrow.variable_shape_tensor` extension and it's C++ implementation and a Python wrapper.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

This introduces new extension type to the user.
* Closes: apache#24868

Lead-authored-by: Rok Mihevc <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…pache#37901)

### Rationale for this change

Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release.

### What changes are included in this PR?

This PR adds jobs to build pyarrow wheels for Python 3.12.

### Are these changes tested?

They will be tested via archery tasks

### Are there any user-facing changes?

No but users will be able to use pyarrow with Python 3.12

* Closes: apache#37880

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

… proxies (apache#38199)

### Rationale for this change

Fixes a bug in the Go Flight client library that makes using AuthenticateBasicToken unreliable behind proxies.

### What changes are included in this PR?

Closes the sending side of the bi-directional stream for the Flight Handshake RPC call before trying to read the headers. This matches the C++ implementation.

<img width="609" alt="Screenshot 2023-10-11 at 6 18 40 PM" src="https://github.com/apache/arrow/assets/879445/05e23c6a-0ff8-41fc-825b-8add7fe938bc">

### Are these changes tested?

I've tested these changes against my service deployed behind CloudFlare and verified the error listed in the linked issue disappears.

### Are there any user-facing changes?

No
* Closes: apache#38198

Authored-by: Phillip LeBlanc <[email protected]>
Signed-off-by: Matt Topol <[email protected]>

…conan (apache#38202)

### Rationale for this change
There is a conflict between the required Zlib version when using both thrift and GRPC.

### What changes are included in this PR?

Pinning zlib when using thrifht.

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* Closes: apache#38201

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…he#38189)

### Rationale for this change

In support of adding an [`arrow.array.ListArray`](apache#37815) MATLAB class, this pull request adds a new `arrow.type.ListType` MATLAB class.

### What changes are included in this PR?

1. New `arrow.list(<type>)` MATLAB construction function.
2. New `arrow.list.ListType` MATLAB class. `ListType` has a property named `Type` which indicates the inner type of the `List`. `Type` can be set to any subclass `arrow.type.Type` (including `arrow.type.ListType`, to support nested lists).
3. New `arrow.type.ID.List` type ID enumeration value.
4. New `arrow.type.traits.ListTraits` type traits class. Some of the properties, such as `ArrayConstructor` and `ArrayProxyClassName`, are set to `missing` because they are dependent on adding [`arrow.array.ListArray`](apache#37815) first.

**Example**
```matlab
% Create a simple List<String> type.
>> stringListType = arrow.list(arrow.string())

stringListType = 

  ListType with properties:

      ID: List
    Type: [1x1 arrow.type.StringType]

% Create a nested List<List<Boolean>> type.
>> nestedListType = arrow.list(arrow.list(arrow.boolean()))

nestedListType = 

  ListType with properties:

      ID: List
    Type: [1x1 arrow.type.ListType]

% Extract the first-level, inner type, which is List<Boolean>.
>> innerType = nestedListType.Type

innerType = 

  ListType with properties:

      ID: List
    Type: [1x1 arrow.type.BooleanType]

% Extract the second-level, nested inner type, which is Boolean.
>> innerType.Type  

ans = 

  BooleanType with properties:

    ID: Boolean
```

### Are these changes tested?

Yes.

1. Added `tListType.m`.
2. Added `tListTraits.m`.
3. Updated `tField.m` to include `arrow.list`.
4. Updated `tID.m` to include `arrow.type.ID.List`.
6. Updated `tTypeDisplay.m` to include `arrow.type.ListType`.
7. Updated `ttraits.m` to include `arrow.type.traits.ListTraits`.

### Are there any user-facing changes?

Yes.

Client MATLAB code can now creates instances of `arrow.type.ListType` by using the `arrow.list(<type>)` construction function.

### Future Directions

1. apache#37815
* Closes: apache#37812

Authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>

### Rationale for this change

The NEWS file needs updating for 14.0.0.

### What changes are included in this PR?

The NEWS file is updated with commits since 13.0.0.

### Are these changes tested?

N/A

### Are there any user-facing changes?

No
* Closes: apache#38142

Lead-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Nic Crane <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>

…eight default (small) on smaller screens (apache#38148)

### Rationale for this change

The Sphinx theme we have been using (PyData Sphinx Theme) has been pinned to an older version for a while now and with the apache#36591 we have updated the code and are now using version 0.14.0 for the dev docs.

This PR fixes bugs we have encountered after the PR updating the theme has been merged.

### What changes are included in this PR?

- Have default header size for smaller screens and keep it increased for bigger screens.

* Closes: apache#38209

Authored-by: AlenkaF <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…naryArray (apache#37418)

### Rationale for this change
A Dictionary Compaction Function for DictionaryArray is supported.

### What changes are included in this PR?
Add a Function for Dictionary Compaction

### Are these changes tested?
Yes

Are there any user-facing changes?
No
* Closes: apache#37378

Lead-authored-by: Junming Chen <[email protected]>
Co-authored-by: Ben Harkins <[email protected]>
Co-authored-by: Benjamin Kietzman <[email protected]>
Signed-off-by: Benjamin Kietzman <[email protected]>

…ordBatch` (apache#38215)

### Rationale for this change

Currently, there is a `NumColumns` property on `arrow.tabular.RecordBatch`, but no `NumRows` property. It would be useful to be able to query the number of rows in a `RecordBatch`.

This pull request adds a `NumRows` property to `arrow.tabular.RecordBatch` to mirror the design of `arrow.tabular.Table`.

### What changes are included in this PR?

1. Added new `NumRows` property to `arrow.tabular.RecordBatch`

**Example**
```matlab
>> matlabTable = array2table(rand(10, 5))           

matlabTable =

  10x5 table

      Var1        Var2       Var3       Var4        Var5  
    ________    ________    _______    _______    ________

     0.76062     0.12009    0.98898    0.29974     0.42165
     0.64994     0.85116    0.71768    0.58693     0.31061
     0.33593     0.87823    0.87766    0.38206     0.45742
    0.031364      0.8336    0.71528    0.14987      0.3618
      0.5986     0.81193    0.25784    0.21073     0.76715
     0.46493     0.40281    0.39729    0.16737     0.94521
     0.18738     0.16351    0.46437    0.45545     0.40774
     0.67682      0.3577    0.94882     0.1295    0.022501
     0.29368     0.47122    0.99682    0.46011     0.34275
      0.6849    0.064717    0.89719    0.38302      0.4523

>> arrowRecordBatch = arrow.recordBatch(matlabTable);

>> arrowRecordBatch.NumRows

ans =

  int64

   10
```

### Are these changes tested?

Yes.

1. Added `NumRows` test to `tRecordBatch` test class.
3. Updated `EmptyTable` test (renamed to `EmptyRecordBatch`) in `tRecordBatch` test class.
4. Added  `FromArraysNoInputs` test to mirror the `FromArraysNoInputs` test in `tTable` test class.

### Are there any user-facing changes?

Yes.

This pull request adds a new public `NumRows` property to the `arrow.tabular.RecordBatch` class. Users can query the number of rows in an `arrow.tabular.RecordBatch` by accessing the `NumRows` property.

### Future Directions

1. apache#38214
3. apache#38213 
* Closes: apache#37592

Authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>

…#38176)

### Rationale for this change

It's an internal bundled library. We should not install it as a part of Arrow.

### What changes are included in this PR?

Exclude all Azure SDK for C++ jobs including install jobs aren't executed by default. Building jobs are executed because they are required to build Arrow.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: apache#37510

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…pache#38222)

### Rationale for this change

Module caches don't have write permission by owner. So we can remove them by `rm -rf`.

### What changes are included in this PR?

Run `go clean -modcache` after all builds.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38200

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change

The test-r-versions job is failing because not all of our dependencies support R 3.5. We follow the tidyverse support policy where possible, which means we only support R 3.6 and above. Thus, we can drop the test for R 3.5.

### What changes are included in this PR?

R 3.5 was removed from the test matrix for test-r-versions

### Are these changes tested?

Yes

### Are there any user-facing changes?

No
* Closes: apache#38226

Authored-by: Dewey Dunnington <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…ncryption tests (apache#38244)

* Closes: apache#38243

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

… and closing delimiters to `arrow::PrettyPrintDelimiters` (apache#38187)

### Rationale for this change

This is a follow up to  apache#37981.

in order to make the [`arrow::PrettyPrint`](https://github.com/apache/arrow/blob/7667b81bffcb5b361fab6d61c42ce396d98cc6e1/cpp/src/arrow/pretty_print.h#L101) functionality for `arrow::Array` more flexible, it would be useful to be able to specify a custom `Array` opening and closing delimiter other than `"["` and `"]"`.

For example, the MATLAB interface wraps the Arrow C++ libraries and being able to specify a custom opening and closing delimiter for `Array` would make it possible to make the display of MATLAB `arrow.array.Array` objects more MATLAB-like.

In order to support custom `Array` opening and closing delimiters, this pull request adds two new properties, `open` and `close`, to the [`arrow::PrettyPrintDelimiters`](https://github.com/apache/arrow/blob/c37059ad7b87f0cbb681f6388aca0e3f02860351/cpp/src/arrow/pretty_print.h#L38) struct.

This enable use cases like the ability to display an `arrow::Array` as `<1,2,3>` instead of `[1,2,3]`, by setting `options.array_delimiters.open = "<"` and `options.array_delimiters.close = ">"`.

### What changes are included in this PR?

This pull request adds two new properties to the [`arrow::PrettyPrintDelimiters`](https://github.com/apache/arrow/blob/c37059ad7b87f0cbb681f6388aca0e3f02860351/cpp/src/arrow/pretty_print.h#L38) struct:

1. `open` - the opening delimiter to use for an `Array` or `ChunkedArray` (default = `[`).
1. `close` - the closing delimiter to use for an `Array` or `ChunkedArray` (default = `]`).

### Are these changes tested?

Yes.

1. Added two new tests: (1) `ArrayCustomOpenCloseDelimiter` and (2) `ChunkedArrayCustomOpenCloseDelimiter`.
2. All existing tests related to `arrow::PrettyPrint` pass.

### Are there any user-facing changes?

Yes.

This pull request adds two new public, user-facing properties, (1) `open` (of type `std::string`) and (2) `close` (also of type `std::string`) to the `PrettyPrintDelimiters` struct. This enables client code to specify custom opening and closing delimiters to use when printing an `arrow::Array` or `arrow::ChunkedArray` by changing the values of the nested `open` and `close` properties of the `array_delimiters`/`chunked_array_delimiters` properties of `PrettyPrintOptions`.

### Notes

1. This pull request was motivated by our desire to improve the display of Arrow related classes in the MATLAB interface, but it is hopefully a generic enough change that it may benefit other use cases too.

### Future Directions

1. Now that client code can easily specify custom opening, closing, and element delimiters, it may make sense to address apache#30951.
* Closes: apache#37979

Authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

apache#38229)

### Rationale for this change

The minimal nightly build are failing with examples that won't run without the dataset feature

### What changes are included in this PR?

- Added `examplesIf` where needed
- Redocumented

### Are these changes tested?

Yes, by all R CMD check jobs

### Are there any user-facing changes?

No
* Closes: apache#38228

Authored-by: Dewey Dunnington <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>

…r@v2 (apache#38218)

### Rationale for this change

CI jobs that used setup-r@ v1 no longer run without error.

### What changes are included in this PR?

- Updated the rchk job to use the `setup-r@ v2`
- Updated the devdocs job to use `setup-r@ v2`. To make this work, we needed to remove the Windows build because it was installing an old version of R. It seems that the job has been running an outdated and unsable (for most users) for a very long time.

### Are these changes tested?

Will be covered by crossbow jobs submitted below.

### Are there any user-facing changes?

No.
* Closes: apache#38197

Lead-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>

…n closing (apache#38110)

### Rationale for this change

The Java FlightStream can raise an error if metadata is transferred and ends up being closed twice.

```
java.lang.IllegalStateException: RefCnt has gone negative
	at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:458)
	at org.apache.arrow.memory.BufferLedger.release(BufferLedger.java:130)
	at org.apache.arrow.memory.BufferLedger.release(BufferLedger.java:104)
	at org.apache.arrow.memory.ArrowBuf.close(ArrowBuf.java:1044)
	at org.apache.arrow.util.AutoCloseables.close(AutoCloseables.java:97)
	at org.apache.arrow.flight.FlightStream.close(FlightStream.java:208)
```

### What changes are included in this PR?

When FlightStream is closed, remove any reference of previous metadata to prevent reference count going negative if closed again. Also added `ExchangeReaderWriter.getResult()` for convenience and clear up ambiguity on error handling.

### Are these changes tested?

Unit tests added for closing with metadata and 

### Are there any user-facing changes?

Added `ExchangeReaderWriter.getResult()`
* Closes: apache#38096

Authored-by: Bryan Cutler <[email protected]>
Signed-off-by: David Li <[email protected]>

…pache#38231)

### Rationale for this change

I have fixed the error occurring when using the "DARROW_FLIGHT_SQL=ON" option on a Mac with ARM Architecture CPU.

### What changes are included in this PR?

It seems that implicit values are not recognized in this environment (Mac ARM), so I have added values explicitly.

### Are these changes tested?

yes (build and run unit test)

### Are there any user-facing changes?

No

* Closes: apache#38210

Authored-by: Hyunseok Seo <[email protected]>
Signed-off-by: David Li <[email protected]>

apache#38232)

### Rationale for this change

We have several nightly builds failing with errors building the manual as a result of unicode characters. The unicode characters aren't new, so I'm not sure why this happened now.

### What changes are included in this PR?

Install a distribution of latex that supports unicode characters (maybe)?

### Are these changes tested?

Yes

### Are there any user-facing changes?

No
* Closes: apache#38227

Lead-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…e#38225)

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.15.0 to 0.17.0.
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/golang/net/commit/b225e7ca6dde1ef5a5ae5ce922861bda011cfabd"><code>b225e7c</code></a> http2: limit maximum handler goroutines to MaxConcurrentStreams</li>
<li><a href="https://github.com/golang/net/commit/88194ad8ab44a02ea952c169883c3f57db6cf9f4"><code>88194ad</code></a> go.mod: update golang.org/x dependencies</li>
<li><a href="https://github.com/golang/net/commit/2b60a61f1e4cf3a5ecded0bd7e77ea168289e6de"><code>2b60a61</code></a> quic: fix several bugs in flow control accounting</li>
<li><a href="https://github.com/golang/net/commit/73d82efb96cacc0c378bc150b56675fc191894b9"><code>73d82ef</code></a> quic: handle DATA_BLOCKED frames</li>
<li><a href="https://github.com/golang/net/commit/5d5a036a503f8accd748f7453c0162115187be13"><code>5d5a036</code></a> quic: handle streams moving from the data queue to the meta queue</li>
<li><a href="https://github.com/golang/net/commit/350aad2603e57013fafb1a9e2089a382fe67dc80"><code>350aad2</code></a> quic: correctly extend peer's flow control window after MAX_DATA</li>
<li><a href="https://github.com/golang/net/commit/21814e71db756f39b69fb1a3e06350fa555a79b1"><code>21814e7</code></a> quic: validate connection id transport parameters</li>
<li><a href="https://github.com/golang/net/commit/a600b3518eed7a9a4e24380b4b249cb986d9b64d"><code>a600b35</code></a> quic: avoid redundant MAX_DATA updates</li>
<li><a href="https://github.com/golang/net/commit/ea633599b58dc6a50d33c7f5438edfaa8bc313df"><code>ea63359</code></a> http2: check stream body is present on read timeout</li>
<li><a href="https://github.com/golang/net/commit/ddd8598e5694aa5e966e44573a53e895f6fa5eb2"><code>ddd8598</code></a> quic: version negotiation</li>
<li>Additional commits viewable in <a href="https://github.com/golang/net/compare/v0.15.0...v0.17.0">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/net&package-manager=go_modules&previous-version=0.15.0&new-version=0.17.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@ dependabot rebase` will rebase this PR
- `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@ dependabot merge` will merge this PR after your CI passes on it
- `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@ dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@ dependabot reopen` will reopen this PR if it is closed
- `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/arrow/network/alerts).

</details>

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Matt Topol <[email protected]>

…pache#38192)

### Rationale for this change

To use a more modern version of `flatc` in Arrow.

### What changes are included in this PR?

1) Re-generating the the C++ files with a `flatc` based on the latest tag on the `flatbuffer` repo.
2) Copy the minimal set of includes to our vendored folder
3) Manually re-apply the patches that have been applied to the previous headers to fix issues

### Are these changes tested?

Tested by building everything correctly.
* Closes: apache#35497

Authored-by: Felipe Oliveira Carvalho <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

### Rationale for this change

The latest version of `r/R/install-arrow.R`  was not working properly, since it was relying on the `on_rosetta()` function, which is not defined elsewhere. I just fixed the identification of rosetta in the script.

With the current code, the following gives an error

````r
> source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R") 
> install_arrow()
Error in on_rosetta() : could not find function "on_rosetta"
````

### What changes are included in this PR?

It only removed the `on_rosetta()` function, which was not defined elsewhere, and reverted back to the `rosetta` object to identify if rosetta is present or not on a user's system.

### Are these changes tested?

Yes. It was tested with the current code and the proposed PR. The proposed PR works as expected.

### Are there any user-facing changes?

No.

* Closes: apache#37907

Lead-authored-by: Fernando Mayer <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Signed-off-by: Nic Crane <[email protected]>

### Rationale for this change

Several PRs over the last few months have update the build system to be more friendly for developers. During this process it has also come to light that we haven't supported the Windows development setup documented here since R 4.1 (released in spring 2021). I had to remove Windows from the test-r-devdocs job because the approach used there was not compatible with the `setup-r@ v2` action, and the job was failing with the `@ v1` action.

### What changes are included in this PR?

- Updated the sections on using pre-built static libraries and bundled builds
- Removed the Windows section regarding the bundled build. This section would need rewriting to support the last two minor releases of R but in the meantime I think it is mostly confusing.

### Are these changes tested?

They are documentation changes. They are also slightly optimisitc: we can fix problems with the developer setup incrementally between releases, but it's more difficult to update our documentation. This PR documents the intended behaviour after apache#38236 .

### Are there any user-facing changes?

No.
* Closes: apache#37945

Lead-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>

…8195)

### Rationale for this change

Previously GCS/S3 support would need to be explicitly enabled in source builds (when they are build without `NOT_CRAN`). As we want the macos binaries to be fully featured we should turn the features on when the dependencies exists. 

### What changes are included in this PR?

This PR enables this behavior for macOS only, on Linux setting `NOT_CRAN` or  `LIBARROW_MINIMAL=false` is still required.

### Are these changes tested?

Crossbow and locally (thanks @ paleolimbot )
* Closes: apache#38043

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

### Rationale for this change

Fix typo in docs introduced in apache#38232

### What changes are included in this PR?

Change spelling of "delimeter" to "delimiter"

### Are these changes tested?

No
### Are there any user-facing changes?

Just docs

Authored-by: Nic Crane <[email protected]>
Signed-off-by: Nic Crane <[email protected]>

…ache#37896)

### Rationale for this change

User scenario: When we use acero plan, many smaller batches may be generated through agg and hashjoin. In addition, due to the mpp database, there is data distribution. When there are many segments, each segment data is compared at this time. Small, in order to improve performance, we hope to merge multiple fragmented small batches into one large batch for calculation together.

### What changes are included in this PR?

record_batch.cc
record_batch.h
record_batch_test.cc

### Are these changes tested?

yes, see record_batch_test.cc

### Are there any user-facing changes?

yes

* Closes: apache#37895

Authored-by: light-city <[email protected]>
Signed-off-by: Benjamin Kietzman <[email protected]>

…rarily (apache#38238)

* Closes: apache#38239

Lead-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…ultiple times in gandiva (apache#37752)

### Rationale for this change
Try to fix apache#37751

### What changes are included in this PR?

The exported functions defined in gandiva are registered multiple times unnecessarily. This PR tries to address this issue.

### Are these changes tested?

Yes.

### Are there any user-facing changes?
No

Authored-by: Yue Ni <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…tring and Binary Types in Hash Join (apache#38147)

### Rationale for this change

We found that the wrong results in inner joins during hash join operations were caused by a problem with how large strings and binary types were handled. The `Slice` function was not calculating their sizes correctly.

To fix this, I changed the `Slice` function to calculate the sizes correctly, based on the type of data for large string and binary. 

* Issue raised: apache#37729 

### What changes are included in this PR?

* The `Slice` function has been updated to correctly calculate the offset for Large String and Large Binary types, and assertion statements have been added to improve maintainability.
* Unit tests (`TEST(KeyColumnArray, SliceBinaryTest)`)for the Slice function have been added. 
* During random tests for Hash Join (`TEST(HashJoin, Random)`), modifications were made to allow the creation of Large String as key column values.

### Are these changes tested?

Yes

### Are there any user-facing changes?

Acero might not have a large user base as it is an experimental feature, but I deemed the issue of incorrect join results as critical and have addressed the bug.

* Closes: apache#38074

Authored-by: Hyunseok Seo <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…egin() where a char pointer is expected (apache#38265)

### Rationale for this change

The MSVC compiler doesn't seem to allow user code to assume `std::string_view::const_iterator` is `const char*`, so using only `re2::StringPiece` and preferring to call `.data()` instead of `.begin()` should make things more uniform across different compilers and STL implementations.

### What changes are included in this PR?

 - Using `re2::StringPiece` instead of `std::string_view` to interact with `re2`
 - Use `data()` instead of `begin()` where a `char*` is expected

### Are these changes tested?

Yes, by existing tests.
* Closes: apache#38263

Authored-by: Felipe Oliveira Carvalho <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

### Rationale for this change

We need more disk space...

### What changes are included in this PR?

Remove more pre-installed files.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38206

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

### Rationale for this change
Making sure the documentation that shows up on pkg.go.dev will show that the package is compatible with go1.19+

### What changes are included in this PR?
slight patch/minor version updates of some dependencies along with a documentation update in `doc.go`.
* Closes: apache#38285

Authored-by: Matt Topol <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

### What changes are included in this PR?

Complete support for the Duration array type in the C# implementation.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

The Duration array type is now supported in the C# library. This also
does some slight refactoring of classes which could impact edge cases of
user scenarios.
* Closes: apache#38061

… of empty (apache#37862)

### Rationale for this change

Fixes apache#37861.

### What changes are included in this PR?

Add `BinaryArray.GetBytes(int, out bool)` overload that enables
`StringArray.GetString` to reliably determine if the value is null or
empty without needing an additional call to `IsNull`.

### Are these changes tested?

Added a test for `StringArray.GetString` (which didn't appear to have
any existing tests). Two of the cases fail without this change. I also
updated the tests that call `BinaryArray.GetBytes` to use the new
overload instead of a separate call to `IsNull` as extra validation.

### Are there any user-facing changes?

New public overload to an existing API.
* Closes: apache#37861

…nature (apache#38283)

### Rationale for this change

The type signature of `ReplaceString` should be identical when arrow is compiled with or without `ARROW_WITH_RE2`.

### What changes are included in this PR?

The right signature + delegating to the implementation that takes `re2::StringPiece`. The conversion should be a no-op when compiled and optimized.

### Are these changes tested?

By existing tests and CI checks.

* Closes: apache#38282

Authored-by: Felipe Oliveira Carvalho <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…images (apache#38287)

### Rationale for this change
Fix CI failures for job that is getting out of space.

### What changes are included in this PR?

Using our free disk space script to add space for the ubuntu-r-only-r images.

### Are these changes tested?

On CI

### Are there any user-facing changes?
No
* Closes: apache#38286

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

### What changes are included in this PR?

Bump versions of Go for our nightly tests to match supported Go versions

### Are these changes tested?
Via archery

### Are there any user-facing changes?

No

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…harp (apache#38290)

Bumps [xunit.runner.visualstudio](https://github.com/xunit/visualstudio.xunit) from 2.5.1 to 2.5.3.
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/6b60a9e56a82a4010e2c65481cfa40c227f97fd5"><code>6b60a9e</code></a> v2.5.3</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/48d5ef68535d44fc58641a51c6844da6e4e3f7b7"><code>48d5ef6</code></a> Add diagnostic messages around DiaSession failures</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/32b7c99f6023cc12552833c740ca2a2fce3bdbd6"><code>32b7c99</code></a> Incorrect URL for Git repository in .nuspec file</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/ce0f849c747ee27ba6c947e71e483f730293a0fd"><code>ce0f849</code></a> Bump up to v2.5.3-pre</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/b56611bde0cdc1513d5420f4a066b63df0de7870"><code>b56611b</code></a> v2.5.2</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/9b843e136f0d5a385846b88ccfae89d667c6093b"><code>9b843e1</code></a> Add Microsoft.TestPlatform.ObjectModel as a dependency in the nuspec for .NET...</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/9d2800688c5a5007fdb9f0f073c9a4b749e62ef4"><code>9d28006</code></a> Add /.vscode/tasks.json for builds inside VS Code</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/af96d96e2cf2f15ce18a21df462553053558d8e0"><code>af96d96</code></a> Latest dependencies</li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/2dd69895010eaff737e38e567f7f792996cd3ba4"><code>2dd6989</code></a> Undo the assembly merge from <a href="https://redirect.github.com/xunit/visualstudio.xunit/issues/383">#383</a></li>
<li><a href="https://github.com/xunit/visualstudio.xunit/commit/1c51d513e6479ba76856ce976433478bf75818ed"><code>1c51d51</code></a> Latest dependencies</li>
<li>Additional commits viewable in <a href="https://github.com/xunit/visualstudio.xunit/compare/2.5.1...2.5.3">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=xunit.runner.visualstudio&package-manager=nuget&previous-version=2.5.1&new-version=2.5.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@ dependabot rebase` will rebase this PR
- `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@ dependabot merge` will merge this PR after your CI passes on it
- `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@ dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@ dependabot reopen` will reopen this PR if it is closed
- `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <[email protected]>

Bumps [pythonnet](https://github.com/pythonnet/pythonnet) from 3.0.2 to 3.0.3.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/pythonnet/pythonnet/releases">pythonnet's releases</a>.</em></p>
<blockquote>
<h2>v3.0.3</h2>
<h2>What's Changed</h2>
<ul>
<li>change PyScope to PyModule by <a href="https://github.com/OmkarBorhade98"><code>@​OmkarBorhade98</code></a> in <a href="https://redirect.github.com/pythonnet/pythonnet/pull/2231">pythonnet/pythonnet#2231</a></li>
<li>use enum name in repr by <a href="https://github.com/koubaa"><code>@​koubaa</code></a> in <a href="https://redirect.github.com/pythonnet/pythonnet/pull/2239">pythonnet/pythonnet#2239</a></li>
<li>ARM CI fix by <a href="https://github.com/lostmsu"><code>@​lostmsu</code></a> in <a href="https://redirect.github.com/pythonnet/pythonnet/pull/2256">pythonnet/pythonnet#2256</a></li>
<li>Python 3.12 by <a href="https://github.com/filmor"><code>@​filmor</code></a> in <a href="https://redirect.github.com/pythonnet/pythonnet/pull/2249">pythonnet/pythonnet#2249</a></li>
<li>Only run docs CI/main CI for corresponding changes by <a href="https://github.com/lostmsu"><code>@​lostmsu</code></a> in <a href="https://redirect.github.com/pythonnet/pythonnet/pull/2257">pythonnet/pythonnet#2257</a></li>
<li>Revert &quot;Only run docs CI/main CI for corresponding changes&quot; by <a href="https://github.com/filmor"><code>@​filmor</code></a> in <a href="https://redirect.github.com/pythonnet/pythonnet/pull/2260">pythonnet/pythonnet#2260</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/OmkarBorhade98"><code>@​OmkarBorhade98</code></a> made their first contribution in <a href="https://redirect.github.com/pythonnet/pythonnet/pull/2231">pythonnet/pythonnet#2231</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a href="https://github.com/pythonnet/pythonnet/compare/v3.0.2...v3.0.3">https://github.com/pythonnet/pythonnet/compare/v3.0.2...v3.0.3</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pythonnet/pythonnet/blob/master/CHANGELOG.md">pythonnet's changelog</a>.</em></p>
<blockquote>
<h2><a href="https://github.com/pythonnet/pythonnet/releases/tag/v3.0.3">3.0.3</a> - 2023-10-11</h2>
<h3>Added</h3>
<ul>
<li>Support for Python 3.12</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Use enum name in <code>repr</code></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pythonnet/pythonnet/commit/0a5a63ccc4ee1e104ab0aae75ede8aa3441a3481"><code>0a5a63c</code></a> Release 3.0.3</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/5a4a986f9e2b9f4f505db68809f45a63c39b1418"><code>5a4a986</code></a> Revert &quot;only run docs CI/main CI for corresponding changes (<a href="https://redirect.github.com/pythonnet/pythonnet/issues/2257">#2257</a>)&quot; (<a href="https://redirect.github.com/pythonnet/pythonnet/issues/2260">#2260</a>)</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/04670ea1c8a761944e19ac89acd18a43a82cc352"><code>04670ea</code></a> only run docs CI/main CI for corresponding changes (<a href="https://redirect.github.com/pythonnet/pythonnet/issues/2257">#2257</a>)</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/88418da6a87f3dcdd861ff0022d8beb6164ae3ee"><code>88418da</code></a> Merge pull request <a href="https://redirect.github.com/pythonnet/pythonnet/issues/2249">#2249</a> from pythonnet/python-3.12</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/293f8b1d810db1e63253b950a4b4844b35c38bab"><code>293f8b1</code></a> Python 3.12 has been released, use final version</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/fb494705652af998a4f346aeb88ebff836c72eb4"><code>fb49470</code></a> Ignore test-case on Python 3.12</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/080d1bd59f2b231db2081474cf126e46aa039505"><code>080d1bd</code></a> For now skip over &quot;new style&quot; weakrefs in clear</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/8dfe4080d642397a7efcb52b8d3aa69da1675713"><code>8dfe408</code></a> Remove deprecated function call</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/d057724e5167d7088a3dc77310a37c38473e318d"><code>d057724</code></a> Update changelog</li>
<li><a href="https://github.com/pythonnet/pythonnet/commit/7a31d38755c71d0385bf28e758ab9cf96a2db481"><code>7a31d38</code></a> Add 3.12 to CI and metadata</li>
<li>Additional commits viewable in <a href="https://github.com/pythonnet/pythonnet/compare/v3.0.2...v3.0.3">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pythonnet&package-manager=nuget&previous-version=3.0.2&new-version=3.0.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@ dependabot rebase` will rebase this PR
- `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@ dependabot merge` will merge this PR after your CI passes on it
- `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@ dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@ dependabot reopen` will reopen this PR if it is closed
- `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <[email protected]>

### What changes are included in this PR?

Fixes apache#38297 

### Are these changes tested?

Yes
* Closes: apache#38297

Bumps [xunit](https://github.com/xunit/xunit) from 2.5.1 to 2.5.2.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/xunit/xunit/commit/32931855e213ccb74d75a899af76f40388b2131b"><code>3293185</code></a>
v2.5.2</li>
<li><a
href="https://github.com/xunit/xunit/commit/4d9d4fcc5d5f0e09445026674f62415993a3fe73"><code>4d9d4fc</code></a>
Roll back singleton NullMessageSink and
NullSourceInformationProvider</li>
<li><a
href="https://github.com/xunit/xunit/commit/45078f3a86e04e495dbbe3e434a8dddc1ce42a8d"><code>45078f3</code></a>
Pick up latest analyzers</li>
<li><a
href="https://github.com/xunit/xunit/commit/8ae0d067bb8f427f38c5524b4f3c47f99e844f77"><code>8ae0d06</code></a>
Remove VisualStudioSourceInformationProvider and DiaSession-related
classes</li>
<li><a
href="https://github.com/xunit/xunit/commit/4a35ce7722aa3ac01a1e04fc3757c448d454aaa5"><code>4a35ce7</code></a>
Missing comparer pass-through on DictionaryExtensions</li>
<li><a
href="https://github.com/xunit/xunit/commit/9ab4ecef596b9677bcc53bdb3b5e1117b0a83bf4"><code>9ab4ece</code></a>
<a href="https://redirect.github.com/xunit/xunit/issues/2755">#2755</a>:
Tests for another embedded collection regression (v2)</li>
<li><a
href="https://github.com/xunit/xunit/commit/e499f14da33189d3c197b5f4abf3a6f769e0a80e"><code>e499f14</code></a>
Bump up to v2.5.2-pre</li>
<li>See full diff in <a
href="https://github.com/xunit/xunit/compare/2.5.1...2.5.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=xunit&package-manager=nuget&previous-version=2.5.1&new-version=2.5.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

### Rationale for this change

The test fail with the latest version of duckdb (0.9.1).

### What changes are included in this PR?

The test was changed so that it did not depend on non-deterministic behaviour. We sort all of the other expectations involving a group_by to avoid this problem...we hadn't changed this one yet because it didn't fail in any previous version of duckdb.

### Are these changes tested?

Yes

### Are there any user-facing changes?

No
* Closes: apache#38293

Authored-by: Dewey Dunnington <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>

…oupReader (apache#37003)

Currently we only can get a ColumnReader for a column from RowGroupReader. We need an API to return a RecordReader for the column.

Moved ComputeLevelInfo from column_writer to level_conversion so that it can be shared between column_writer and file_reader.
* Closes: apache#37002

Lead-authored-by: Fatemah Panahi <[email protected]>
Co-authored-by: Fatemah Panahi <[email protected]>
Signed-off-by: Gang Wu <[email protected]>

### What changes are included in this PR?

The primary C# test project is updated to run for .NET 4.7.2 only on Windows platforms.

### Are these changes tested?

Yes.
* Closes: apache#36815

Authored-by: Curt Hagenlocher <[email protected]>
Signed-off-by: Curt Hagenlocher <[email protected]>

…y environment variable (apache#38267)

### Rationale for this change

It's useful when troubleshooting issues with Arrow's S3 filesystem implementation to raise the log level. Currently, this can only be done in C++ and Python, but not from R. In addition, the log level can only be set during S3 initialization and not directly so the user has to introduce explicit S3 initialization code to turn on logging and must make sure this code is called before S3 initialization.

While discussing exposing control of log level to R, we realized that allowing the log level to be controlled by environment variable may be more intuitive and useful and would just be a good addition for C++, Python, and R. 

### What changes are included in this PR?

- A new environment variable `AWS_S3_LOG_LEVEL` with documentation for controlling S3 log level
- Updated documentation for C++, Python, and R
- A new `InitializeS3()` as a quality-of-life thing for C++ users. Feel free to ask me to remove this.

No changes are needed directly for Python and R because these implementation uses the internal implicit initializer `EnsureS3Initialized` rather than the explicit form, `InitializeS3`. And it's the behavior of the `EnsureS3Initialized` routine that's changed here.

### Are these changes tested?

Yes. I added a unit test for the new `GetS3LogLevelFromEnvOrDefault` and tested from Python and R manually. I didn't add a test to make sure the underlying `AwsInstance` gets set up correctly because it looked like it would require a refactor and didn't seem worth it.

### Are there any user-facing changes?

Yes. A new way to turn on logging for S3 and matching docs in C++, Python, and R.

* Closes: apache#35260

Lead-authored-by: Bryce Mecum <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Nic Crane <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…rsions.json (apache#38241)

This PR corrects the version for the `version_match` to be equal to the version defined in versions.json. This way the text is correctly displayed in the version switcher button.
* Closes: apache#38240

Authored-by: AlenkaF <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…pache#38302)

### Rationale for this change

test-r-rhub-ubuntu-gcc-release-latest doesn't have enough disk space.

### What changes are included in this PR?

Remove pre-installed files on Azure Pipelines too.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38295

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

### Rationale for this change

Verify JDK 21 in CI in time for the Arrow v14 release.

### What changes are included in this PR?

* Bump latest Java version from 20 -> 21 in CI

### Are these changes tested?

Yes, via CI.

### Are there any user-facing changes?

No.
* Closes: apache#36994

Authored-by: Dane Pitkin <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…eference docs (apache#38277)

* Closes: apache#36044

Authored-by: Divyansh200102 <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…w parquet encoding option (apache#38070)

### Rationale for this change

Since parquet C++ has complete all encoding, we can publish this in Python doc.

### What changes are included in this PR?

Add encoding in document.

### Are these changes tested?

No

### Are there any user-facing changes?

No

* Closes: apache#37312

Lead-authored-by: mwish <[email protected]>
Co-authored-by: mwish <[email protected]>
Co-authored-by: Rok Mihevc <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…he#38303)

### Rationale for this change

`expr` was printing the number of matching chars which showed up as noise in the log (which we want to avoid as much as possible to avoid any false positive checks)
See apache#38236 (comment) for @ jonkeane's investigation.

### What changes are included in this PR?

Replace use of expr with test.

### Are these changes tested?
Crossbow

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Signed-off-by: Jonathan Keane <[email protected]>

…che#38258)

fixes apache#38216 
* Closes: apache#38216

Lead-authored-by: Divyansh200102 <[email protected]>
Co-authored-by: Divyansh200102 <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Signed-off-by: Jonathan Keane <[email protected]>

…the sidebar TOC (apache#38313)

* Closes: apache#38312

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>

…#38288)

Bumps [@ babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse) from 7.22.5 to 7.23.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/babel/babel/releases"><code>@​babel/traverse</code>'s releases</a>.</em></p>
<blockquote>
<h2>v7.23.2 (2023-10-11)</h2>
<p><strong>NOTE</strong>: This release also re-publishes <code>@ babel/core</code>, even if it does not appear in the linked release commit.</p>
<p>Thanks <a href="https://github.com/jimmydief"><code>@​jimmydief</code></a> for your first PR!</p>
<h4>:bug: Bug Fix</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16033">#16033</a> Only evaluate own String/Number/Math methods (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16022">#16022</a> Rewrite <code>.tsx</code> extension when using <code>rewriteImportExtensions</code> (<a href="https://github.com/jimmydief"><code>@​jimmydief</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16017">#16017</a> Fix: fallback to typeof when toString is applied to incompatible object (<a href="https://github.com/JLHwung"><code>@​JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>, <code>babel-plugin-transform-modules-commonjs</code>, <code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>, <code>babel-runtime</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16025">#16025</a> Avoid override mistake in namespace imports (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h4>Committers: 5</h4>
<ul>
<li>Babel Bot (<a href="https://github.com/babel-bot"><code>@​babel-bot</code></a>)</li>
<li>Huáng Jùnliàng (<a href="https://github.com/JLHwung"><code>@​JLHwung</code></a>)</li>
<li>James Diefenderfer (<a href="https://github.com/jimmydief"><code>@​jimmydief</code></a>)</li>
<li>Nicolò Ribaudo (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
<li><a href="https://github.com/liuxingbaoyu"><code>@​liuxingbaoyu</code></a></li>
</ul>
<h2>v7.23.1 (2023-09-25)</h2>
<p>Re-publishing <code>@ babel/helpers</code> due to a publishing error in 7.23.0.</p>
<h2>v7.23.0 (2023-09-25)</h2>
<p>Thanks <a href="https://github.com/lorenzoferre"><code>@​lorenzoferre</code></a> and <a href="https://github.com/RajShukla1"><code>@​RajShukla1</code></a> for your first PRs!</p>
<h4>:rocket: New Feature</h4>
<ul>
<li><code>babel-plugin-proposal-import-wasm-source</code>, <code>babel-plugin-syntax-import-source</code>, <code>babel-plugin-transform-dynamic-import</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15870">#15870</a> Support transforming <code>import source</code> for wasm (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-module-transforms</code>, <code>babel-helpers</code>, <code>babel-plugin-proposal-import-defer</code>, <code>babel-plugin-syntax-import-defer</code>, <code>babel-plugin-transform-modules-commonjs</code>, <code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>, <code>babel-runtime</code>, <code>babel-standalone</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15878">#15878</a> Implement <code>import defer</code> proposal transform support (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>, <code>babel-parser</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15845">#15845</a> Implement <code>import defer</code> parsing support (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
<li><a href="https://redirect.github.com/babel/babel/pull/15829">#15829</a> Add parsing support for the &quot;source phase imports&quot; proposal (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>, <code>babel-helper-module-transforms</code>, <code>babel-parser</code>, <code>babel-plugin-transform-dynamic-import</code>, <code>babel-plugin-transform-modules-amd</code>, <code>babel-plugin-transform-modules-commonjs</code>, <code>babel-plugin-transform-modules-systemjs</code>, <code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15682">#15682</a> Add <code>createImportExpressions</code> parser option (<a href="https://github.com/JLHwung"><code>@​JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-standalone</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15671">#15671</a> Pass through nonce to the transformed script element (<a href="https://github.com/JLHwung"><code>@​JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-function-name</code>, <code>babel-helper-member-expression-to-functions</code>, <code>babel-helpers</code>, <code>babel-parser</code>, <code>babel-plugin-proposal-destructuring-private</code>, <code>babel-plugin-proposal-optional-chaining-assign</code>, <code>babel-plugin-syntax-optional-chaining-assign</code>, <code>babel-plugin-transform-destructuring</code>, <code>babel-plugin-transform-optional-chaining</code>, <code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>, <code>babel-runtime</code>, <code>babel-standalone</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15751">#15751</a> Add support for optional chain in assignments (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>, <code>babel-plugin-proposal-decorators</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15895">#15895</a> Implement the &quot;decorator metadata&quot; proposal (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15893">#15893</a> Add <code>t.buildUndefinedNode</code> (<a href="https://github.com/liuxingbaoyu"><code>@​liuxingbaoyu</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code></li>
</ul>

</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/babel/babel/blob/main/CHANGELOG.md"><code>@​babel/traverse</code>'s changelog</a>.</em></p>
<blockquote>
<h2>v7.23.2 (2023-10-11)</h2>
<h4>:bug: Bug Fix</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16033">#16033</a> Only evaluate own String/Number/Math methods (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16022">#16022</a> Rewrite <code>.tsx</code> extension when using <code>rewriteImportExtensions</code> (<a href="https://github.com/jimmydief"><code>@​jimmydief</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16017">#16017</a> Fix: fallback to typeof when toString is applied to incompatible object (<a href="https://github.com/JLHwung"><code>@​JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>, <code>babel-plugin-transform-modules-commonjs</code>, <code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>, <code>babel-runtime</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/16025">#16025</a> Avoid override mistake in namespace imports (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h2>v7.23.0 (2023-09-25)</h2>
<h4>:rocket: New Feature</h4>
<ul>
<li><code>babel-plugin-proposal-import-wasm-source</code>, <code>babel-plugin-syntax-import-source</code>, <code>babel-plugin-transform-dynamic-import</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15870">#15870</a> Support transforming <code>import source</code> for wasm (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-module-transforms</code>, <code>babel-helpers</code>, <code>babel-plugin-proposal-import-defer</code>, <code>babel-plugin-syntax-import-defer</code>, <code>babel-plugin-transform-modules-commonjs</code>, <code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>, <code>babel-runtime</code>, <code>babel-standalone</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15878">#15878</a> Implement <code>import defer</code> proposal transform support (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>, <code>babel-parser</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15845">#15845</a> Implement <code>import defer</code> parsing support (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
<li><a href="https://redirect.github.com/babel/babel/pull/15829">#15829</a> Add parsing support for the &quot;source phase imports&quot; proposal (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>, <code>babel-helper-module-transforms</code>, <code>babel-parser</code>, <code>babel-plugin-transform-dynamic-import</code>, <code>babel-plugin-transform-modules-amd</code>, <code>babel-plugin-transform-modules-commonjs</code>, <code>babel-plugin-transform-modules-systemjs</code>, <code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15682">#15682</a> Add <code>createImportExpressions</code> parser option (<a href="https://github.com/JLHwung"><code>@​JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-standalone</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15671">#15671</a> Pass through nonce to the transformed script element (<a href="https://github.com/JLHwung"><code>@​JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-function-name</code>, <code>babel-helper-member-expression-to-functions</code>, <code>babel-helpers</code>, <code>babel-parser</code>, <code>babel-plugin-proposal-destructuring-private</code>, <code>babel-plugin-proposal-optional-chaining-assign</code>, <code>babel-plugin-syntax-optional-chaining-assign</code>, <code>babel-plugin-transform-destructuring</code>, <code>babel-plugin-transform-optional-chaining</code>, <code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>, <code>babel-runtime</code>, <code>babel-standalone</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15751">#15751</a> Add support for optional chain in assignments (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>, <code>babel-plugin-proposal-decorators</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15895">#15895</a> Implement the &quot;decorator metadata&quot; proposal (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15893">#15893</a> Add <code>t.buildUndefinedNode</code> (<a href="https://github.com/liuxingbaoyu"><code>@​liuxingbaoyu</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15913">#15913</a> Add <code>rewriteImportExtensions</code> option to TS preset (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-parser</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15896">#15896</a> Allow TS tuples to have both labeled and unlabeled elements (<a href="https://github.com/yukukotani"><code>@​yukukotani</code></a>)</li>
</ul>
</li>
</ul>
<h4>:bug: Bug Fix</h4>
<ul>
<li><code>babel-plugin-transform-block-scoping</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15962">#15962</a> fix: <code>transform-block-scoping</code> captures the variables of the method in the loop (<a href="https://github.com/liuxingbaoyu"><code>@​liuxingbaoyu</code></a>)</li>
</ul>
</li>
</ul>
<h4>:nail_care: Polish</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15797">#15797</a> Expand evaluation of global built-ins in <code>@ babel/traverse</code> (<a href="https://github.com/lorenzoferre"><code>@​lorenzoferre</code></a>)</li>
</ul>
</li>
<li><code>babel-plugin-proposal-explicit-resource-management</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15985">#15985</a> Improve source maps for blocks with <code>using</code> declarations (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h4>:microscope: Output optimization</h4>
<ul>
<li><code>babel-core</code>, <code>babel-helper-module-transforms</code>, <code>babel-plugin-transform-async-to-generator</code>, <code>babel-plugin-transform-classes</code>, <code>babel-plugin-transform-dynamic-import</code>, <code>babel-plugin-transform-function-name</code>, <code>babel-plugin-transform-modules-amd</code>, <code>babel-plugin-transform-modules-commonjs</code>, <code>babel-plugin-transform-modules-umd</code>, <code>babel-plugin-transform-parameters</code>, <code>babel-plugin-transform-react-constant-elements</code>, <code>babel-plugin-transform-react-inline-elements</code>, <code>babel-plugin-transform-runtime</code>, <code>babel-plugin-transform-typescript</code>, <code>babel-preset-env</code>
<ul>
<li><a href="https://redirect.github.com/babel/babel/pull/15984">#15984</a> Inline <code>exports.XXX =</code> update in simple variable declarations (<a href="https://github.com/nicolo-ribaudo"><code>@​nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h2>v7.22.20 (2023-09-16)</h2>

</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/babel/babel/commit/b4b9942a6cde0685c222eb3412347880aae40ad5"><code>b4b9942</code></a> v7.23.2</li>
<li><a href="https://github.com/babel/babel/commit/b13376b346946e3f62fc0848c1d2a23223314c82"><code>b13376b</code></a> Only evaluate own String/Number/Math methods (<a href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/16033">#16033</a>)</li>
<li><a href="https://github.com/babel/babel/commit/ca58ec15cb6dde6812c36997477e44880bec0bba"><code>ca58ec1</code></a> v7.23.0</li>
<li><a href="https://github.com/babel/babel/commit/0f333dafcf470f1970083e4e695ced6aec8bead0"><code>0f333da</code></a> Add <code>createImportExpressions</code> parser option (<a href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15682">#15682</a>)</li>
<li><a href="https://github.com/babel/babel/commit/3744545649fdc21688a2f3c97e1e39dbebff0d21"><code>3744545</code></a> Fix linting</li>
<li><a href="https://github.com/babel/babel/commit/c7e6806e2194deb36c330f543409c792592b22d4"><code>c7e6806</code></a> Add <code>t.buildUndefinedNode</code> (<a href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15893">#15893</a>)</li>
<li><a href="https://github.com/babel/babel/commit/38ee8b4dd693f1e2bd00107bbc1167ce84736ea0"><code>38ee8b4</code></a> Expand evaluation of global built-ins in <code>@ babel/traverse</code> (<a href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15797">#15797</a>)</li>
<li><a href="https://github.com/babel/babel/commit/9f3dfd90211472cf0083a3234dd1a1b857ce3624"><code>9f3dfd9</code></a> v7.22.20</li>
<li><a href="https://github.com/babel/babel/commit/3ed28b29c1fb15588369bdd55187b69f1248e87d"><code>3ed28b2</code></a> Fully support <code>||</code> and <code>&amp;&amp;</code> in <code>pluginToggleBooleanFlag</code> (<a href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15961">#15961</a>)</li>
<li><a href="https://github.com/babel/babel/commit/77b0d7359909c94f3797c24006f244847fbc8d6d"><code>77b0d73</code></a> v7.22.19</li>
<li>Additional commits viewable in <a href="https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=@ babel/traverse&package-manager=npm_and_yarn&previous-version=7.22.5&new-version=7.23.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@ dependabot rebase` will rebase this PR
- `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@ dependabot merge` will merge this PR after your CI passes on it
- `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@ dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@ dependabot reopen` will reopen this PR if it is closed
- `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@ dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/arrow/network/alerts).

</details>

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <[email protected]>

…set-file-parquet-encryption-test (apache#38306)

If OpenSSL initializes itself from a non-main thread, it can fail deallocating all memory at shutdown.

This is really a benign leak, but we don't want any spurious CI errors.

* Closes: apache#38304

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

### Rationale for this change

Broken link on the [arrow/README.md](https://github.com/apache/arrow/blob/main/README.md?plain=1#L102) after the landing page for the dev docs has been changed in apache#36591.
### What changes are included in this PR?

Update the link.

Authored-by: AlenkaF <[email protected]>
Signed-off-by: AlenkaF <[email protected]>

…37797)

### Rationale for this change

### What changes are included in this PR?

* A new specification for Arrow PyCapsules and related dunder methods
* Implementing the dunder methods for `DataType`, `Field`, `Schema`, `Array`, `RecordBatch`, `Table`, and `RecordBatchReader`.

### Are these changes tested?

Yes, I've added various roundtrip tests for each of the types.

### Are there any user-facing changes?

This introduces some new APIs and documents them.

* Closes: apache#34031
* Closes: apache#35531

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

apache#38314)

### Rationale for this change
The usage of `SetFinalizer` means that it's not *guaranteed* that calling `Release()` on an imported Record or Array will actually free the memory during the lifetime of the process. Instead we can leverage a shared buffer count, atomic ref counting and a custom allocator to ensure proper and more timely memory releasing when importing from C Data interface.

### What changes are included in this PR?
* Some simplifications of code to use `unsafe.Slice` instead of the deprecated handling of `reflect.SliceHeader` to improve readability
* Updating tests using `mallocator.Mallocator` in order to easily allow testing to ensure that memory is being cleaned up and freed
* Fixing a series of memory leaks subsequently found by the previous change of using the `mallocator.Mallocator` to track the allocations used for testing arrays.

### Are these changes tested?
Yes, unit tests are updated and included.

* Closes: apache#38281

Authored-by: Matt Topol <[email protected]>
Signed-off-by: Matt Topol <[email protected]>

…bility-version (apache#36595)

### Rationale for this change

`MSVC_VERSION` isn't used correctly. 

### What changes are included in this PR?

* Remove MSVC 2013 or earlier check because it reached EOL
* Don't change `-fms-compatibility-version` depending on `MSVC_VERSION`
   * Always use the lowest supported Visual Studio version for it

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: apache#36594

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

)

### Rationale for this change

Our release script (`dev/release/02-source.sh`) resolves symlinks in source archive but our lint script (`dev/archery/archery/utils/source.py`) doesn't resolve symlinks. So we may detect RAT problem by our CI.

### What changes are included in this PR?

Resolve symlinks in our lint script too.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#38332

Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…che#38324)

### Rationale for this change

gdb on conda-forge frequently conflicts with the newest Python version (example for Python 3.12: conda-forge/gdb-feedstock#66)

### What changes are included in this PR?

Use the system-provided gdb instead of trying to install it from conda-forge.

This works because the Python interpreter used for gdb is entirely independent from the Python interpreter under test, and gdb itself can debug any binary, regardless of binutils version.

However, there is a complication: if the system Python version and the conda-forge Python version are the same, then for some reason `pyarrow/tests/test_gdb.py` would fail with the system gdb. In that case we still install the conda-forge gdb.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.

* Closes: apache#38323

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…ation testing (apache#38248)

### Rationale for this change

### What changes are included in this PR?

### Are these changes tested?

### Are there any user-facing changes?

* Closes: apache#37910

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

…he#38269)

### Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step. 

### What changes are included in this PR?

Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. 

I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. 

### Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. 
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. 

### Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported. 

* Closes: apache#37511

Lead-authored-by: Thomas Newton <[email protected]>
Co-authored-by: Benjamin Kietzman <[email protected]>
Signed-off-by: Benjamin Kietzman <[email protected]>