GH-37917: [Parquet] Add OpenAsync for FileSource #37918

eeroel · 2023-09-27T20:32:17Z

Rationale for this change

Improves performance of file reads with an expensive Open operation.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

No

Closes: [Parquet] Implement OpenAsync for FileSource #37917

github-actions · 2023-09-27T20:32:45Z

⚠️ GitHub issue #37917 has been automatically assigned in GitHub to PR creator.

eeroel · 2023-09-27T20:37:05Z

Not sure about the details of the implementation but got it to compile and it seems to work as expected.

mapleFU · 2023-09-28T06:29:48Z

Windows failed is not related, but I think here we need a clang-format:

--- /arrow/cpp/src/arrow/dataset/file_parquet.cc
+++ /arrow/cpp/src/arrow/dataset/file_parquet.cc (after clang format)

you can just clang-format -i /path/to/arrow/cpp/src/arrow/dataset/file_parquet.cc

mapleFU

General LGTM

mapleFU · 2023-09-28T06:31:07Z

cpp/src/arrow/dataset/file_base.cc

+    return Future<std::shared_ptr<io::RandomAccessFile>>::MakeFinished(std::make_shared<io::BufferReader>(buffer_));
+  }
+
+  return Future<std::shared_ptr<io::RandomAccessFile>>::MakeFinished(custom_open_());


Oh, since custom_open_ might still blocking here?

Yes, wasn't sure how to deal with it. FileSource would need its own IOContext instance to run it async?

You can leave a comment here, and keep it here? I think at lease, open via fileSystem could be help, and custom_open_ won't be worse

👍 left a todo

I think this TODO is valuable but should be linked to a new issue where we extend custom_open_ to return Future<std::share_ptr<io::RandomAccessFile>>. That would be necessary to enable a custom but non blocking open

Created a new issue and updated the comment #37962

mapleFU · 2023-09-28T06:33:42Z

cpp/src/arrow/dataset/file_parquet.cc

+      [path](
+          const Status& status) -> Result<std::unique_ptr<parquet::ParquetFileReader>> {
+        return WrapSourceError(status, path);
+      });


Would WrapSourceError called twice since L516-L519 already called that?

Good point, I removed the callback

mapleFU · 2023-09-28T08:35:35Z

cpp/src/arrow/dataset/file_parquet.cc

+        auto rfut = parquet::ParquetFileReader::OpenAsync(
+            std::move(input), std::move(properties), metadata);
+        ARROW_ASSIGN_OR_RAISE(auto reader, rfut.MoveResult());
+        return reader;


Can we just return rfut here?

The return value needs to be of type Result, so the future needs to be unwrapped. Not sure if this was the right way to do it, but seems to work

For a continuation (callback passed to Future::Then) the return type can also be Future. Please change it to Future, then you can return rfut directly instead of blocking in a continuation

Ah, I see, thanks. For some reason I get a note: copy constructor is implicitly deleted because 'unique_ptr<parquet::ParquetFileReader>' has a user-declared move constructor if I try to return the future, any idea what I'm missing?

eeroel · 2023-09-28T10:10:46Z

Hmm, noticed that the execution freezes when I set io_thread_count <= fragment_readahead (testing with Pyarrow). A deadlock probably? I wonder if it's introduced by this change, or if it's a bug that just manifests now because of higher concurrency. I think I have observed such a freeze recently with a recent Pyarrow release, but couldn't reproduce.

bkietz · 2023-09-28T13:45:13Z

cpp/src/arrow/dataset/file_base.cc

+    return Future<std::shared_ptr<io::RandomAccessFile>>::MakeFinished(std::make_shared<io::BufferReader>(buffer_));
+  }
+
+  return Future<std::shared_ptr<io::RandomAccessFile>>::MakeFinished(custom_open_());


I think this TODO is valuable but should be linked to a new issue where we extend custom_open_ to return Future<std::share_ptr<io::RandomAccessFile>>. That would be necessary to enable a custom but non blocking open

bkietz · 2023-09-28T13:49:07Z

cpp/src/arrow/dataset/file_parquet.cc

+      input_fut.Then([=](const std::shared_ptr<arrow::io::RandomAccessFile>&) mutable
+                     -> Result<std::unique_ptr<parquet::ParquetFileReader>> {
+        ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::io::RandomAccessFile> input,
+                              input_fut.MoveResult());


Instead of copying input_fut implicitly by keeping it in closure, I think it'd be more clear to use the value passed to the callback:

Suggested change

input_fut.Then([=](const std::shared_ptr<arrow::io::RandomAccessFile>&) mutable

-> Result<std::unique_ptr<parquet::ParquetFileReader>> {

ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::io::RandomAccessFile> input,

input_fut.MoveResult());

source.OpenAsync().Then([=](std::shared_ptr<arrow::io::RandomAccessFile> input)

-> Result<std::unique_ptr<parquet::ParquetFileReader>> {

(The TODO(ARROW-12259) only applies to futures of move-only types as returned by ParquetFileReader::OpenAsync(), and doesn't apply to the return value of source.OpenAsync().)

Sorry, let me give a more complete suggestion

Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReaderAsync( const FileSource& source, const std::shared_ptr<ScanOptions>& options, const std::shared_ptr<parquet::FileMetaData>& metadata) const { ARROW_ASSIGN_OR_RAISE( auto parquet_scan_options, GetFragmentScanOptions<ParquetFragmentScanOptions>(kParquetTypeName, options.get(), default_fragment_scan_options)); auto properties = MakeReaderProperties(*this, parquet_scan_options.get(), options->pool); auto self = checked_pointer_cast<const ParquetFileFormat>(shared_from_this()); return source.OpenAsync().Then([=](const std::shared_ptr<io::RandomAccessFile>& input) mutable { return parquet::ParquetFileReader::OpenAsync(input, std::move(properties), metadata) .Then( [=](const std::unique_ptr<parquet::ParquetFileReader>& reader) mutable -> Result<std::shared_ptr<parquet::arrow::FileReader>> { auto arrow_properties = MakeArrowReaderProperties( *self, *reader->metadata(), *options, *parquet_scan_options); std::unique_ptr<parquet::arrow::FileReader> arrow_reader; RETURN_NOT_OK(parquet::arrow::FileReader::Make( options->pool, // TODO(ARROW-12259): workaround since we have Future<(move-only type)> // It *wouldn't* be safe to const_cast reader except that here we know // there are no other waiters on the reader. std::move( const_cast<std::unique_ptr<parquet::ParquetFileReader>&>(reader)), std::move(arrow_properties), &arrow_reader)); return std::move(arrow_reader); }, [path = source.path()](const Status& status) -> Result<std::shared_ptr<parquet::arrow::FileReader>> { return WrapSourceError(status, path); }); }); }

@lidavidm re my alternative ARROW-12259 workaround

a bit yikes, but if it works

Thank you very much, updated code to this and it seems to work. The deadlock issue I mentioned earlier is also gone.

bkietz · 2023-09-28T13:53:32Z

cpp/src/arrow/dataset/file_parquet.cc

+        auto rfut = parquet::ParquetFileReader::OpenAsync(
+            std::move(input), std::move(properties), metadata);
+        ARROW_ASSIGN_OR_RAISE(auto reader, rfut.MoveResult());
+        return reader;


For a continuation (callback passed to Future::Then) the return type can also be Future. Please change it to Future, then you can return rfut directly instead of blocking in a continuation

mapleFU · 2023-09-29T07:16:08Z

Hmm, noticed that the execution freezes when I set io_thread_count <= fragment_readahead (testing with Pyarrow). A deadlock probably? I wonder if it's introduced by this change, or if it's a bug that just manifests now because of higher concurrency. I think I have observed such a freeze recently with a recent Pyarrow release, but couldn't reproduce.

@eeroel https://github.com/apache/arrow/pull/37918/files#r1340188707 I think this is the problem that we call blocking in async execution.

It's a bit like this problem I fixed: #37514 . The problem is that blocking in async function under same Executor is a bad behavior, which might cause deadlock.

eeroel · 2023-09-30T15:49:18Z

Hmm, noticed that the execution freezes when I set io_thread_count <= fragment_readahead (testing with Pyarrow). A deadlock probably? I wonder if it's introduced by this change, or if it's a bug that just manifests now because of higher concurrency. I think I have observed such a freeze recently with a recent Pyarrow release, but couldn't reproduce.

@eeroel https://github.com/apache/arrow/pull/37918/files#r1340188707 I think this is the problem that we call blocking in async execution.

It's a bit like this problem I fixed: #37514 . The problem is that blocking in async function under same Executor is a bad behavior, which might cause deadlock.

OK, thanks for the explanation. Looks like this is fixed in @bkietz suggestion

mapleFU · 2023-09-30T17:01:35Z

LGTM now. Lets wait for committers review.

mapleFU · 2023-10-02T15:17:43Z

I think we need to clang-format the file /arrow/cpp/src/arrow/dataset/file_parquet.cc, other looks ok to me

eeroel · 2023-10-02T15:45:10Z

I think we need to clang-format the file /arrow/cpp/src/arrow/dataset/file_parquet.cc, other looks ok to me

✅

eeroel · 2023-10-02T17:39:53Z

Sorry, still had an issue with formatting in file_base.cc, should be fixed now

mapleFU · 2023-10-03T17:59:32Z

CI passed, should we merge or trigger more CI @bkietz

bkietz · 2023-10-03T18:29:34Z

@eeroel could you please rebase to pick up the fix ( https://github.com/apache/arrow/pull/37867/files#diff-1bba462ab050e89360fd88110a689e85ee037749cea091a1848ab574381d3795R155 ) for several of the CI failures ( https://github.com/apache/arrow/actions/runs/6383550289/job/17363301059?pr=37918#step:6:1217 )

eeroel · 2023-10-03T19:18:29Z

@eeroel could you please rebase to pick up the fix ( https://github.com/apache/arrow/pull/37867/files#diff-1bba462ab050e89360fd88110a689e85ee037749cea091a1848ab574381d3795R155 ) for several of the CI failures ( https://github.com/apache/arrow/actions/runs/6383550289/job/17363301059?pr=37918#step:6:1217 )

👍 done

bkietz · 2023-10-04T14:46:56Z

CI failure seems unrelated.

+1, thanks!
I'll merge shortly

conbench-apache-arrow · 2023-10-04T20:27:05Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 02de3c1.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change Improves performance of file reads with an expensive Open operation. ### What changes are included in this PR? ### Are these changes tested? ### Are there any user-facing changes? No * Closes: apache#37917 Authored-by: Eero Lihavainen <[email protected]> Signed-off-by: Benjamin Kietzman <[email protected]>

eeroel requested a review from westonpace as a code owner September 27, 2023 20:32

github-actions bot added Component: C++ awaiting review Awaiting review labels Sep 27, 2023

mapleFU reviewed Sep 28, 2023

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Sep 28, 2023

eeroel force-pushed the feat/s3_async_open branch from c9b99f2 to 7e2d91c Compare September 28, 2023 07:55

mapleFU requested review from bkietz and pitrou September 28, 2023 08:31

mapleFU approved these changes Sep 28, 2023

View reviewed changes

mapleFU reviewed Sep 28, 2023

View reviewed changes

bkietz requested changes Sep 28, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 28, 2023

eeroel mentioned this pull request Sep 30, 2023

Support async CustomOpen in FileSource #37962

Open

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 30, 2023

bkietz approved these changes Oct 2, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Oct 2, 2023

eeroel added 7 commits October 3, 2023 22:16

feat: add openasync for filesource

78df341

add todo

d14ffe8

formatting

b36912e

new version

6ee9a0e

update todo issue

6d4d217

formatting

d90f24d

more formatting

51bad89

eeroel force-pushed the feat/s3_async_open branch from 290ba22 to 51bad89 Compare October 3, 2023 19:16

bkietz merged commit 02de3c1 into apache:main Oct 4, 2023
33 of 34 checks passed

bkietz removed the awaiting merge Awaiting merge label Oct 4, 2023

GH-37917: [Parquet] Add OpenAsync for FileSource #37918

GH-37917: [Parquet] Add OpenAsync for FileSource #37918

Conversation

eeroel commented Sep 27, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Sep 27, 2023

eeroel commented Sep 27, 2023

mapleFU commented Sep 28, 2023

mapleFU left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eeroel commented Sep 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mapleFU commented Sep 29, 2023 • edited Loading

eeroel commented Sep 30, 2023

mapleFU commented Sep 30, 2023

mapleFU commented Oct 2, 2023

eeroel commented Oct 2, 2023

eeroel commented Oct 2, 2023

mapleFU commented Oct 3, 2023

bkietz commented Oct 3, 2023

eeroel commented Oct 3, 2023

bkietz commented Oct 4, 2023

conbench-apache-arrow bot commented Oct 4, 2023

eeroel commented Sep 27, 2023 •

edited by github-actions bot

Loading

eeroel commented Sep 28, 2023 •

edited

Loading

mapleFU commented Sep 29, 2023 •

edited

Loading