Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perspective reading LZ4 compressed RecordBatches. #2337

Closed

Conversation

sinistersnare
Copy link
Contributor

Now you can drop a file created by

import pandas as pd
import pyarrow.feather as feather
data = [['a', 10], ['b', 15], ['c', 14]]
df = pd.DataFrame(data, columns=['Name', 'Age'])
# This defaults to compressing with LZ4.
feather.write_feather(df, 'lz4_compressed_data.feather')

@sinistersnare sinistersnare force-pushed the feature/lz4-feather branch 3 times, most recently from bc8898d to 8628390 Compare August 10, 2023 18:50
Copy link
Member

@texodus texodus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! A few additional issues to the comments below:

  • Doesn't compile on Windows.
  • Needs tests.

Closing this in favor of #2339 which addresses these issues and adds write support, benchmarks and documentation.

set(ARROW_SRCS
${ARROW_SRCS}
# Use our vendored reader that does not use threads.
${PSP_CPP_SRC}/src/cpp/vendor/single_threaded_reader.cpp)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should IMO move the arrow_single_threaded_reader.cpp files here also from cpp/perspective/CMakeLists.txt

@@ -0,0 +1,46 @@
# Licensed from VTK under BSD 3 clause License
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this file - there is no corresponding call to find_package, also this library is explicitly built statically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants