Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test 1336 #1347

Merged
merged 12 commits into from
Jan 10, 2023
Merged

Test 1336 #1347

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7353eee
Update pyarrow requirement from <8.1.0,>=8.0.0 to >=8.0.0,<10.1.0
dependabot[bot] Nov 1, 2022
ee3a17f
Update pyarrow version
3dbrows Nov 17, 2022
c3ddda9
Update pyarrow version
3dbrows Nov 17, 2022
457d4b5
Update DESCRIPTION.md to include pyarrow version bump
3dbrows Nov 17, 2022
8d56334
Update libarrow version
3dbrows Nov 21, 2022
8da9fcb
C++ ArrowIterator updates
3dbrows Nov 25, 2022
0c4d819
Use C++ 17, add libarrow_dataset
3dbrows Nov 25, 2022
addf41f
Fix Windows lib reference
3dbrows Nov 25, 2022
c22318b
Version floor 10.0.1
3dbrows Nov 25, 2022
a4fd127
Fixed linting error
3dbrows Dec 15, 2022
794ce59
move c++ std to be set on Windows too
sfc-gh-mkeller Jan 6, 2023
684ba3e
fixing merge conflict
sfc-gh-mkeller Jan 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions DESCRIPTION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne

# Release Notes

- v2.9.1(unreleased)

- Bumped pyarrow dependency from >=8.0.0,<8.1.0 to >=10.0.1,<10.1.0

- v2.9.0(December 9, 2022)

- Fixed a bug where the permission of the file downloaded via GET command is changed
Expand Down Expand Up @@ -41,6 +45,8 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
- When closing connection async query status checking is now parallelized
- Fixed a bug where test logging would be enabled on Jenkins workers in non-Snowflake Jenkins machines
- Enhanced the atomicity of write_pandas when overwrite is set to True
- Enhanced the atomicity of write_pandas when overwrite is set to True
- Bumped pyarrow dependency from >=8.0.0,<8.1.0 to >=10.0.0,<10.1.0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those two should be moved under 2.9.1?


- v2.8.0(September 27,2022)

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ requires = [
"wheel",
"cython",
# Must be kept in sync with the `setup_requirements` in `setup.cfg`
"pyarrow>=8.0.0,<8.1.0",
"pyarrow>=10.0.1,<10.1.0",
]

[tool.cibuildwheel]
Expand Down
2 changes: 1 addition & 1 deletion setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,6 @@ development =
pytzdata
pandas =
pandas>=1.0.0,<1.6.0
pyarrow>=8.0.0,<8.1.0
pyarrow>=10.0.1,<10.1.0
secure-local-storage =
keyring!=16.1.0,<24.0.0
50 changes: 38 additions & 12 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,23 +71,45 @@ class MyBuildExt(build_ext):
# this list should be carefully examined when pyarrow lib is
# upgraded
arrow_libs_to_copy = {
"linux": ["libarrow.so.800", "libarrow_python.so.800", "libparquet.so.800"],
"linux": [
"libarrow.so.1000",
"libarrow_dataset.so.1000",
sfc-gh-aling marked this conversation as resolved.
Show resolved Hide resolved
"libarrow_python.so.1000",
"libparquet.so.1000",
],
"darwin": [
"libarrow.800.dylib",
"libarrow_python.800.dylib",
"libparquet.800.dylib",
"libarrow.1000.dylib",
"libarrow_dataset.1000.dylib",
"libarrow_python.1000.dylib",
"libparquet.1000.dylib",
],
"win32": [
"arrow.dll",
"arrow_dataset.dll",
"arrow_python.dll",
"parquet.dll",
],
"win32": ["arrow.dll", "arrow_python.dll", "parquet.dll"],
}

arrow_libs_to_link = {
"linux": ["libarrow.so.800", "libarrow_python.so.800", "libparquet.so.800"],
"linux": [
"libarrow.so.1000",
"libarrow_dataset.so.1000",
"libarrow_python.so.1000",
"libparquet.so.1000",
],
"darwin": [
"libarrow.800.dylib",
"libarrow_python.800.dylib",
"libparquet.800.dylib",
"libarrow.1000.dylib",
"libarrow_dataset.1000.dylib",
"libarrow_python.1000.dylib",
"libparquet.1000.dylib",
],
"win32": [
"arrow.lib",
"arrow_dataset.lib",
"arrow_python.lib",
"parquet.lib",
],
"win32": ["arrow.lib", "arrow_python.lib", "parquet.lib"],
}

def build_extension(self, ext):
Expand Down Expand Up @@ -126,13 +148,15 @@ def build_extension(self, ext):
ext.include_dirs.append(LOGGING_SRC_DIR)

if sys.platform == "win32":
if not any("/std" not in s for s in ext.extra_compile_args):
ext.extra_compile_args.append("/std:c++17")
Comment on lines +151 to +152
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the c++ standard has to be set on Windows now too @xhochy and @3dbrows

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Arrow 10 switched to C++17 as the minimal supported C++ standard.

ext.include_dirs.append(pyarrow.get_include())
ext.include_dirs.append(numpy.get_include())
elif sys.platform == "linux" or sys.platform == "darwin":
ext.extra_compile_args.append("-isystem" + pyarrow.get_include())
ext.extra_compile_args.append("-isystem" + numpy.get_include())
if "std=" not in os.environ.get("CXXFLAGS", ""):
ext.extra_compile_args.append("-std=c++11")
ext.extra_compile_args.append("-std=c++17")
ext.extra_compile_args.append("-D_GLIBCXX_USE_CXX11_ABI=0")

ext.library_dirs.append(
Expand Down Expand Up @@ -160,9 +184,11 @@ def _get_arrow_lib_dir(self):
def _copy_arrow_lib(self):
libs_to_bundle = self.arrow_libs_to_copy[sys.platform]

build_dir = os.path.join(self.build_lib, "snowflake", "connector")
os.makedirs(build_dir, exist_ok=True)

for lib in libs_to_bundle:
source = f"{self._get_arrow_lib_dir()}/{lib}"
build_dir = os.path.join(self.build_lib, "snowflake", "connector")
copy(source, build_dir)

def _get_arrow_lib_as_linker_input(self):
Expand Down
2 changes: 1 addition & 1 deletion src/snowflake/connector/cpp/ArrowIterator/BinaryConverter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ PyObject* BinaryConverter::toPyObject(int64_t rowIndex) const
{
if (m_array->IsValid(rowIndex))
{
arrow::util::string_view sv = m_array->GetView(rowIndex);
std::string_view sv = m_array->GetView(rowIndex);
sfc-gh-aling marked this conversation as resolved.
Show resolved Hide resolved
return PyByteArray_FromStringAndSize(sv.data(), sv.size());
}
else
Expand Down
2 changes: 1 addition & 1 deletion src/snowflake/connector/cpp/ArrowIterator/StringConverter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ PyObject* StringConverter::toPyObject(int64_t rowIndex) const
{
if (m_array->IsValid(rowIndex))
{
arrow::util::string_view sv = m_array->GetView(rowIndex);
std::string_view sv = m_array->GetView(rowIndex);
return PyUnicode_FromStringAndSize(sv.data(), sv.size());
}
else
Expand Down