Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breaking change with numpy 2.0.0 #206

Closed
isobel-taylor opened this issue Jun 17, 2024 · 2 comments · Fixed by #207
Closed

Breaking change with numpy 2.0.0 #206

isobel-taylor opened this issue Jun 17, 2024 · 2 comments · Fixed by #207

Comments

@isobel-taylor
Copy link

We've identified an issue in our data pipeline caused by the recent release of version 2.0.0 of numpy. The error is thrown by pandas:

2024-06-17T00:07:40.703462Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.703817Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/bin/target-snowflake", line 5, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704162Z [info     ]     from target_snowflake import main cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704395Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/target_snowflake/__init__.py", line 18, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704583Z [info     ]     from target_snowflake.file_formats import parquet cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704823Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/target_snowflake/file_formats/parquet.py", line 3, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.705053Z [info     ]     import pandas              cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709128Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/__init__.py", line 22, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709460Z [info     ]     from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401 cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709691Z [info     ]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709920Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/compat/__init__.py", line 25, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710221Z [info     ]     from pandas.compat.numpy import ( cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710407Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710584Z [info     ]     from pandas.util.version import Version cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710760Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/util/__init__.py", line 2, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710948Z [info     ]     from pandas.util._decorators import (  # noqa:F401 cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711135Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/util/_decorators.py", line 14, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711330Z [info     ]     from pandas._libs.properties import cache_readonly cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711512Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/_libs/__init__.py", line 13, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711709Z [info     ]     from pandas._libs.interval import Interval cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711941Z [info     ]   File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.712244Z [info     ] ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake

Looking online, it seems that this version introduced a breaking change. We've remedied this by forcing numpy to version 1.26.4. We tried upgrading the outer venv to 2.0.0, but this did not fix the issue - it seems like this is a true incompatibility with version 2.0.0.

@edgarrmondragon
Copy link
Member

edgarrmondragon commented Jun 17, 2024

I just released meltanolabs-target-snowflake 0.7.1 which pins numpy to <2 until all issues are fixed upstream :)

Thanks for logging this issue!

@isobel-taylor
Copy link
Author

Awesome, thank you for jumping on this so fast!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants