Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use object dtype for ARRAY columns in to_pandas() with pandas 1.x #329

Merged
merged 7 commits into from
Jan 22, 2024

Conversation

tswast
Copy link
Collaborator

@tswast tswast commented Jan 18, 2024

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes internal issue 321013333
🦕

@tswast tswast requested review from a team as code owners January 18, 2024 22:04
@tswast tswast requested a review from shobsi January 18, 2024 22:04
@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Jan 18, 2024
bigframes/session/_io/pandas.py Outdated Show resolved Hide resolved
tests/system/small/ml/test_preprocessing.py Outdated Show resolved Hide resolved
@tswast tswast requested a review from shobsi January 19, 2024 21:41
@tswast tswast added automerge Merge the pull request once unit tests and other checks pass. owlbot:run Add this label to trigger the Owlbot post processor. labels Jan 22, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 22, 2024
@tswast
Copy link
Collaborator Author

tswast commented Jan 22, 2024

Doctest failed with

2631             >>> @bpd.remote_function([str], bool, reuse=False)
UNEXPECTED EXCEPTION: InternalServerError('Could not create Cloud Run service bigframes-597cc02ef5ce0525e4f51697b5a83b6c-32er4jqu. ')
Traceback (most recent call last):
  File "/tmpfs/src/github/python-bigquery-dataframes/.nox/doctest/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 79, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmpfs/src/github/python-bigquery-dataframes/.nox/doctest/lib/python3.11/site-packages/grpc/_channel.py", line 1160, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmpfs/src/github/python-bigquery-dataframes/.nox/doctest/lib/python3.11/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Could not create Cloud Run service bigframes-597cc02ef5ce0525e4f51697b5a83b6c-32er4jqu. "
	debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.126.95:443 {grpc_message:"Could not create Cloud Run service bigframes-597cc02ef5ce0525e4f51697b5a83b6c-32er4jqu. ", grpc_status:13, created_time:"2024-01-22T17:51:55.118019527+00:00"}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/doctest.py", line 1351, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest series.Series.mask[7]>", line 1, in <module>
  File "/tmpfs/src/github/python-bigquery-dataframes/bigframes/functions/remote_function.py", line 851, in wrapper
    rf_name, cf_name = remote_function_client.provision_bq_remote_function(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmpfs/src/github/python-bigquery-dataframes/bigframes/functions/remote_function.py", line 472, in provision_bq_remote_function
    cf_endpoint = self.create_cloud_function(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmpfs/src/github/python-bigquery-dataframes/bigframes/functions/remote_function.py", line 415, in create_cloud_function
    operation = self._cloud_functions_client.create_function(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmpfs/src/github/python-bigquery-dataframes/.nox/doctest/lib/python3.11/site-packages/google/cloud/functions_v2/services/function_service/client.py", line 1007, in create_function
    response = rpc(
               ^^^^
  File "/tmpfs/src/github/python-bigquery-dataframes/.nox/doctest/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmpfs/src/github/python-bigquery-dataframes/.nox/doctest/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 81, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InternalServerError: 500 Could not create Cloud Run service bigframes-597cc02ef5ce0525e4f51697b5a83b6c-32er4jqu. 
/[tmpfs/src/github/python-bigquery-dataframes/third_party/bigframes_vendored/pandas/core/series.py:2631](https://cs.corp.google.com/piper///depot/google3/tmpfs/src/github/python-bigquery-dataframes/third_party/bigframes_vendored/pandas/core/series.py?l=2631): UnexpectedException
...
FAILED third_party/bigframes_vendored/pandas/core/frame.py::frame.DataFrame.map
FAILED third_party/bigframes_vendored/pandas/core/series.py::series.Series.apply
FAILED third_party/bigframes_vendored/pandas/core/series.py::series.Series.mask

As these are unrelated and remote_function is notoriously flaky, I'll override this check for now.

@tswast tswast merged commit 374ddb5 into main Jan 22, 2024
13 of 15 checks passed
@tswast tswast deleted the issue321013333-array-io branch January 22, 2024 18:05
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: xl Pull request size is extra large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants