feat(grpc): refine error message on deadline exceeded

[rexledesma]

Summary & Motivation

DEADLINE_EXCEEDED usually indicates that the user defined compute function for schedules/sensors is too slow. Rather than just giving the error, actually give actionable steps for the user to try in order to optimize their code to avoid the error.

References:

• https://stackoverflow.com/questions/55077340/how-to-send-response-message-with-non-ok-status-from-a-python-service-behind-grp

How I Tested These Changes

TODO: pytest

local, after overriding an rpc method to return DEADLINE_EXCEEDED:

internal-3.8.7 ❯ dagit -f test.py
Using temporary directory /Users/rexledesma/elementl/dbt_testing/tmp_wnvfpf1 for storage. This will be removed when dagit exits.
To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.

0it [00:00, ?it/s]
0it [00:00, ?it/s]
2022-12-20 20:41:38 -0500 - dagster - INFO - Started Dagster code server for file test.py in process 69431
/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_core/workspace/context.py:548: UserWarning: Error loading repository location test.py:dagster._core.errors.DagsterUserCodeUnreachableError: Your code location server took longer to respond than the 60 second timeout. One reason this might happen is if you are running a schedule or sensor that is taking a long time to evaluate its tick.

You can fix this error by optimizing your evaluation function to be faster than the 60 second timeout, or by setting the environment variable `DAGSTER_GRPC_TIMEOUT_SECONDS` to be a value larger than 60.

Stack Trace:
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_core/workspace/context.py", line 545, in _load_location
    location = self._create_location_from_origin(origin)
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_core/workspace/context.py", line 478, in _create_location_from_origin
    return GrpcServerRepositoryLocation(
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_core/host_representation/repository_location.py", line 573, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_api/list_repositories.py", line 19, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_grpc/client.py", line 223, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_grpc/client.py", line 142, in _query
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = ""
	debug_error_string = "{"created":"@1671586898.933942000","description":"Error received from peer unix:/var/folders/yb/qqcnzws17_v917wn0t26nsfw0000gn/T/tmpt32wvnf1","file":"src/core/lib/surface/call.cc","file_line":953,"grpc_message":"","grpc_status":4}"
>

Stack Trace:
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_grpc/client.py", line 139, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File "/Users/rexledesma/elementl/dagster/python_modules/dagster/dagster/_grpc/client.py", line 129, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/Users/rexledesma/.pyenv/versions/internal-3.8.7/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/rexledesma/.pyenv/versions/internal-3.8.7/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)

  warnings.warn(
2022-12-20 20:41:39 -0500 - dagit - INFO - Serving dagit on http://127.0.0.1:3000 in process 69417
2022-12-20 20:42:24 -0500 - dagster - INFO - Shutting down Dagster code server for file test.py in process 69431

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Ignored Deployments

Name	Status	Preview	Comments	Updated
dagit-storybook	⬜️ Ignored (Inspect)			Dec 21, 2022 at 1:49AM (UTC)
dagster	⬜️ Ignored (Inspect)			Dec 21, 2022 at 1:49AM (UTC)

[rexledesma]

Current dependencies on/for this PR:

• master
  • PR feat(grpc): refine error message on deadline exceeded #11305 👈

This comment was auto-generated by Graphite.

[rexledesma]

The correct way to actually do this, referencing a sample implementation from googleapis/python-api-core#286

• We need some wrapper exception around grpc calls
• The error details should be emitted using the Status Proto (from the grpc_status extension: https://github.com/grpc/proposal/blob/master/L44-python-rich-status.md)

[rexledesma]

#11576 is a better take on this