Error: google.api_core.exceptions.Unknown: None Stream removed

[EsbernTK]

I am analyzing a large amount of individual video files through a python script using this API, most of the time it works well but every so often it gives this error. This happens about 1-2 times per 10 files. When i re-analyze the failed files, many of them succeed, so i know it is not a codec error or something like that.

Environment details

• OS type and version: windows 10
• Python version: 3.6
• pip version: 19.0.3
• google-cloud-videointelligence version: 1.14.0
• google-api-core version: 1.17.0
• grpc version: 1.29.0

Stack trace

Traceback (most recent call last):
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\grpc\_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\grpc\_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Stream removed"
	debug_error_string = "{"created":"@1596454420.367000000","description":"Error received from peer ipv4:216.58.211.10:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Stream removed","grpc_status":2}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-8bdb6b8826b4>", line 1, in <module>
    runfile('C:/Users/mads/Documents/Kaspar/kaspar-notebooks/src/util/model_utils/model_runner.py', wdir='C:/Users/mads/Documents/Kaspar/kaspar-notebooks/src/util/model_utils')
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.3.2\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.3.2\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/mads/Documents/Kaspar/kaspar-notebooks/src/util/model_utils/model_runner.py", line 158, in <module>
    runner.run_model_on_films(["**/videos/other-projects/emil_kids/*"])
  File "C:/Users/mads/Documents/Kaspar/kaspar-notebooks/src/util/model_utils/model_runner.py", line 142, in run_model_on_films
    self.run_model(file, save_path)
  File "C:/Users/mads/Documents/Kaspar/kaspar-notebooks/src/util/model_utils/model_runner.py", line 129, in run_model
    self.manager.annotate_cloud_video(input_file,save_path)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\src\util\cloud_utils\google_cloud_api_managers.py", line 247, in annotate_cloud_video
    result = job.result()
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\future\polling.py", line 125, in result
    self._blocking_poll(timeout=timeout)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\future\polling.py", line 104, in _blocking_poll
    retry_(self._done_or_raise)()
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\retry.py", line 184, in retry_target
    return target()
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\future\polling.py", line 83, in _done_or_raise
    if not self.done():
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\operation.py", line 169, in done
    self._refresh_and_update(retry)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\operation.py", line 157, in _refresh_and_update
    self._operation = self._refresh(retry=retry)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\operations_v1\operations_client.py", line 131, in get_operation
    return self._get_operation(request, retry=retry, timeout=timeout)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\gapic_v1\method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\retry.py", line 184, in retry_target
    return target()
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "C:\Users\mads\Documents\Kaspar\kaspar-notebooks\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.Unknown: None Stream 

[telpirion]

Hello @EsbernTK , thank you for contacting us.

Some questions and suggestions:

• Would you be willing to provide some code?
• Are you using the streaming API or the long-running API?
• If you are using the long-running API, you may need to adjust your retries and timeouts. I've found that shorter timeouts but more retries tends to be more successful.

[EsbernTK]

@telpirion
I am using the version that is installed with
pip install google-cloud-videointelligence
I have assumed that is the long-running API when implementing.

The code is this:

from google.cloud import videointelligence
from google.cloud.videointelligence import enums
from google.protobuf.json_format import MessageToJson
import json
def annotate_cloud_video(video_path: str,save_path: str,bucket_path: str, features: list = None):
	client = videointelligence.VideoIntelligenceServiceClient()
	if(features is None):
		features = [enums.Feature.LABEL_DETECTION,
                            enums.Feature.SHOT_CHANGE_DETECTION,
                            enums.Feature.FACE_DETECTION, 
                            enums.Feature.OBJECT_TRACKING,
                            enums.Feature.SPEECH_TRANSCRIPTION, 
                            enums.Feature.LOGO_RECOGNITION,
                            enums.Feature.TEXT_DETECTION]
	path  = bucket_path+video_path
	job = client.annotate_video(
		input_uri=path,
		features=features
	)
	result = job.result()
	serialized = MessageToJson(result)
	json_result = json.loads(serialized)
	with open(str(save_path), 'w') as outfile:
		json.dump(json_result, outfile)
	return json_result

As you can see i have used the default timeout and retries settings, though the error seems to be very weird if it is caused by a timeout.
I will try with some other timeout and retry settings, though it would be nice to have some default settings that worked more often, and some error messages that makes it easier to trace the problem back to those settings.

[telpirion]

@busunkim96 Do you have any recommendations from a retry / timeout perspective?

@EsbernTK You might try increasing the timeout period and/or adding more retries. See the documentation here for details. I will try to come up with a code snippet for you, too.

[busunkim96]

Apparently the same message (Stream removed) pops up for Pub/Sub in both Python and Node. (I also see an internal question about Pub/Sub in C#). It might be worth reaching to the Pub/Sub folks.

• 1.17.0 breaking pubsub python-api-core#25 (comment)
• Stream: Removed nodejs-pubsub#892

grpc/grpc#23277 also seems similar to this one (intermittent failure) but there are other issues in gRPC related to "stream removed" that you can look into. https://github.com/grpc/grpc/search?p=1&q=%22stream+removed%22&type=issues

[munkhuushmgl]

Due to inactivity, I am closing this issue.

[hminle]

I've met the same issue, does anyone know how to solve it?
Thanks

[michael-wise]

Just received the error while using the google library language_v1.LanguageServiceClient.analyze_sentiment
(aware this is not python-videointelligence. But it might give a hint of what's causing the exception in core.)

[dtvn-toannong]

I've met the same issue, does anyone know how to solve it?
Thanks very much

[ScorIsAFox]

I've met the same issue, does anyone know how to solve it? It happens a lot.
Thanks very much