[IE PYTHON] wait for infer callback to complete #7374

AlexeyLebedev1 · 2021-09-05T16:41:37Z

Tickets:

63914

nkogteva · 2021-09-06T15:09:14Z

inference-engine/ie_bridges/python/tests/test_InferRequest.py

@@ -350,7 +351,7 @@ def execute(self, input_data):
            self.cv.release()
            status = self.request.wait(ie.WaitMode.RESULT_READY)
            assert status == ie.StatusCode.OK
-            assert self.status_code == ie.StatusCode.OK
+            assert self.status_code == ie.StatusCode.RESULT_NOT_READY


It looks strange from API level perspective. User calls wait with wait mode == RESULT_READY, but receives RESULT_NOT_READY

I am curious about it, why we get RESULT_NOT_READY? I understand the idea behind it is to capture status inside callback which will be equal to RESULT_NOT_READY, and only after returning to python we can obtain OK. However this could render previous callbacks' "policy" absolutely obsolete. The behavior was even present around March of the current year:

openvino/inference-engine/ie_bridges/python/sample/classification_sample_async/classification_sample_async.py

Lines 24 to 29 in 6478f17

def callback(self, statusCode, userdata):

if (userdata != self.id):

log.error(f"Request ID {self.id} does not correspond to user data {userdata}")

elif statusCode != 0:

log.error(f"Request {self.id} failed with status code {statusCode}")

self.cur_iter += 1

I totally understand an idea to be able to call wait inside callback itself. However why we are still passing "the same" status code as a parameter? To my knowledge this statusCode would be equal to OK as cpp implementation marks it this way as we are able to enter a callback and not to confuse users with different names of callback's exit codes. The parameter is here:

openvino/inference-engine/ie_bridges/python/tests/test_InferRequest.py

Line 341 in 809bb8a

def callback(self, statusCode, userdata):

This needs a follow-up (in my opinion) as it requires:

either reconsidering a behavior of the callbacks in python and picking of the one "true" way of obtaining the status inside the callback

or changing c++ core code to apply aligned behavior in both APIs.

@nkogteva @akuporos what are your views on that?
@AlexeyLebedev1 could you elaborate more on this change?

@jiwaszki From my point of view, we can't disallow the user to call wait inside callback and vice versa we should not force to do it just to check the status code. I agree that it is strange if status codes received in these two ways would be different. At least it should be documented.

But anyway the behavior should be aligned with C++. cc @ilya-lavrenov

The main point of these changes that InferRequest.wait(RESULT_READY) must block the thread until infer_callback function completes, as in c++ api. When we call wait inside callback with no-wait status, we get RESULT_NOT_READY, also like in c++ api and it can confuse user, i agree, but if we want to wait for infer_callback maybe it makes sense.

One more note: as far as I know user can access output blobs inside callback like in demo pipeline. Thus it makes sense to provide the correct status code which informs of present state - which is OK. @ilya-lavrenov is it possible to obtain results in C++ API's callbacks as well? If yes, we should consider changes to C++ API to return OK in callback or some new status that is clearer about output blobs status instead of focusing on InferRequest's "life".

If we are going to align code to C++ without any changes then it should be properly documented and reflected on the python side.

How i knew from @ilya-lavrenov , c++ api supposses that callback is the last stage of inference, that's why InferRequest.wait(0) inside callback returns RESULT_NOT_READY. I don't see why we should worry about this status code

openvino/inference-engine/ie_bridges/python/tests/test_InferRequest.py

Line 341 in f79d649

self.status_code = self.request.wait(ie.WaitMode.STATUS_ONLY)

, if we already got status code, as an argument,

openvino/inference-engine/ie_bridges/python/tests/test_InferRequest.py

Line 340 in f79d649

def callback(self, statusCode, userdata):

, which is OK, otherway we would get an exception

openvino/inference-engine/ie_bridges/python/src/openvino/inference_engine/ie_api_impl.cpp

Lines 538 to 545 in f79d649

if (code != InferenceEngine::StatusCode::OK) {

IE_EXCEPTION_SWITCH(code,

ExceptionType,

InferenceEngine::details::ThrowNow<ExceptionType>{} <<=

std::stringstream{}

<< IE_LOCATION

<< InferenceEngine::details::ExceptionTraits<ExceptionType>::string());

}

, so we can safety access output blobs inside callback.

inference-engine/ie_bridges/python/src/openvino/inference_engine/ie_api.pyx

jiwaszki

I love the concept of the change however it needs some refining. Please also look at this:

openvino/inference-engine/ie_bridges/python/sample/classification_sample_async/classification_sample_async.py

Lines 123 to 133 in 9e68a67

    
           for i in output_queue: 
        
               # Immediately returns a inference status without blocking or interrupting 
        
               infer_status = exec_net.requests[i].wait(0) 
        
               if infer_status == StatusCode.RESULT_NOT_READY: 
        
                   continue 
        
               log.info(f'Infer request {i} returned {infer_status}') 
        
               if infer_status != StatusCode.OK: 
        
                   return -2

There can be a lot more places where this behavior occurs - please look through the project where the similar "no-wait" call can occur, also with a usage of ExecutableNetwork since it is using the same mechanisms underneath.

jiwaszki · 2021-09-07T07:48:31Z

inference-engine/ie_bridges/python/tests/test_InferRequest.py

@@ -350,7 +351,7 @@ def execute(self, input_data):
            self.cv.release()
            status = self.request.wait(ie.WaitMode.RESULT_READY)
            assert status == ie.StatusCode.OK
-            assert self.status_code == ie.StatusCode.OK
+            assert self.status_code == ie.StatusCode.RESULT_NOT_READY


I am curious about it, why we get RESULT_NOT_READY? I understand the idea behind it is to capture status inside callback which will be equal to RESULT_NOT_READY, and only after returning to python we can obtain OK. However this could render previous callbacks' "policy" absolutely obsolete. The behavior was even present around March of the current year:

openvino/inference-engine/ie_bridges/python/sample/classification_sample_async/classification_sample_async.py

Lines 24 to 29 in 6478f17

def callback(self, statusCode, userdata):

if (userdata != self.id):

log.error(f"Request ID {self.id} does not correspond to user data {userdata}")

elif statusCode != 0:

log.error(f"Request {self.id} failed with status code {statusCode}")

self.cur_iter += 1

I totally understand an idea to be able to call wait inside callback itself. However why we are still passing "the same" status code as a parameter? To my knowledge this statusCode would be equal to OK as cpp implementation marks it this way as we are able to enter a callback and not to confuse users with different names of callback's exit codes. The parameter is here:

openvino/inference-engine/ie_bridges/python/tests/test_InferRequest.py

Line 341 in 809bb8a

def callback(self, statusCode, userdata):

This needs a follow-up (in my opinion) as it requires:

either reconsidering a behavior of the callbacks in python and picking of the one "true" way of obtaining the status inside the callback

or changing c++ core code to apply aligned behavior in both APIs.

@nkogteva @akuporos what are your views on that?
@AlexeyLebedev1 could you elaborate more on this change?

inference-engine/ie_bridges/python/tests/test_InferRequest.py

AlexeyLebedev1 · 2021-09-07T09:00:24Z

I love the concept of the change however it needs some refining. Please also look at this:

openvino/inference-engine/ie_bridges/python/sample/classification_sample_async/classification_sample_async.py

Lines 123 to 133 in 9e68a67

for i in output_queue:

# Immediately returns a inference status without blocking or interrupting

infer_status = exec_net.requests[i].wait(0)

if infer_status == StatusCode.RESULT_NOT_READY:

continue

log.info(f'Infer request {i} returned {infer_status}')

if infer_status != StatusCode.OK:

return -2

There can be a lot more places where this behavior occurs - please look through the project where the similar "no-wait" call can occur, also with a usage of ExecutableNetwork since it is using the same mechanisms underneath.

Thanks, i also will update ExecutableNetwork.wait()

…n/infer_callback

please, review once again

* test commit * Add test * Remove threading.event from InferRequest * Fix wait for ExecutableNetwork

Alexey Lebedev added 2 commits September 5, 2021 19:21

test commit

ba2a45d

Add test

fa5798a

AlexeyLebedev1 added category: Python API OpenVINO Python bindings do not merge labels Sep 5, 2021

AlexeyLebedev1 requested review from a team and akuporos September 5, 2021 16:41

AlexeyLebedev1 removed the do not merge label Sep 6, 2021

Remove threading.event from InferRequest

809bb8a

akuporos requested review from nkogteva and jiwaszki September 6, 2021 14:29

akuporos assigned nkogteva Sep 6, 2021

akuporos added this to the 2022.1 milestone Sep 6, 2021

nkogteva reviewed Sep 6, 2021

View reviewed changes

inference-engine/ie_bridges/python/src/openvino/inference_engine/ie_api.pyx Show resolved Hide resolved

nkogteva approved these changes Sep 6, 2021

View reviewed changes

jiwaszki previously requested changes Sep 7, 2021

View reviewed changes

Alexey Lebedev added 3 commits September 7, 2021 12:05

Merge remote-tracking branch 'upstream/master' into alexeyl1/ie_pytho…

159e8e1

…n/infer_callback

Fix wait for ExecutableNetwork

03f643d

Resolve conflicts

da642cd

AlexeyLebedev1 changed the title ~~[IE PYTHON] wait while infer callback will not finish~~ [IE PYTHON] wait for infer callback completes Sep 8, 2021

AlexeyLebedev1 changed the title ~~[IE PYTHON] wait for infer callback completes~~ [IE PYTHON] wait for infer callback to complete Sep 8, 2021

Resolve conflicts

f79d649

AlexeyLebedev1 requested a review from jiwaszki September 13, 2021 15:17

akuporos approved these changes Sep 15, 2021

View reviewed changes

akuporos merged commit 6df94af into openvinotoolkit:master Sep 15, 2021

akuporos pushed a commit to akuporos/openvino that referenced this pull request Sep 29, 2021

[IE PYTHON] wait for infer callback to complete (openvinotoolkit#7374)

23d5371

* test commit * Add test * Remove threading.event from InferRequest * Fix wait for ExecutableNetwork

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IE PYTHON] wait for infer callback to complete #7374

[IE PYTHON] wait for infer callback to complete #7374

AlexeyLebedev1 commented Sep 5, 2021

nkogteva Sep 6, 2021

jiwaszki Sep 7, 2021

nkogteva Sep 7, 2021

AlexeyLebedev1 Sep 7, 2021

jiwaszki Sep 7, 2021

AlexeyLebedev1 Sep 13, 2021

jiwaszki left a comment

jiwaszki Sep 7, 2021

AlexeyLebedev1 commented Sep 7, 2021

	def callback(self, statusCode, userdata):
	if (userdata != self.id):
	log.error(f"Request ID {self.id} does not correspond to user data {userdata}")
	elif statusCode != 0:
	log.error(f"Request {self.id} failed with status code {statusCode}")
	self.cur_iter += 1

	if (code != InferenceEngine::StatusCode::OK) {
	IE_EXCEPTION_SWITCH(code,
	ExceptionType,
	InferenceEngine::details::ThrowNow<ExceptionType>{} <<=
	std::stringstream{}
	<< IE_LOCATION
	<< InferenceEngine::details::ExceptionTraits<ExceptionType>::string());
	}

	for i in output_queue:
	# Immediately returns a inference status without blocking or interrupting
	infer_status = exec_net.requests[i].wait(0)

	if infer_status == StatusCode.RESULT_NOT_READY:
	continue

	log.info(f'Infer request {i} returned {infer_status}')

	if infer_status != StatusCode.OK:
	return -2

[IE PYTHON] wait for infer callback to complete #7374

[IE PYTHON] wait for infer callback to complete #7374

Conversation

AlexeyLebedev1 commented Sep 5, 2021

Tickets:

nkogteva Sep 6, 2021

Choose a reason for hiding this comment

jiwaszki Sep 7, 2021

Choose a reason for hiding this comment

nkogteva Sep 7, 2021

Choose a reason for hiding this comment

AlexeyLebedev1 Sep 7, 2021

Choose a reason for hiding this comment

jiwaszki Sep 7, 2021

Choose a reason for hiding this comment

AlexeyLebedev1 Sep 13, 2021

Choose a reason for hiding this comment

jiwaszki left a comment

Choose a reason for hiding this comment

jiwaszki Sep 7, 2021

Choose a reason for hiding this comment

AlexeyLebedev1 commented Sep 7, 2021