Updating inference logic to add node level request-response logging

[SachinVarghese]

What this PR does / why we need it:
This PR corrects the paylod logging mechanism in seldon executor for logging node level request-response pairs in complex inference graphs
Which issue(s) this PR fixes:
Fixes #3873

[seldondev]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign sachinvarghese
You can assign the PR to them by writing /assign @sachinvarghese in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[michaelcheah]

I'm not a fan of logging inside transformInput. It is not clear from the Predict function where transformInput is called that the message may or may not be logged.

edit; unfortunately, due to the way the code has been structured, this is the easiest way to do what's needed because the callTransformInput and callModel flags are determined in here.

[SachinVarghese]

Yeah I also tried structuring the code in a different manner to start with but this seemed easiest considering other aspects of the implementation.

[ukclivecox]

/test integration

[ukclivecox]

/test notebooks

[RafalSkolasinski]

/test integration

[RafalSkolasinski]

logs of integration tests are quite confusing, it is not clear if there was an actual failure or not

===Flaky Test Report===

test_namespace_update[1.10.0] passed 1 out of the required 1 times. Success!
test_namespace_update[1.11.0] passed 1 out of the required 1 times. Success!
test_namespace_update[1.12.0] passed 1 out of the required 1 times. Success!
test_rolling_update6[ambas] passed 1 out of the required 1 times. Success!
test_rolling_update6[istio] passed 1 out of the required 1 times. Success!
test_rolling_update7[ambas] passed 1 out of the required 1 times. Success!
test_rolling_update7[istio] passed 1 out of the required 1 times. Success!
test_rolling_update8[ambas] failed (4 runs remaining out of 5).
	<class 'AssertionError'>
	assert 503 == 200
  -503
  +200
	[<TracebackEntry /workspace/source/testing/scripts/test_rolling_updates.py:140>]
test_rolling_update8[ambas] failed (3 runs remaining out of 5).
	<class 'AssertionError'>
	assert 503 == 200
  -503
  +200
	[<TracebackEntry /workspace/source/testing/scripts/test_rolling_updates.py:140>]
test_rolling_update8[ambas] failed (2 runs remaining out of 5).
	<class 'AssertionError'>
	assert 500 == 200
  -500
  +200
	[<TracebackEntry /workspace/source/testing/scripts/test_rolling_updates.py:140>]
test_rolling_update8[ambas] passed 1 out of the required 1 times. Success!
test_rolling_update8[istio] passed 1 out of the required 1 times. Success!
test_rolling_update9[ambas] passed 1 out of the required 1 times. Success!
test_rolling_update9[istio] passed 1 out of the required 1 times. Success!

===End Flaky Test Report===

========== 52 passed, 6 skipped, 67 deselected in 6307.24s (1:45:07) ===========
Test returned errors
kind delete cluster
Deleting cluster "kind" ...
Stopping Docker: dockerProgram process in pidfile '/var/run/docker-ssd.pid', 1 process(es), refused to die.
�[31m
Pipeline failed on stage 'integration-test-task' : container 'step-integration-step'. The execution of the pipeline has stopped.�[0m

if it was, it could have been on test_rolling_update8[ambas] failed but why it does not count towards total number in the summary?

[RafalSkolasinski]

Actually, I also see failure on TestPrepack.test_mlflow_v2.

Triggered tests again anyway to rule out flakiness first

[RafalSkolasinski]

/test integration

[RafalSkolasinski]

The TestPrepack.test_mlflow_v2 seem to passed I do see failure now on test_rolling_deployment[graph1.json-graph4.json-True-ambas].