executor: tags do not propagate through inference graph

[RafalSkolasinski]

For example, same requests with engine

curl -X POST -H 'Content-Type: application/json' \
        -d '{"data": {"names": ["get"], "ndarray": ["data"]}}' \
        http://localhost:8003/seldon/seldon/step-experiments/api/v0.1/predictions
{
  "meta": {
    "puid": "qgrk9e6dlvpcdcgu0ip539afgt",
    "tags": {
      "current": "three",
      "one": "yes",
      "value_at_one": ["data"],
      "two": "yes",
      "value_at_two": ["one"],
      "three": "yes",
      "value_at_three": ["two"]
    },
    "routing": {
      "step-one": -1,
      "step-two": -1
    },
    "requestPath": {
      "step-three": "seldon-experiments:step-3",
      "step-one": "seldon-experiments:step-1",
      "step-two": "seldon-experiments:step-2"
    },
    "metrics": []
  },
  "data": {
    "names": [],
    "ndarray": ["two"]
  }

and with executor

curl -X POST -H 'Content-Type: application/json' \
        -d '{"data": {"names": ["get"], "ndarray": ["data"]}}' \
        http://localhost:8003/seldon/seldon/step-experiments/api/v0.1/predictions
{"data":{"names":[],"ndarray":["two"]},"meta":{"tags":{"current":"three","three":"yes","value_at_three":["two"]}}}

both performed using 1.0.3-SNAPSHOT and for following seldon deployment:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: step-experiments
spec:
  name: three-steps
  annotations:
    seldon.io/executor: "true"
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldon-experiments:step-1
          name: step-one
          imagePullPolicy: IfNotPresent
          env:
          - name: SELDON_LOG_LEVEL
            value: DEBUG
        - image: seldon-experiments:step-2
          name: step-two
          imagePullPolicy: IfNotPresent
          env:
          - name: SELDON_LOG_LEVEL
            value: DEBUG
        - image: seldon-experiments:step-3
          name: step-three
          imagePullPolicy: IfNotPresent
          env:
          - name: SELDON_LOG_LEVEL
            value: DEBUG
    graph:
      children:
      - children:
        - endpoint:
            type: REST
          name: step-three
          type: MODEL
        endpoint:
          type: REST
        name: step-two
        type: MODEL
      endpoint:
        type: REST
      name: step-one
      type: MODEL
    name: steps-graph
    replicas: 1

(see models here)

Issue related to #1362 in order to provide backwards compatibility with behaviour present with engine.

[ukclivecox]

The current executor will pass the payloads through it won't modify any payloads. I assume this is why you just see the response meta from the final model.
If you want to keep metadata its up to each model to copy from request to response.

[RafalSkolasinski]

The question is how much we need to preserve functionality provided by .tags method in order to not introduce backward incompatible changes to our users.

For just context of request metadata I am just about open another issue that tackle this specific metadata aspect.

[RafalSkolasinski]

Just to define a behaviour we want to preserve I suggest to consider minimal scenario:

• graph with few steps as above
• model in each step has tags() method define
• each tags() method takes no arguments (except self)
• each tags() method output must be present in meta.tags of the final response (unless it is overwritten by later steps)

The means by which it happens should not be a concern of the user.

Before we move handling of metadata to headers (not sure if we gonna do it even at this point) the fix for that could happen at the python wrapper / microservice level.

I really wouldn't want require from users to define

def predict(self, data, names, meta):
    self._meta = meta

as this will just become ugly boilerplate.

[RafalSkolasinski]

Should not forget to verify if graphs with combiner will behave also properly.

[RafalSkolasinski]

#1481 Seems to have right now a working solution to the problem.
Here is a request with engine

curl -X POST -H 'Content-Type: application/json' \
        -d '{"meta": {"some": "input meta"}, "data": {"names": ["get"], "ndarray": ["data"]}}' \
        http://localhost:8003/seldon/seldon/step-experiments/api/v0.1/predictions | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   658  100   577  100    81  10122   1421 --:--:-- --:--:-- --:--:-- 11543
{
  "meta": {
    "puid": "lruk80c6r9f4m3r2fqt61r0ns1",
    "tags": {
      "current": "three",
      "one": "yes",
      "value_at_one": [
        "data"
      ],
      "two": "yes",
      "value_at_two": [
        "one"
      ],
      "three": "yes",
      "value_at_three": [
        "two"
      ]
    },
    "routing": {
      "step-one": -1,
      "step-two": -1
    },
    "requestPath": {
      "step-three": "seldon-experiments:step-3",
      "step-one": "seldon-experiments:step-1",
      "step-two": "seldon-experiments:step-2"
    },
    "metrics": []
  },
  "data": {
    "names": [],
    "ndarray": [
      "two"
    ]
  }
}

and here is with executor

curl -X POST -H 'Content-Type: application/json' \
        -d '{"meta": {"some": "input meta"}, "data": {"names": ["get"], "ndarray": ["data"]}}' \
        http://localhost:8003/seldon/seldon/step-experiments/api/v0.1/predictions | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   267  100   186  100    81   5166   2250 --:--:-- --:--:-- --:--:--  7416
{
  "data": {
    "names": [],
    "ndarray": [
      "two"
    ]
  },
  "meta": {
    "tags": {
      "current": "three",
      "one": "yes",
      "three": "yes",
      "two": "yes",
      "value_at_one": [
        "data"
      ],
      "value_at_three": [
        "two"
      ],
      "value_at_two": [
        "one"
      ]
    }
  }
}

[RafalSkolasinski]

And here is test with GRPC.

Engine:

cd /home/rskolasinski/work/seldon-core/executor/proto && grpcurl -d '{"data": {"names": ["get"], "ndarray": ["data"]}}' -rpc-header seldon:step-experiments-grpc -rpc-header namespace:seldon -plaintext -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict | jq .
{
  "meta": {
    "puid": "ph3cafjilu65adnrke44to9kvj",
    "tags": {
      "current": "three",
      "one": "yes",
      "three": "yes",
      "two": "yes",
      "value_at_one": [
        "data"
      ],
      "value_at_three": [
        "two"
      ],
      "value_at_two": [
        "one"
      ]
    },
    "routing": {
      "step-one": -1,
      "step-two": -1
    },
    "requestPath": {
      "step-one": "seldon-experiments-grpc:step-1",
      "step-three": "seldon-experiments-grpc:step-3",
      "step-two": "seldon-experiments-grpc:step-2"
    }
  },
  "data": {
    "ndarray": [
      "two"
    ]
  }
}

Executor:

cd /home/rskolasinski/work/seldon-core/executor/proto && grpcurl -d '{"data": {"names": ["get"], "ndarray": ["data"]}}' -rpc-header seldon:step-experiments-grpc -rpc-header namespace:seldon -plaintext -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict | jq .
{
  "meta": {
    "tags": {
      "current": "three",
      "one": "yes",
      "three": "yes",
      "two": "yes",
      "value_at_one": [
        "data"
      ],
      "value_at_three": [
        "two"
      ],
      "value_at_two": [
        "one"
      ]
    }
  },
  "data": {
    "ndarray": [
      "two"
    ]
  }
}

[RafalSkolasinski]

Just updated PR with applying fix to more methods and adding unit tests.

[RafalSkolasinski]

The PR does not (and probably won't) contain fix to send_feedback method.
This is because usage of tags in there is questionable and fix would require adjustment of protobufs.

[gmattar]

I'm still having some problems related to this.

This is my graph:

Basically, all models create tags and the Transformer Output needs to read them before forwarding the result of Model 3. I'm using Seldon 1.1 with REST requests.

The problem is that I can only see the tags from Model 3 inside Transformer.
These are the steps I took and what I found after inspecting the components:

• First I included a breakpoint inside aggregate but was not able to see the tags, the meta parameter was empty. Then, I included an aggregate_raw function and was able to see them. meta is not passed to client_aggregate in this scenario:

  seldon-core/python/seldon_core/seldon_methods.py

  Line 433 in de2dce0

  client_response = client_aggregate(user_model, features_list, names_list)

• I changed SELDON_LOG_LEVEL to DEBUG and checked the response sent by the Combiner component. The tags were passed to executor

• Next, I checked the meta parameter passed to Model 3 and the metas were empty again.

• Last, I checked the tags in Transformer Ouput and only the data of Model 3 was available

[ukclivecox]

I think transform output should have the meta. @RafalSkolasinski ?

[RafalSkolasinski]

@gmattar It seems that aggregate methods are not passing meta argument which contain tags.
Could you open a new issue for this and add information what version of wrapper are you using?