Re-setting numpy random seed to zero on every explain request

[SachinVarghese]

What this PR does / why we need it: This PR sets the numpy random seed to zero on every explain request. The main reason to do this is that the explainer request results are not reproducible otherwise since the seed value is based on current timestamp during explain call. This implementation is also limited to anchor explanation of tabular, text and image data.

[jklaise]

Thanks! Two thoughts:

• The seed should be configurable by the user
• It should be an option to configure not seeding at all (one use case down the line is to explain the same instance multiple times to gauge variation in the explanation - additional source of info)

I'm not sure how we handle customization of pre-packaged servers like these, is it purely via environment variables or are there some configs available too?

[SachinVarghese]

Thanks @jklaise for your comments. Both requirements makes sense and I can make minor changes to accomodate both with some caveats while still maintaining the scope of this PR as a quick win.

Background: The alibi explain server is configured by the seldon core operator during reconcilliation of a seldon deployment CRD and currently there is limited configuration available at the CRD level to setup explainer env variables at initialization time, except for envs from a k8s secret. So to answer your question on customisation of these servers, while this is surely possible with env vars indirectly but it isn't the best solution since you have to reference a secret for every seldon deployment explainer.

Proposed solution: Assuming any CRD changes is out of scope for this PR, I suggest the following fix to solve the issue at hand. By default, we can have random seeding enabled for anchor explanation techniques and the seed value set to zero, both of these can be overridden/removed by env vars at server level. These settings can be configured by k8s secret at init time as mentioned before at every explainer level. I feel this should serve as a good feature for reproducible explanation results in the near term. It would be good to hear any thoughts on this proposal.

Future work: There could also be a feature to configure such variables at explain time, but will defer that to future work due to complexity of passing config parameters as part of http explain requests metadata (something that doesn't exist yet in this server). Also, explain time parameters are present in MLserver alibi explain runtime though so this should be interesting for @sakoush.

[seldondev]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jklaise

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jklaise]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[seldondev]

Failed to merge this PR due to:

failed merging [4076]: [Method Not Allowed]