Jobs evicted for lack of memory, potentially add resource field to tf-job prototype

[cwbeitel]

I'm seeing jobs being evicted due to a out of memory error, e.g.

> kubectl get pods -n rl --show-all
agents-ppo-ant-feedforward-93168c9e-master-...           0/1       Evicted   0          19h
agents-ppo-ant-feedforward-93168c9e-master-...           0/1       Evicted   0          18h
agents-ppo-ant-feedforward-93168c9e-master-...           1/1       Running   0          9h
agents-ppo-ant-feedforward-93168c9e-master-...           0/1       Evicted   0          18h
agents-ppo-ant-feedforward-93168c9e-tensorboard-...   1/1       Running   0          19h
agents-ppo-ant-ee3e499e-master-aiod-0-768ls           1/1       Running   0          18h
agents-ppo-ant-ee3e499e-tensorboard-aiod-76dbf4brlz   1/1       Running   0          18h
...

> kubectl describe pod -n rl agents-ppo-ant-feedforward-93168c9e-master-3ti2-0-9st9r
...
Status:         Failed
Reason:         Evicted
Message:        The node was low on resource: memory.
...

and wondering what the correct way is to specify resource limits - whether it's to add such a field to the tf-job ksonnet prototype or something else.

[cwbeitel]

Sorry tf-job prototype issues belong in https://github.com/google/kubeflow