Design Doc For Katib

[jlewi]

A design doc explaining the architecture of Katib would be extremely useful.

There are a number of components in Katib e.g.

• Multiple databases
• Frontend
• CLI
• Controller programs

Understanding how all these elements fit together would be extremely useful. A number of issues have come up where I think a design doc would be useful; e.g.

• Ensuring durable storage Katib needs to use durable storage that outlives the pod. #137
• How to run HP tuning without writing code Study controller - Running HP Jobs without writing code #87
• How to support auto ML (e.g. Cisco's AutoML framework)

Some other things to think about

• Extensibility

  • How can we give users flexibility so that researchers can easily add new algorithms as part of their research

• Monitoring

• Scalability

  • Do we need K8s to provide a job queuing system?

[jlewi]

/assign @YujiOshima

[ddysher]

@YujiOshima any progress on this? I think this is probably the highest priority for now :)

/cc @gaocegege

[gaocegege]

Yeah, I also think it is necessary. While if @YujiOshima does not have time I can write the first version.

[YujiOshima]

@ddysher @gaocegege Thanks!
I'm working on here https://docs.google.com/document/d/1ZEKhou4z1utFTOgjzhSsnvysJFNEJmygllgDCBnYvm8/edit .
Very welcome to edit!

[jlewi]

Going to close this issue since we have the initial doc.