DARTS Suggestion

[andreyvelich]

Part of #1167.

This is Suggestion and Training Container for DARTS.

To support DARTS with current Katib implementation workflow looks like this:

1. Suggestion sends Search Space (Primitives) and Algorithm Settings (Epoch, etc.) to the training container.
2. Training container runs DARTS algorithm and creates Best Genotype with neural network architecture after training.

Currently, it supports only one GPU
Running on Tesla T4 GPU for 3 Epoch, 3 NN layers and 4 DARTS nodes takes 21 minutes.

In this PR, you can see example that I ran.

Unfortunately, Katib controller doesn't support String Objective Metric Value: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/trial/trial_controller_util.go#L180, but in MySQL DB we can see reported Genotype with the best architecture.

Do we want to support string objective metric value in the future, what do you think @gaocegege @johnugeorge ? Will it be useful for the Katib user?

This is example of Best Genotype that currently saves in DB:

Genotype(
normal=[
[('max_pooling_3x3', 0), ('max_pooling_3x3', 1)],
[('max_pooling_3x3', 0), ('max_pooling_3x3', 1)],
[('max_pooling_3x3', 0), ('max_pooling_3x3', 1)],
[('max_pooling_3x3', 0), ('max_pooling_3x3', 1)]
],
normal_concat=range(2, 6),
reduce=[
[('dilated_convolution_5x5', 1), ('separable_convolution_3x3', 0)],
[('dilated_convolution_5x5', 2), ('dilated_convolution_5x5', 1)],
[('dilated_convolution_5x5', 3), ('dilated_convolution_5x5', 2)],
[('dilated_convolution_5x5', 3), ('dilated_convolution_5x5', 4)]
],
reduce_concat=range(2, 6)
)

We can think about better representation later.

It supports only unrolled model, according the paper it produces better results.

I will update an issue with remained work.

/assign @johnugeorge @gaocegege

[kubeflow-bot]

This change is 

[gaocegege]

I think we can support the string metrics.

/lgtm

Thanks for your contribution! 🎉 👍

[gaocegege]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gaocegege]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment