Add additional VM configuration options

[stoyanr]

Currently it's already possible to specify essential VM parameters such as CPUs, memory, disk size, and storage class for the VM disk. We should be able to specify also additional VM parameters, usually for an enhanced VM performance. These could include, among others:

• hugepages support
• dedicated CPU
• disk device cache

We should discuss what additional parameters should be added, and how should they be configured. The options are:

• In the spec.providerConfig section of the cloud profile, if to configure machine types shared between multiple shoots
• In the provider.workers[n].providerConfig section of the shoot resource, to configure individual shoots.

/area performance
/kind enhancement
/priority normal
/platform kubevirt

[stoyanr]

/cc @mfranczy Feel free to add your ideas and comments

[mfranczy]

Overall we should support:

• hugepages support

• customizable CPU topology

• support for resource over-commit https://kubevirt.io/user-guide/#/usage/overcommit

• high performance options for disks and volumes https://kubevirt.io/user-guide/#/creation/disks-and-volumes?id=high-performance-features (IOThreads, Virtio Block Multi-Queue, DiskDevice cache)

• networkInterfaceMultiqueue support https://kubevirt.io/user-guide/#/creation/interfaces-and-networks?id=virtio-net-multiqueue

- [ ] support for VM run strategies https://kubevirt.io/user-guide/#/creation/run-strategies

- [ ] set liveness and readiness probes https://kubevirt.io/user-guide/#/creation/probes

• support for DNS options https://kubevirt.io/user-guide/#/usage/dns

[mfranczy]

/assign @mfranczy

[mfranczy]

@stoyanr @moadqassem we have to discuss how we want to support requests and limits in terms of specified resources. Currently in a machineType object we can only set requested resources, the question is how we want to handle limits.
Let's discuss this on our daily sync.

[mfranczy]

@stoyanr I removed support for RunStrategy and probes, giving this a second thought, since MCM has a timeout for machine to join nodes that would bring another complexity level for MCM-provider-kubevirt which is not needed.

The only left things are disks performance settings and validator which I am working on now.