Elastic resize broken in v2 operator: Horovod requires the slots parameter

[alculquicondor]

In the v2 controller, the slots parameter in the hostfile was replaced by environment variables:

mpi-operator/v2/pkg/controller/mpi_job_controller.go

Lines 116 to 117 in b88edad

	openMPISlotsEnv = "OMPI_MCA_orte_set_default_slots"
	intelMPISlotsEnv = "I_MPI_PERHOST"

However, horovod still requires the number of slots. So we need to put the parameter back. The duplication of information should be fine.
It's unclear if horovod supports intel.

We need an E2E test to ensure that the fix works long term.

[alculquicondor]

/help

[google-oss-prow]

@alculquicondor:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

Also, we need to adapt the intel entry point to the change of the hostfile format: https://github.com/kubeflow/mpi-operator/blob/master/examples/base/intel-entrypoint.sh

[alculquicondor]

@tenzen-y we should fix this as part of the next release #507, but I don't have bandwidth. Do you?

[tenzen-y]

@tenzen-y we should fix this as part of the next release #507, but I don't have bandwidth. Do you?

@alculquicondor Thanks for mentioning this. Sure.

IIUC, we need to modify the hostfile in

mpi-operator/pkg/controller/mpi_job_controller.go

Line 1186 in 31d4575

buffer.WriteString(fmt.Sprintf("%s%s-%d.%s.%s.svc\n", mpiJob.Name, workerSuffix, i, workersService, mpiJob.Namespace))

to buffer.WriteString(fmt.Sprintf("%s%s-%d.%s slots=%d\n", mpiJob.Name, workerSuffix, i, workersService, slots)), right?

Also, we need to adapt the new hostfile format in

mpi-operator/examples/base/intel-entrypoint.sh

Line 30 in db6930d

cat /etc/mpi/hostfile | while read host

.

[tenzen-y]

/assign