Removal Of Spark History service [ui-svc]

[nileshbhadana]

When spark job is created it creates a k8s service for Spark UI.
Is there any configuration by which we can stop this service from getting created ?

Steps to reproduce:
- Run a spark job
- Wait for it to complete and see kubectl get svc -n <namespace>

Thanks in Advance

[jkleckner]

@nileshbhadana The driver pod sticks around even after completed. The service and driver pod will both go away when you delete the spark application.

[nileshbhadana]

@jkleckner I don't want the service to be created.

[kz33]

@nileshbhadana I think there should be no such configuration, but you modify the code of operoter and implement this function through a custom operator。

If you can, you can try to comment the following code in pkg/controller/sparkapplication/controller.go：

service, err := createSparkUIService(app, c.kubeClient)
if err != nil {
	glog.Errorf("failed to create UI service for SparkApplication %s/%s: %v", app.Namespace, app.Name, err)
} else {
	app.Status.DriverInfo.WebUIServiceName = service.serviceName
	app.Status.DriverInfo.WebUIPort = service.servicePort
	app.Status.DriverInfo.WebUIAddress = fmt.Sprintf("%s:%d", service.serviceIP, app.Status.DriverInfo.WebUIPort)
	// Create UI Ingress if ingress-format is set.
	if c.ingressURLFormat != "" {
		ingress, err := createSparkUIIngress(app, *service, c.ingressURLFormat, c.kubeClient)
		if err != nil {
			glog.Errorf("failed to create UI Ingress for SparkApplication %s/%s: %v", app.Namespace, app.Name, err)
		} else {
			app.Status.DriverInfo.WebUIIngressAddress = ingress.ingressURL
			app.Status.DriverInfo.WebUIIngressName = ingress.ingressName
		}
	}
}

But I am not sure if it will cause other problems 😂

[sairamankumar2]

Hi @kz33 @jkleckner,
We use this operator in production for running 1000's of spark job and we don't really use UI service. Also, there is a limitation on number of services that can be created in a namespace. As a side note, we have hit that limit quite few times even after using
a TTL of 1 second. If there was variable to disable the UI service, it would be of great help.

Would be happy to contribute a PR for the same (would be my first PR in open-source project).