As a downstream consumer of aiSSEMBLE, I want my SparkApplications to interface with the v2 SparkHistory chart by default.

[peter-mcclonski]

Definition of Done

• Add an alternate values file within the SparkApplication base chart which supports:
  • The SparkApplication dev template shall be updated to default to mounting the Spark History event volume.
  • The SparkApplication dev template shall default to writing spark event logs to the Spark History event volume.
• Create the Spark Ecosystem V2 chart template for downstream projects
• Ensure that the Spark Ecosystem V2 chart for downstream projects supports the new Spark History V2 chart.

Test Steps

1. Generate a new project with the following command:

mvn archetype:generate -B -DarchetypeGroupId=com.boozallen.aissemble \
                          -DarchetypeArtifactId=foundation-archetype \
                          -DarchetypeVersion=1.7.0-SNAPSHOT \
                          -DartifactId=test-project\
                          -DgroupId=org.test \
                          -DprojectName='Test' \
                          -DprojectGitUrl=test.org/test-project\
&& cd test-project

2. Add the following pipeline definitions to test-project-pipeline-models/src/main/resources/pipelines/:

{
	"name": "SimplePipeline",
	"package": "org.test",
	"type": {
		"name": "data-flow",
		"implementation": "data-delivery-spark"
	},
	"steps": [
		{
			"name": "Ingest",
			"type": "synchronous",
			"alerting": {
				"enabled": false
			},
			"dataProfiling": {
				"enabled": false
			},
			"provenance": {
				"enabled": false
			}
		}
	]
}

{
	"name": "SimplePipelinePython",
	"package": "org.test",
	"type": {
		"name": "data-flow",
		"implementation": "data-delivery-pyspark"
	},
	"steps": [
		{
			"name": "Ingest",
			"type": "synchronous",
			"alerting": {
				"enabled": false
			},
			"dataProfiling": {
				"enabled": false
			},
			"provenance": {
				"enabled": false
			}
		}
	]
}

3. Execute mvn clean install repeatedly, resolving all manual actions until none remain.
4. Modify the contents of test-project-deploy/pom.xml, replacing <profile>aissemble-spark-infrastructure-deploy</profile> with <profile>aissemble-spark-infrastructure-deploy-v2</profile>
5. Delete the directory test-project-deploy/src/main/resources/apps/spark-infrastructure along with its contents.
6. Execute mvn clean install
7. _OTS ONLY: _ Replace the repository in the generated Chart.yaml with the absolute path to the spark-history chart in the local aissemble repository, and re-execute mvn clean install
8. Use kubectl apply -f to apply the following yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-config
data: {}

9. Update the contents of test-project-pipelines/simple-pipeline/src/main/resources/apps/simple-pipeline-base-values.yaml and test-project-pipelines/simple-pipeline-python/src/simple_pipeline_python/resources/apps/simple-pipeline-python-base-values.yaml to remove references to hadoop-aws and aws-java-sdk-bundle
10. Update the contents of test-project-pipelines/simple-pipeline/src/main/resources/apps/simple-pipeline-dev-values.yaml and test-project-pipelines/simple-pipeline-python/src/simple_pipeline_python/resources/apps/simple-pipeline-python-dev-values.yaml to remove their spark.eventLog configurations.
11. Save the following content at the root of test-project with the name values-migrate-dev.yaml:

########################################
## CONFIG | Spark Configs
########################################
metadata:
  namespace: default
sparkApp:
  spec:
    sparkConf:
      spark.eventLog.enabled: "true"
      spark.eventLog.dir: "/opt/spark/spark-events"
    type: "placeholder" #required for a dry run test to pass, this should always be overridden
    mode: cluster
    imagePullPolicy: IfNotPresent
    restartPolicy:
      type: Never
    sparkVersion: "3.4.0"
    sparkConfigMap: spark-config
    dynamicAllocation:
      enabled: true
      initialExecutors: 0
      minExecutors: 0
      maxExecutors: 4
    volumes:
      - name: ivy-cache
        persistentVolumeClaim:
          claimName: ivy-cache
      - name: spark-events
        persistentVolumeClaim:
          claimName: spark-events-claim
    driver:
      cores: 1
      coreLimit: "1200m"
      memory: "512m"
      serviceAccount: spark
      volumeMounts:
        - name: ivy-cache
          mountPath: "/opt/spark/.ivy2"
        - name: spark-events
          mountPath: "/opt/spark/spark-events"
    executor:
      cores: 1
      coreLimit: "1200m"
      memory: "512m"
      labels:
        version: 3.4.0
      volumeMounts:
        - name: ivy-cache
          mountPath: "/opt/spark/.ivy2"
        - name: spark-events
          mountPath: "/opt/spark/spark-events"
service:
  enabled: false
  spec:
    ports:
      - name: "debug"
        port: 4747
        targetPort: 4747

12. Search for spark-application in your TiltFile, adding --values values-migrate-dev.yaml after --version %s in both cases.
13. Execute mvn clean install -Dmaven.build.cache.skipCache=true, resolving any lingering manual actions
14. Execute tilt up and wait for all resources to be ready
15. Trigger execution of the simple-pipeline resource, ensuring it completes successfully
16. Trigger execution of the simple-pipeline-python resource, ensuring it completes successfully
17. Navigate to localhost:18080
18. Ensure that spark events are visible in the spark-history UI from both simple-pipeline and simple-pipeline-python

[csun-cpointe]

Test passed !!