[Bug]: k8s spark optimizer release error

[lintingbin]

What happened?

When I release the Spark K8s optimizer, an error is reported. I found that the namespace for the kill command is not the one I configured in the resource group, but is "default".

Affects Versions

0.7.1

What table formats are you seeing the problem on?

Iceberg

What engines are you seeing the problem on?

AMS

How to reproduce

No response

Relevant log output

2025-01-09 08:14:57,208 INFO [JettyServerThreadPool-1470070] [org.apache.amoro.server.manager.SparkOptimizerContainer] [] - Releasing spark optimizer using command: export SPARK_HOME=/opt/spark && export HADOOP_USER_NAME=hive && /opt/spark/bin/spark-submit --kill default:amoro-optimizer-6i6ld3rup897p9dhdpf0ifgosh --master k8s://https://kubernetes.default.svc --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.executor.memory=4g --conf spark.driver.memory=2g --conf spark.driver.cores=1 --conf spark.kubernetes.executor.podTemplateFile=gs://hdp-vvp/spark/executor_pod_template.yaml --conf spark.executor.cores=1 --conf spark.dynamicAllocation.maxExecutors=5 --conf spark.dynamicAllocation.shuffleTracking.enabled=true --conf spark.kubernetes.namespace=amoro --conf spark.kubernetes.container.image=asia.gcr.io/farlight-hadoop/amoro-spark-optimizer:0.7.1-incubating-spark3.3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=amoro --conf spark.shuffle.service.enabled=false --conf spark.dynamicAllocation.enabled=true
root@amoro-76fcd56c4d-jrgjh:/usr/local/amoro# export SPARK_HOME=/opt/spark && export HADOOP_USER_NAME=hive && /opt/spark/bin/spark-submit --kill default:amoro-optimizer-6i6ld3rup897p9dhdpf0ifgosh --master k8s://https://kubernetes.default.svc --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.executor.memory=4g --conf spark.driver.memory=2g --conf spark.driver.cores=1 --conf spark.kubernetes.executor.podTemplateFile=gs://hdp-vvp/spark/executor_pod_template.yaml --conf spark.executor.cores=1 --conf spark.dynamicAllocation.maxExecutors=5 --conf spark.dynamicAllocation.shuffleTracking.enabled=true --conf spark.kubernetes.namespace=amoro --conf spark.kubernetes.container.image=asia.gcr.io/farlight-hadoop/amoro-spark-optimizer:0.7.1-incubating-spark3.3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=amoro --conf spark.shuffle.service.enabled=false --conf spark.dynamicAllocation.enabled=true

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tcodehuber]

@lintingbin Which namespace is your AMS installed in K8s?

[lintingbin]

@tcodehuber We specified the namespace on the resource group through spark-conf.spark.kubernetes.namespace=amoro, but when killing, it was the default namespace. You can look at the PR I submitted and you will understand what is going on.