Additional configuration options for Spark jobs

[mike0sv]

Is your feature request related to a problem? Please describe.
Now there is no way to provide additional configuration options for spark jobs. For example, this is required to provide credentials for s3 storage.

Describe the solution you'd like
I'm using python sdk, so I dont know if this can be replicated in java and go, but my proposal is to add a configuration option, something like FEAST_SPARK_ADDITIONAL_OPTS which can be set to something like "option1=value1;option2=value2" and this options would be added in spark_submit method for standalone, dataproc_sumbit for dataproc and _submit_emr_job for emr.
We need to carefully pick the right format for this configuration option, considering there might be ; or = in option names or values.

Describe alternatives you've considered
Maybe explicitly provide those options with every client.start.... call, but it would break existing api, so it is way worse in my opinion.

[woop]

my proposal is to add a configuration option, something like FEAST_SPARK_ADDITIONAL_OPTS which can be set to something like "option1=value1;option2=value2"

Sounds good to me.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.