Ability to pass runner option during ingestion job submission

[pradithya]

Runner option is currently hardcoded in deployment (specifically core's deployment). Some ingestion jobs might need to override those value without redeploying feast.
One way to do it is by adding runnerOptions field in the SubmitImportJobRequest as follow:

 message SubmitImportJobRequest {
        feast.specs.ImportSpec importSpec = 1;
        string name = 2; 
        map<string, string> runnerOptions = 3; // NEW, optional
    }

This runnerOptions is then used to override default runner option and then passed to the runner.

thought? @woop @zhilingc

[zhilingc]

The CLI command would look something like this?
feast jobs run import_spec.yml --with-options=k1=v1;k2=v2

[woop]

Runner option is currently hardcoded in deployment (specifically core's deployment). Some ingestion jobs might need to override those value without redeploying feast.
One way to do it is by adding runnerOptions field in the SubmitImportJobRequest as follow:

 message SubmitImportJobRequest {
        feast.specs.ImportSpec importSpec = 1;
        string name = 2; 
        map<string, string> runnerOptions = 3; // NEW, optional
    }

This runnerOptions is then used to override default runner option and then passed to the runner.

thought? @woop @zhilingc

I don't have the full context on these options. Can you point to an example of how it is currently done? Or at least the code?

[pradithya]

In core's application.properties file we have
feast.jobs.options=${JOB_OPTIONS:{}} which accept a json value.
The json value contains runner's specific parameter, for example if the runner is Dataflow we have to pass (not limited to):

• project
• region
• tempLocation
  Those json values is then converted into beam's parameter when triggering ingestion job.
  'java -jar --project=projectName --region=regionName --tempLocation=gs://bucket ingestion.jar"
  Some value is specific to infrastructure setup, but in some case it isn't. One concrete example I can think of is paralellism for Flink.

[woop]

Thanks @pradithya, that makes a lot more sense. I agree that this is a problem we should think of solving. I guess we need to decide whether exposing that at an SubmitImportJobRequest is the right place.

Is it only exposed to the user through the CLI and Python SDK?

For the CLI, would it be better to use Helm style
feast jobs run import_spec.yml --set k1=v1 --set k2=v2
or possibly your version, but with a comma instead of semi-colon
feast jobs run import_spec.yml --options=k1=v1,k2=v2

[tims]

These are ingestion options not just runner options right? Theoretically you might want to pass in things like the temp directory or metric sinks.

[woop]

These are ingestion options not just runner options right? Theoretically you might want to pass in things like the temp directory or metric sinks.

Yes

[woop]

This issue isn't relevant any more due to changes in Feast 0.3.0.