Async job management

[zhilingc]

Currently jobs are kicked off in a synchronous way when feature sets are applied. This is an issue because it is:

• prone to race conditions which leads to failures when multiple feature sets are being applied.
• unresponsive for the user and inefficient on feast's side, particularly when the dataflow runner is used

This PR moves all job management to a worker thread.

The goal is to:

• Allow feature sets to be applied without blocking for a more snappy user experience
• Allow job updates to happen in the background without the user knowing about it
• Test and prevent any race conditions

[feast-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: zhilingc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [zhilingc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[zhilingc]

/hold

[woop]

This is still a [WIP]?

[zhilingc]

Yeah. I haven't implemented safe ingestion yet (hence the e2e tests are failing)

[zhilingc]

Pushed changes for safe ingestion: This mainly involves

• maintaining feature set status via the list of jobs that have to run,
• returning this status to clients via the GetFeatureSet/ListFeatureSet methods.
• Clients poll until feast indicates that the it is safe to ingest the feature set, then ingest the data.

Unfortunately this might introduce quite a bit of lag, particularly if a user submits their feature set directly after feast starts updating its jobs. Will need to think this through and fix in a further PR, but it works decently in the interim.

The change to the protos is more or less consistent with what is described in the issue #363.

[zhilingc]

/hold cancel

[zhilingc]

/retest

[zhilingc]

/retry

[woop]

/lgtm