fix: Redundant feature materialization and premature incremental materialization timestamp updates

[james-crabtree-sp]

What this PR does / why we need it:

This PR fixes 2 issues encountered while attempting to run very large materializations (Several billion rows as part of an initial run of online materialization).

1. Every pod created materializes every file in the dataset. This is redundant and costs a lot of time and money. To fix this, I updated the dataflow code to select only the single file corresponding to the running pod's JOB_COMPLETION_INDEX as input.
2. The last materialized date used for incremental materialization was being updated before materialization had even run. This meant that if materialization had failed for some reason, many updates could be ignored, because the next run would continue from beyond the end date of the failed run. I solved this by implementing a synchronous mode for bytewax materialization that will wait for materialization of a feature to complete successfully before returning and updating the last materialized date
3. Bytewax jobs can run infinitely if node crashes occur often enough that the job has to continuously reprocess the same pods after they are lost. I mitigated this by adding a configurable activeDeadlineSeconds and by adding an option to synchronous materialization to allow these jobs to be split into smaller batches so fewer pods have to be rerun in the event of a node failure

Which issue(s) this PR fixes:
Fixes #3786
Fixes #3787
Fixes #3788

[sudohainguyen]

great, I have been working on the similar PR, but you did it first and way better 💯

[whoahbot]

@james-crabtree-sp Thanks again for putting this PR together. I think in order to merge, you'll need to sign your commits: https://docs.feast.dev/project/development-guide#signing-off-commits

[james-crabtree-sp]

@james-crabtree-sp Thanks again for putting this PR together. I think in order to merge, you'll need to sign your commits: https://docs.feast.dev/project/development-guide#signing-off-commits

I think they are all signed off already. I see that the DCO task that checks this completed successfully https://github.com/feast-dev/feast/pull/3789/checks?check_run_id=17541354092

[whoahbot]

@james-crabtree-sp Thanks again for putting this PR together. I think in order to merge, you'll need to sign your commits: https://docs.feast.dev/project/development-guide#signing-off-commits

I think they are all signed off already. I see that the DCO task that checks this completed successfully https://github.com/feast-dev/feast/pull/3789/checks?check_run_id=17541354092

Sorry for the confusion, I was referring to the requirements to merge, and linked to the wrong documentation.

This is the requirement that I was referring to:

Merging is blocked
The base branch requires all commits to be signed. Learn more about signing commits.

[sudohainguyen]

hey @james-crabtree-sp this is a great PR, you need any hands to make it merged? I see there's a lot of commits in the past unverified

if you don't mind I can create another PR to mimic this one

[james-crabtree-sp]

@whoahbot @sudohainguyen Ahh, I understand now. Commits are now verified

[sudohainguyen]

@whoahbot @sudohainguyen Ahh, I understand now. Commits are now verified

exactly 🤣 looks good now but you need to run code format though

[james-crabtree-sp]

@whoahbot @sudohainguyen Ahh, I understand now. Commits are now verified

exactly 🤣 looks good now but you need to run code format though

Noticed that too. Weird rebase issue. Should be fixed now

[sudohainguyen]

@james-crabtree-sp I've been adopting your changes on my side

so far do we need to wait the job to be done, like forever without timeout?

[james-crabtree-sp]

@james-crabtree-sp I've been adopting your changes on my side

so far do we need to wait the job to be done, like forever without timeout?

Well, this PR adds the configurable active_deadline_seconds setting, which can be used to impose a timeout on the kubernetes jobs. Does that answer your question?

[sudohainguyen]

@james-crabtree-sp I've been adopting your changes on my side
so far do we need to wait the job to be done, like forever without timeout?

Well, this PR adds the configurable active_deadline_seconds setting, which can be used to impose a timeout on the kubernetes jobs. Does that answer your question?

cool clear