The status of the AWS batch job should become failed once the retry limit exceeded

[pingsutw]

Signed-off-by: Kevin Su [email protected]

TL;DR

The task status is always running if the task failure is retryable. In other words, the propeller will keep waiting no matter how often the task is re-run. Take a look at these code. If all the sub-task failed and all the failure is retryable, the state of the task will never become PhaseWriteToDiscoveryThenFail. (totalRetryableFailures is equal to minSuccesses here)

In this pr, we kept tracking how often the task has retried and counted the total number of sub-tasks that exceeded the retry limit. Stop the task when totalRunning+totalSuccesses+totalWaitingForResources+totalRetryableFailures-totalRetryLimitExceeded < minSuccesses is true

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

Tracking Issue

flyteorg/flyte#2979

Follow-up issue

NA

[codecov]

Codecov Report

Merging #291 (6809ea4) into master (eed7326) will decrease coverage by 1.04%.
The diff coverage is 70.00%.

@@            Coverage Diff             @@
##           master     #291      +/-   ##
==========================================
- Coverage   63.32%   62.27%   -1.05%     
==========================================
  Files         145      145              
  Lines        9311    11511    +2200     
==========================================
+ Hits         5896     7169    +1273     
- Misses       2872     3797     +925     
- Partials      543      545       +2     

Flag	Coverage Δ
unittests	62.27% <70.00%> (-0.47%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
go/tasks/plugins/array/awsbatch/executor.go	36.95% <0.00%> (-1.10%)	⬇️
go/tasks/plugins/array/awsbatch/monitor.go	67.42% <73.68%> (+5.62%)	⬆️
go/tasks/plugins/k8s/pod/container.go	57.14% <0.00%> (-9.53%)	⬇️
.../plugins/k8s/kfoperators/common/common_operator.go	71.30% <0.00%> (-8.08%)	⬇️
...o/tasks/pluginmachinery/utils/secrets/marshaler.go	64.28% <0.00%> (-7.94%)	⬇️
go/tasks/plugins/webapi/athena/utils.go	67.92% <0.00%> (-7.35%)	⬇️
...o/tasks/pluginmachinery/ioutils/raw_output_path.go	68.42% <0.00%> (-6.58%)	⬇️
go/tasks/plugins/hive/client/qubole_client.go	59.44% <0.00%> (-6.38%)	⬇️
...sks/plugins/k8s/sagemaker/hyperparameter_tuning.go	55.20% <0.00%> (-5.25%)	⬇️
go/tasks/pluginmachinery/core/plugin.go	70.00% <0.00%> (-5.00%)	⬇️
... and 122 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.