Fixing ArrayNode integration with backoff controller

[hamersaw]

Tracking issue

NA

Why are the changes needed?

ArrayNode integrates into the core execution log of FlytePropeller. This means that, unlike the existing maptask implementation, it forwards through FlytePropeller's backoff controller when creating k8s resources. If the backoff controller detects FlytePropeller is starting creating too many resources it will throttle by setting the task phase to WaitingForResources. Previously, ArrayNode mapped this phase to Running and then immediately failed when the resource did not exist in the next evaluation round.

What changes were proposed in this pull request?

This PR treats the task phase WaitingForResources as NotYetStarted so that evaluation of the subNode integrates with FlytePropeller's backoff controller.

How was this patch tested?

Local testing with varying resource requests and k8s resource quotas. Also testing on Union Cloud.

Setup process

Create a k8s resource quota, example:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: foo
  namespace: flytesnacks-development
spec:
  hard:
    limits.cpu: "2"
    limits.memory: 2000Mi

And start an ArrayNode that will exceed this quota, example:

import time
import typing
from flytekit import task, workflow, dynamic, Resources
from flytekit.experimental import map_task
#from flytekit import map_task

@task(requests=Resources(cpu="1", mem="1Gi"))
def say_hello(name: str) -> str:
    time.sleep(1)
    return f"hello, {name} {name}"

@workflow
def wf(names: typing.List[str]) -> typing.List[typing.Optional[str]]:
    return map_task(say_hello)(name=names)

with pyflyte run --remote array-node.py wf --names='["foo","bar","baz"]'

Screenshots

NA

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (cb37291) 68.09% compared to head (096ac74) 58.12%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4640      +/-   ##
==========================================
- Coverage   68.09%   58.12%   -9.98%     
==========================================
  Files          31      626     +595     
  Lines        1680    53815   +52135     
==========================================
+ Hits         1144    31278   +30134     
- Misses        456    20035   +19579     
- Partials       80     2502    +2422     

Flag	Coverage Δ
unittests	58.12% <100.00%> (-9.98%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.