[BUG] Dynamic workflows won't run with flytekit>=1.3.0

[bstadlbauer]

Describe the bug

After upgrading to flytekit>=1.3.0, the following error message appears when workflows are run:

Workflow[dask-plugin:development:demo.dask_demo.workflows.dynamic_workflow] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]

Example workflow:

from flytekit import workflow, task, dynamic


@task
def some_task():
    print("Do something")


@dynamic
def dynamic_task():
    some_task()


@workflow
def dynamic_workflow():
    dynamic_task()

The workflow was serialized using pyflyte -k <package> serialize --image <image> workflows -f <output_dir.

A bit of investigation lead to possibly flyteorg/flytekit#1378 being the root cause of this.

Expected behavior

The dynamic workflow should run

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[eapolinario]

Speaking about the plan for this bug. We're reverting the offending change and getting a patch release of flytekit out (version 1.3.2).

Members of the core team got together earlier today and the consensus is that we have a bug in the backend, more specifically the compiler, but while we work on a fix, we're mitigating this issue by the plan described ahead.

We decided against yanking the versions 1.3.0 and 1.3.1 of flytekit because this bug only affects dynamic workflows.

[eapolinario]

Just adding a breadcrumb, #3169 (comment) describes the rationale for the offending change.

We'll be revisiting this in the next few days.

[lindseynield]

@eapolinario do you know when the 1.3.2 patch will get propagated to the feedstock?

[eapolinario]

@lindseynield , 1.3.2 change is happening in conda-forge/flytekit-feedstock#38 and should land in the next day or two.

[wild-endeavor]

can we close this issue? or are we still waiting on something from conda?

@bstadlbauer I re-patched in flyteorg/flytekit#1529 - mind taking a look?

[eapolinario]

@eapolinario do you know when the 1.3.2 patch will get propagated to the feedstock?

@lindseynield , flytekit 1.3.2 is in conda: https://anaconda.org/conda-forge/flytekit