[BUG] Failed to run blob task when cache is enabled

[ddl-rliu]

Describe the bug

Failed to run any task with FlyteFIle input/outputs when cache is enabled because type validation is failing.

{"json":{"exec_id":"x","node":"n0","ns":"x","res_ver":"73517980","routine":"worker-1",
"wf":"x.training_workflow_cache"},"level":"error",
"msg":"DataCatalog failed to get outputs from artifact adbf03db-daea-4108-9f8f-c352ac395a6b, 
err: unexpected artifactData: [processed_data] type: [blob:{}] does not match any 
task output type: [blob:{format:\"csv\"}]","ts":"2024-06-26T00:05:39Z"}

Note that this bug has a similar flavor to: #2864

When the cache is enabled, the task retrieves artifacts from datacatalog and checks if the blob format match the expected type.

However, this check fails.

Proposed fix: Add a Blob typechecker that allows empty format blobs to match generic blobs – #5519

Expected behavior

Blob task should run successfully even if the cache is enabled

Additional context to reproduce

import csv
from collections import defaultdict
from pathlib import Path
from typing import List, TypeVar, NamedTuple

import flytekit
from flytekit import task, workflow
from flytekit.types.file import FlyteFile


@task(cache=True, cache_version="3.0")
def t1() -> FlyteFile[TypeVar("csv")]:
    columns_to_normalize = ["a", "b"]
    out_path = Path(flytekit.current_context().working_directory) / f"normalized.csv"
    with out_path.open(mode="w") as output_file:
        writer = csv.DictWriter(output_file, fieldnames=columns_to_normalize)
        writer.writeheader()

    print(columns_to_normalize)
    return FlyteFile(path=out_path)


@task(cache=True, cache_version="3.0")
def t2(ff: FlyteFile[TypeVar("csv")]) -> FlyteFile[TypeVar("csv")]:
    print("hello")
    return ff


@workflow
def wf() -> FlyteFile[TypeVar("csv")]:
    return t2(ff=t1())


if __name__ == "__main__":
    wf()

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

The issue you described appears to be related to type validation when caching is enabled for tasks dealing with blob types. The error message indicates a mismatch between the expected and actual blob formats when retrieving cached artifacts from the DataCatalog.

This problem is similar to issue #2864, where type validation fails due to discrepancies in expected and actual blob formats. The error occurs because the cached artifact's format does not match the expected format specified in the task's output type.

Would you like more details on possible workarounds or solutions for this issue?

Sources:

• Issue #2864
• Type validation code

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[ddl-rliu]

As the fix was merged #5519 , now closing this issue.