[Core feature] LiteralBlob and StructuredDataset metadata

[kumare3]

Motivation: Why do you think this is important?

At runtime, Flyte knows about various metadata elements for a file / underlying dataset.
It would be amazing to catalog them and show them in the UI.
This metadata should be associated with the runtime objects and should be optional

Goal: What should the final outcome look like, ideally?

Example:

1. For FlyteFile - Blob.Single

• Size of the file,
• zipped boolean
• (we already store the format in the type)

2. For Directory / Blob.Multi

• Size of the total dataset
• zipped boolean
• an Index for the directory with all the files stored
• (we already store the format in the type)

3. StructureDataset

• Number of paritions
• Zipped
• Format
• Index of all the files / partitions

Note
We could also use this metadata to document lifecycle - for example if an expiration is set.
This can be useful in determining cache hits based on data TTLs. For example, each File / dataset etc can have the max-age set and this can be valid until date.

Describe alternatives you've considered

Store is separately, but then every File will become a directory and the format is only python. UI will have to implement this again.

Propose: Link/Inline OR Additional context

Add Blob metadata to

flyte/flyteidl/protos/flyteidl/core/literals.proto

Line 39 in cd37d1b

BlobType type = 1;

And
Structured Dataset metadata to

flyte/flyteidl/protos/flyteidl/core/literals.proto

Line 69 in cd37d1b

StructuredDatasetType structured_dataset_type = 1;

Also ideally the FileAccessProvider interface returns this metadata after each upload?
https://github.com/flyteorg/flytekit/blob/070027d8781d9b9f830723a9a4cae5daa6359a33/flytekit/core/data_persistence.py#L311

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

To address the issue of associating metadata with LiteralBlob and StructuredDataset at runtime, Flyte can leverage its existing metadata handling capabilities. According to the Flyte documentation, Flyte already manages metadata for task inputs and outputs, storing references to large objects in backend storage.

For FlyteFile and Blob.Single, metadata such as file size and whether the file is zipped can be added. Similarly, for Directory or Blob.Multi, metadata can include the total dataset size, zipped status, and an index of all files. For StructuredDataset, metadata can include the number of partitions, zipped status, format, and an index of all files/partitions.

Would you like more detail on how to implement this in Flyte?

Sources:

• Flyte Data Management
• Flyte Issue #4683

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!