feat: Support bytestream in Unstructured API

[alperkaya]

Related Issues

• fixes Unstructured: support ByteStream input in run method #1075

Proposed Changes:

Unstructured API can also be called with Bytestream now in addition of Path.

How did you test it?

Added extra unit tests

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[CLAassistant]

All committers have signed the CLA.

[alperkaya]

@vblagoje , I try to address your request in this PR.

[silvanocerza]

@alperkaya can you please sign the CLA? Otherwise we can't merge this. :)

[alperkaya]

@alperkaya can you please sign the CLA? Otherwise we can't merge this. :)

done ;)

[silvanocerza]

This is wrong.

If you have a combination of paths and ByteStream in your sources and a list for meta you risk assigning the wrong meta to the wrong Document.

You can verify with something like this:

from pathlib import Path

from haystack.dataclasses import ByteStream
from haystack_integrations.components.converters.unstructured import (
    UnstructuredFileConverter,
)

converter = UnstructuredFileConverter()

sources = [
    "README.md",
    ByteStream(data=b"content", meta={"file_path": "some_file.md"}),
    Path(__file__),
    ByteStream(data=b"content", meta={"file_path": "yet_another_file.md"}),
    ByteStream(data=b"content", meta={"file_path": "my_file.md"}),
]

meta = [
    {"type": "str"},
    {"type": "ByteStream"},
    {"type": "Path"},
    {"type": "ByteStream"},
    {"type": "ByteStream"},
]

res = converter.run(sources=sources, meta=meta)

Also I notice that some meta fields are completely lost.

[silvanocerza]

This implementation has quite some problems, I would suggest taking a look at the existing converters we have in core.
https://github.com/deepset-ai/haystack/tree/main/haystack/components/converters

[alperkaya]

Hi @silvanocerza,

I try to address your comment by checking the existing converters.

This version covers these cases without losing meta fields.

Case 1: Files with Meta as None
Case 2: ByteStreams with Meta as None
Case 3: Files with Meta as a Dictionary
Case 4: ByteStreams with Meta as a Dictionary
Case 5: Files with Meta as a List
Case 6: ByteStreams with Meta as a List
Case 7: Directory with Meta as a Dictionary
Case 8: Directory with Meta as a List (Should Fail)
Case 9: Combination of File Paths, ByteStreams, and Directory with Meta as a Dictionary

[silvanocerza]

This is still not working as expected, I strongly suggest you copy the implementation from core.

[alperkaya]

This is still not working as expected, I strongly suggest you copy the implementation from core.

Hi, in the core repo, the solution handles either file paths or bytestreams. However, in this PR, I’m managing not just file paths and bytestreams, but also directories, where I need to fetch all files without entering subfolders. Given these additional requirements, could we explore how to modify the core solution to meet this use case, or would an alternative approach be better suited here?