docs: review and normalize haystack.components.fetchers

[wochinge]

Related Issues

• fixes Docstrings - haystack.components.fetchers #7107
• fixes API Docs - haystack.components.fetchers #7162

Proposed Changes:

• make protected / private what's not needed to be public
• normalize / clean

How did you test it?

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[wochinge]

---

title: Fetchers
excerpt: Fetches content from a list of URLs and returns a list of extracted content streams.
category: placeholder-haystack-api
slug: fetchers-api
parentDoc:
order: 80
hidden: false

Module link_content

LinkContentFetcher

@component
class LinkContentFetcher()

LinkContentFetcher is a component for fetching and extracting content from URLs.

It supports handling various content types, retries on failures, and automatic user-agent rotation for failed web
requests.

Usage example:

from haystack.components.fetchers.link_content import LinkContentFetcher

fetcher = LinkContentFetcher()
streams = fetcher.run(urls=["https://www.google.com"])["streams"]

assert len(streams) == 1
assert streams[0].meta == {'content_type': 'text/html', 'url': 'https://www.google.com'}
assert streams[0].data

LinkContentFetcher.__init__

def __init__(raise_on_failure: bool = True,
             user_agents: Optional[List[str]] = None,
             retry_attempts: int = 2,
             timeout: int = 3)

Initializes the component.

Arguments:

• raise_on_failure: If True, raises an exception if it fails to fetch a single URL.
  For multiple URLs, it logs errors and returns the content it successfully fetched. Default is True.
• user_agents: User agents
  for fetching content. If None, a default user agent is used.
• retry_attempts: Specifies how many times you want it to retry to fetch the URL's content. Default is 2.
• timeout: Timeout in seconds for the request. Default is 3.

LinkContentFetcher.run

@component.output_types(streams=List[ByteStream])
def run(urls: List[str])

Fetches content from a list of URLs and returns a list of extracted content streams.

Each content stream is a ByteStream object containing the extracted content as binary data.
Each ByteStream object in the returned list corresponds to the contents of a single URL.
The content type of each stream is stored in the metadata of the ByteStream object under
the key "content_type". The URL of the fetched content is stored under the key "url".

Arguments:

• urls: A list of URLs to fetch content from.

Raises:

• Exception: If the provided list of URLs contains only a single URL, and raise_on_failure is set to
  True, an exception will be raised in case of an error during content retrieval.
  In all other scenarios, any retrieval errors are logged, and a list of successfully retrieved ByteStream
  objects is returned.

Returns:

ByteStream objects representing the extracted content.

[coveralls]

Pull Request Test Coverage Report for Build 8078330587

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 17 unchanged lines in 1 file lost coverage.
• Overall coverage remained the same at 90.058%

---

Files with Coverage Reduction	New Missed Lines	%
components/fetchers/link_content.py	17	79.52%

Totals
Change from base Build 8077602639:	0.0%
Covered Lines:	5281
Relevant Lines:	5864

---

💛 - Coveralls

[anakin87]

In general, it looks good...

Please just remove defaults from docstrings.

[anakin87]

👍