feat: Add DocumentLanguageClassifier 2.0 #6037
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
topic:tests
type:documentation
Pull Request Test Coverage Report for Build 6508357776
💛 - Coveralls |
|
In the end to end test I cam across a problem with the connection named "text/plain" and opened an issue in canals: deepset-ai/canals#130 |
This was referenced Oct 13, 2023
Closed
masci
suggested changes
Oct 18, 2023
haystack/preview/components/preprocessors/document_language_classifier.py
Outdated
Show resolved
Hide resolved
haystack/preview/components/preprocessors/document_language_classifier.py
Outdated
Show resolved
Hide resolved
| ```python | ||
| document_store = MemoryDocumentStore() | ||
| p = Pipeline() | ||
| p.add_component(instance=TextFileToDocument(), name="text_file_converter") |
There was a problem hiding this comment.
Not strictly related to this PR, but why inverting the order of the original signature:
def add_component(self, name: str, instance: Component) -> None:
if passing the instance first is more intuitive we're on time to change it!
There was a problem hiding this comment.
I think it's more intuitive, yes. I created an issue here: https://github.com/deepset-ai/canals/issues/137
haystack/preview/components/preprocessors/document_language_classifier.py
Outdated
Show resolved
Hide resolved
haystack/preview/components/preprocessors/document_language_classifier.py
Outdated
Show resolved
Hide resolved
masci
approved these changes
Oct 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
LanguageClassifier#5677 (1st part is feat: Add TextLanguageClassifier 2.0 #6026)LanguageClassifier#5677Proposed Changes:
How did you test it?
new unit tests. e2e test still pending.
Notes for the reviewer
Storing the detected language in the Document's metadata could still be done. Not convinced that we need it at this point. Happy to discuss.
Regarding the e2e test, I am working on getting the other PR #5976 merged first. Then I'll add more assertions.
The component is in the preprocessing module right now but could also fit in the routers. I'd keep it in preprocessors because it not only routes but also detects the language.
Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:.