Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug/ingest language should not default to 'eng' but None #1715

Closed
rbiseck3 opened this issue Oct 11, 2023 · 1 comment · Fixed by #1729
Closed

bug/ingest language should not default to 'eng' but None #1715

rbiseck3 opened this issue Oct 11, 2023 · 1 comment · Fixed by #1729
Assignees
@rbiseck3
Labels
bug Something isn't working

Comments

@rbiseck3
Copy link
Contributor

Describe the bug
By defaulting to eng, this actually injects information that might not be accurate and breaks the language detection library being used. This should instead default to None and let the library do what it needs to to detect the language.

To Reproduce
Running local connector on the example-docs/language-docs/UDHR_first_article_all.txt file produces only english languages.

Expected behavior
Running local connector on the example-docs/language-docs/UDHR_first_article_all.txt should result in ['ind', 'est']
@rbiseck3 rbiseck3 added bug Something isn't working ingest labels Oct 11, 2023
@rbiseck3 rbiseck3 self-assigned this Oct 11, 2023
@Coniferish
Copy link
Collaborator

*Commenting to track this

@rbiseck3 rbiseck3 linked a pull request Oct 12, 2023 that will close this issue
roman/bugfix fix default language ingest option #1729
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rbiseck3 rbiseck3

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roman/bugfix fix default language ingest option Unstructured-IO/unstructured
2 participants
@Coniferish @rbiseck3