Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose discard_compound_token option to kuromoji_tokenizer #57421

Merged

Conversation

johtani
Copy link
Contributor

@johtani johtani commented Jun 1, 2020

In Lucene 8.5, Japanese Tokenizer introduced new option DiscardCmpoundToken(LUCENE-9123)

This PR exposes new option to Elasticsearch kuromoji plugin.

And I found 2 bug around Kuromoji Tokenizer.

  1. Fix typo(discartPunctuation) in KuromojiTokenizerFactory
  2. Fix documentation bug in kuromoji_tokenizer extended mode
    • The extended mode output of 関西国際空港 is same as search mode output.

Please let me know, if I should split this PR to new feature and fix typo.

  • Have you signed the contributor license agreement? Yes
  • Have you followed the contributor guidelines? Yes
  • If submitting code, have you built your formula locally prior to submission with gradle check? Yes, with -p plugins/analysis-kuromoji option
  • If submitting code, is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed. Yes, against master
  • If submitting code, have you checked that your submission is for an OS and architecture that we support? Yes
  • If you are submitting this code for a class then read our policy for that. Yes

Fix typo in KuromojiTokenizerFactory
Fix documentation bug in kuromoji_tokenizer extended mode
@cbuescher cbuescher added :Search Relevance/Analysis How text is split into tokens >enhancement labels Jun 2, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/Analysis)

@elasticmachine elasticmachine added the Team:Search Meta label for search team label Jun 2, 2020
@jimczi
Copy link
Contributor

jimczi commented Jun 4, 2020

@elasticmachine ok to test

@jimczi
Copy link
Contributor

jimczi commented Jun 5, 2020

@elasticmachine update branch

Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, It's nice to see you around here again @johtani ;)

@johtani
Copy link
Contributor Author

johtani commented Jun 5, 2020

@jimczi Thanks for reviewing my PR :D

@jimczi jimczi merged commit 9d5409a into elastic:master Jun 5, 2020
jimczi pushed a commit that referenced this pull request Jun 5, 2020
This commit exposes the new Lucene option `discard_compound_token` to the Elasticsearch Kuromoji plugin.
@johtani johtani deleted the expose-kuromoji-discard-compound-token-option branch June 8, 2020 02:50
russcam added a commit to elastic/elasticsearch-net that referenced this pull request Jul 31, 2020
russcam added a commit to elastic/elasticsearch-net that referenced this pull request Aug 4, 2020
github-actions bot pushed a commit to elastic/elasticsearch-net that referenced this pull request Aug 4, 2020
github-actions bot pushed a commit to elastic/elasticsearch-net that referenced this pull request Aug 4, 2020
russcam added a commit to elastic/elasticsearch-net that referenced this pull request Aug 5, 2020
russcam added a commit to elastic/elasticsearch-net that referenced this pull request Aug 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Relevance/Analysis How text is split into tokens Team:Search Meta label for search team v7.9.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants