Integrate ICU Analysis plugin into upgraded search

[safwanrahman]

While upgrading elasticsearch to 6.x in #4211, we skipped integrating the ICU Analysis plugin integration.
This plugin need to be integrated with the upgraded code.
This need to be fixed before #2328

@ericholscher Should we use ICU plugin as analyzer or you have any other in mind?

[ericholscher]

@safwanrahman We use ICU for extended unicode support. I don't believe there is another module that we should be using to do that. It will mostly matter for Asian languages, so we can likely put off the ICU work until we start testing those languages.

[safwanrahman]

@ericholscher the default analyzer has lower case token filter, which already implements case insensitive search functionality. So if we dont add ICU plugin, we already have case insensitive search in the upgraded code. I suspect it will be complex to implement case insensitive search with Elasticsearch ICU plugin.

[ericholscher]

@safwanrahman it looks like you can add the lowercase filter to it if we want, as noted in this example: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-folding.html

[ericholscher]

I think we can leave out the ICU stuff for now, and we can come back to it as an added feature. Once we get english text working well, we can add in ICU & tests for Asian languages, and confirm that the normal english tests don't break around case insensitivity. I think it's added complexity for now that we don't need yet!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

I'm not sure if this issue should be closed or not. Pinging @safwanrahman here.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.