No documentation on what stopwords are in each set

[soaxelbrooke]

For instance, there is no documentation of what words are in the _english_ stop words set.

[clintongormley]

Yeah, I've tried to put together a list before, but they're quite difficult to round up in Lucene.

[hjc]

I can certainly confirm that these are quite hard to round up. We had a client demand the list for all stop words we were using for every language configured. Here is a quick description on what it took for me to find, and prove, that the list of stop words that ES is using come from org.apache.lucene.analysis.snowball and where to find the files. As you can see, it wasn't fun.

Maybe this is something I can take if I have some free time? Seems like a great beginner ticket and I have experience doing this already!

[clintongormley]

@hjc1710 please do - would be great to have this info in the docs

[ppf2]

https://github.com/apache/lucene-solr/tree/master/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball

[debadair]

We should link to the Lucene docs.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jrodewig]

[docs issue triage]

leaving open as this is still relevant.

[ScottieL]

Hi, I am a beginner who is looking to contribute. This issue seems fairly straightforward. From what I have looked at so far, it seems like the stop words are coming from a combination of different sources such as lucene snowball and the IR Multilingual Resources at UniNE.

french

arabic

I can go ahead and look through all the languages and start compiling a list of words. I have uploaded an example of the arabic stopwords. I cannot obtain a translation for them, but can work towards it. I figured getting a list of all the language's stopwords would be a better first step. Hopefully I am on the right track. Please review the attached file and let me know if this format would satisfy documentation efforts.

arabic_stop.txt

[jrodewig]

@ScottieL Thanks for offering to contribute. Please feel free to open a PR.

As @debadair previously mentioned, I believe that linking to the stop word source will be sufficient. I don't think we need to translate them or necessarily include them verbatim in the Elasticsearch docs.

This will let us point users to the stopwords without getting out of sync.