Added default stopwords and link to Lucene #41173
Conversation
kat257
added
>docs
|
Pinging @elastic/es-search |
| list of stop words. Defaults to `_english_`. | ||
|
|
||
| `stopwords_path`:: | ||
|
|
||
| The path to a file containing stop words. This path is relative to the | ||
| Elasticsearch `config` directory. | ||
|
|
||
| The default English stopwords used in Elasticsearch are: | ||
|
|
||
| a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, |
There was a problem hiding this comment.
Is this still the case?
I know Lucene has a file of EN stop words here:
https://github.com/apache/lucene-solr/blob/master/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/english_stop.txt
Might be worth checking with an engineer in @elastic/es-search.
There was a problem hiding this comment.
It also may make more sense to move this section under the stopwords param. Or link to this section from there.
There was a problem hiding this comment.
The list looks good, the file you linked is for the snowball filters and analyzers but we don't use them in the stop filter. The default list can be found here: https://github.com/apache/lucene-solr/blob/master/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/EnglishAnalyzer.java#L46
| for that language. | ||
|
|
||
| Stop words in other supported languages can be accessed in Lucene: | ||
| https://github.com/apache/lucene-solr/tree/master/lucene/analysis/common/src/resources/org/apache/lucene/analysis[Stopwords] |
There was a problem hiding this comment.
It might be worth mentioning that some of these are under the snowball path:
https://github.com/apache/lucene-solr/tree/master/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball
There was a problem hiding this comment.
The list of stopwords for each language is different from snowball. You can find the entire list here:
https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/index/analysis/Analysis.java#L115
| words, you can reduce the size of an index and increase performance, which is | ||
| important when disk space and memory are critical factors. | ||
|
|
||
|
|
||
|
|
||
| [float] |
There was a problem hiding this comment.
Probably want to add an anchor here like [[stop-analyzer-example-output]]
There was a problem hiding this comment.
Stop words are common words set not to be indexed. By removing a list of common
words, you can reduce the size of an index and increase performance, which is
important when disk space and memory are critical factors.
This section is a bit misleading since we added optimizations in Lucene 8 to speed up queries that contain stop words. Removing stop words is also challenging for other token filters like synonym_graph and synonym so now that we are able to run queries with stop words efficiently I am not sure we should add such advice in the docs. We could maybe just mention the disk space savings and add a note regarding the faster search if the total hits that match the query is not requested (making the removal of stop words not necessary) ?
| they, this, to, was, will, with | ||
|
|
||
|
|
||
| ==== Stop words in supported languages |
There was a problem hiding this comment.
Add an anchor here as well.
| Each language analyzer defaults to using the appropriate stopwords list | ||
| for that language. | ||
|
|
||
| Stop words in other supported languages can be accessed in Lucene: |
| list of stop words. Defaults to `_english_`. | ||
|
|
||
| `stopwords_path`:: | ||
|
|
||
| The path to a file containing stop words. This path is relative to the | ||
| Elasticsearch `config` directory. | ||
|
|
||
| The default English stopwords used in Elasticsearch are: |
| list of stop words. Defaults to `_english_`. | ||
|
|
||
| `stopwords_path`:: | ||
|
|
||
| The path to a file containing stop words. This path is relative to the | ||
| Elasticsearch `config` directory. | ||
|
|
||
| The default English stopwords used in Elasticsearch are: | ||
|
|
||
| a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, |
There was a problem hiding this comment.
The list looks good, the file you linked is for the snowball filters and analyzers but we don't use them in the stop filter. The default list can be found here: https://github.com/apache/lucene-solr/blob/master/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/EnglishAnalyzer.java#L46
| words, you can reduce the size of an index and increase performance, which is | ||
| important when disk space and memory are critical factors. | ||
|
|
||
|
|
||
|
|
||
| [float] |
There was a problem hiding this comment.
Stop words are common words set not to be indexed. By removing a list of common
words, you can reduce the size of an index and increase performance, which is
important when disk space and memory are critical factors.
This section is a bit misleading since we added optimizations in Lucene 8 to speed up queries that contain stop words. Removing stop words is also challenging for other token filters like synonym_graph and synonym so now that we are able to run queries with stop words efficiently I am not sure we should add such advice in the docs. We could maybe just mention the disk space savings and add a note regarding the faster search if the total hits that match the query is not requested (making the removal of stop words not necessary) ?
| for that language. | ||
|
|
||
| Stop words in other supported languages can be accessed in Lucene: | ||
| https://github.com/apache/lucene-solr/tree/master/lucene/analysis/common/src/resources/org/apache/lucene/analysis[Stopwords] |
There was a problem hiding this comment.
The list of stopwords for each language is different from snowball. You can find the entire list here:
https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/index/analysis/Analysis.java#L115
jimczi
changed the title
Closes #16561