Add a config property/env variable specific index field

[marko-bekhta]

I was looking into Max's comment quarkusio/quarkusio.github.io#1825 (comment) and I've tried a few things like shingle filter and some other stuff..
The idea I've ended up with is that we add a config property/env variable specific field. Using a regex tokenizer we'll put only tokens that are config properties or env variables. Then, while searching, we'd use a keyword tokenizer on this particular field, which would mean if we find a match there we can boost it with > 1 boost since the search term seems to be looking suspiciously similar to a config property.

[yrodiere]

I need to have a closer look, but first: why don't we currently have a match exactly? I'd expect quarkus.websocket.max-frame-size to match quarkus.websocket.max-frame-size even if it's tokenized at some point... There's something odd going on, no?

[marko-bekhta]

hey, yes sure.
So the original problem (the odd part happening) is the content extraction part, basically the changes from the second commit are to solve that specific issue:

-            Element content = body.selectFirst(".content .grid__item");
+            Element content = body.selectFirst(".guide");

if you go to a quarkus page https://quarkus.io/guides/all-config and then try running a selection query (that is currently used) from a dev tools console document.querySelector('.content .grid__item') you'd get a

Quarkus is open. All dependencies of this project are available under the [Apache Software License 2.0](https://www.apache.org/licenses/LICENSE-2.0) or compatible license.

section from the footer ... so it just didn't index anything from the page, hence no results.
then the other commit with a regex tokenizer and everything else ... I was just thinking that usually those env variables or config properties are long and people may be searching only by part of them so that is what I was trying to cover there 😃

[yrodiere]

I see, thanks for the explanation. I'd have expected just raising the max_gram on the existing autocomplete analyzer to be enough, but if not, and if this does not make indexing outrageously slow... +1 let's do this.

I'll try this locally and then will merge.

[marko-bekhta]

yeah, give it a try and see what you think 😃. I wasn't sure about increasing the max on the current analyzer, since it would probably breakup more words than just properties... but then I didn't really check how much relly loooooong words we have in the guides 😃 . And then there's also the fact that a config property would be tokenized and not treated as a single keyword... I was thinking: if I'm searching for a config and start with something like quarkus.foo-bar then I probably would want a result where a property is present to show up higher than a doc where we just have foo and some text then bar in Quarkus

*EDIT: oh and because of that regex tokenizer it should only include the env variables and config properties, no other text

[yrodiere]

Ok, so this almost doubles the size of indexes (from 500MB to 1GB), which is kind of odd considering the new field supposedly only indexes config properties...

But I guess that's okay, since we have more than enough room to accommodate these relatively still small indexes (see #119 :x )

Will merge, thanks!