search does not find results for distopia (and others...)

[orbeckst]

I searched for "distopia" and "CalcBondsOrtho" and no hits (even though its sitemap has been added in #201 (Issue #200) .

Probably similar issues as #199 but not clear yet why the algolia search does not seem to index these parts of the site. Perhaps the config options need to be adjusted.

EDIT: While investigating the issue it is becoming clear that other docs are also not being indexed or the index is not being updated. I am making this issue a catch all.

[orbeckst]

I fixed a missing tag in eb82c82 but I doubt that this will fix everything... let's give it 24h.

[hmacdope]

Thanks @orbeckst! I am not much help here ...

[orbeckst]

I raised algolia/docsearch-configs#4699, perhaps we can get some help from the algolia folks.

I also created PR algolia/docsearch-configs#4700 to include code samples (pre tag) and definition items in definition lists (dt tags) in the index.

[orbeckst]

I installed the docsearch-scraper locally and I’m able to run it so I can now debug more easily.

[orbeckst]

Well... maybe not that simple:

$ ./docsearch run ../docsearch-configs/configs/mdanalysis.json
...
algoliasearch.exceptions.RequestException: Record quota exceeded. Change plan or delete records.

Nb hits: 10415
previous nb_hits: 85975

Will need to see how to work within these limitations.

[hmacdope]

I don’t know much about algolia but could a lack of quota be why it’s not being indexed?

On Sat, 9 Oct 2021 at 2:05 pm, Oliver Beckstein ***@***.***> wrote: Well... maybe not that simple: $ ./docsearch run ../docsearch-configs/configs/mdanalysis.json ... algoliasearch.exceptions.RequestException: Record quota exceeded. Change plan or delete records. Nb hits: 10415 previous nb_hits: 85975 Will need to see how to work within these limitations. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#202 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF3RHC6P76S6BRLSBOZHBUDUF6WORANCNFSM5E27VQHA> .

-- *Hugo MacDermott-Opeskin* PhD Candidate, RSC ANU Email: ***@***.*** ***@***.***>

[orbeckst]

I don't think this applies when you're running through their infrastructure as an approved open source documentation site.

What I am trying is to use their "commercial" analytics infrastructure on the free plan with our own scraped index. I am really only interested in running the scraper and seeing how changing the config file changes what it picks up. If I find some time I'll just try to disable the sending of results and retain the local index building.

[orbeckst]

To run docsearch without commiting changes to the actual index, see orbeckst/docsearch-scraper#1 for the required code change.

[orbeckst]

Sitemaps for multiple projects are now broken because a release string was inserted into the URL even though this does not reflect the deployment URL. My suspicion is that something changed in the sphinx_sitemap plugin or in our Sphinx configuration.

It seems unlikely that this is due to the GH actions workflow because PMDA (which has been using Travis and not actions) also hast the same problem.

[orbeckst]

Looking at the docs for the sitemap plugin, the solution appears to be set in conf.py

sitemap_url_scheme = "{link}"

so that the version is not included.

[orbeckst]

New PR algolia/docsearch-configs#4751 but there are still many pages that are not showing up. See the PR for notes.

[orbeckst]

With #211 , the changes to the config are done through the Crawler web interface. Initial testing indicates that even after switching to v3 (see PR #212) we still have the same issues that are noted here. Furthermore, I see multiple versions of the docs (1.0.1, 1.1.1, 2.0.0, ...) show up. This should really only be stable.