Restore ability to searching for public body by note body

[gbp]

On /body/list/all searching for public bodies by note body hasn't been working correct since #7212 as its not using the new Note model. In #7269 the spec for this feature has been disabled.

We could switch this search to use Xapian searching and remove a complex SQL query in the process.

[garethrees]

+1 on rationalising the searching.

They currently give different results, which is quite annoying.

• SQL: https://www.whatdotheyknow.com/body/list/all?utf8=%E2%9C%93&public_body_query=cardiff+council&commit=Search
• Xapian: https://www.whatdotheyknow.com/search/cardiff%20council/all

The one benefit of the SQL search was it tended to rank direct name matches higher – see #4426 for lots of discussion on that.

The benefit of the Xapian searching is that you can generally find something if you don't know the exact name:

e.g. searching for "City of Cardiff Council" (which is in the notes field of that body:

• https://www.whatdotheyknow.com/body/list/all?utf8=%E2%9C%93&public_body_query=City+of+Cardiff+Council&commit=Search – 0 results (but likely due to [#4478] New notes model and admin controller #7212)
• https://www.whatdotheyknow.com/search/City%20of%20Cardiff%20Council/all – 5 results (but last in list)
• https://www.whatdotheyknow.com/search/%22City%20of%20Cardiff%20Council%22/all – 1 result (quoted search term).

Either way, I think having fewer search mechanisms is generally better, and then we can spend the time tuning the search engine.

[gbp]

Yes the SQL search is given better results I wonder if we could do something like #7553 to improve Xapian. Would need to consider how to update our large Xapian indexes.

[garethrees]

I wonder if we could do something like #7553 to improve Xapian

Looks like something we considered last time we looked at this, but can't immediately see any evidence that we tried it #4426 (comment)

[garethrees]

This appears to be the case right now. I just dealt with a case identical to #4426 (comment) (NHS England not appearing for the first 10 pages). The search results were all highlighting matches of "NHS England" in notes.

Is this just because these bodies haven't been reindexed since #7212? That doesn't seem true, as e.g. this authority was last updated at 2023-08-28, so presumably long after.

[HelenWDTK]

This issue has been automatically closed due to a lack of discussion or resolution for over 12 months.
Should we decide to revisit this issue in the future, it can be reopened.