Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DuckDuckGo is broken with multiple site: operators #1415

Closed
dgw opened this issue Nov 29, 2018 · 7 comments · Fixed by #2254
Closed

DuckDuckGo is broken with multiple site: operators #1415

dgw opened this issue Nov 29, 2018 · 7 comments · Fixed by #2254
Labels
Bug Things to squish; generally used for issues Low Priority
Milestone

Comments

@dgw
Copy link
Member

dgw commented Nov 29, 2018

This affects both complex user-initiated queries and the xkcd module (see #1410), at least.

DuckDuckGo's plain HTML site, for some reason, no longer returns any results at all if the query includes more than one site: operator. This isn't an issue in Sopel per se, but it does affect some operations. The plain HTML version of DuckDuckGo is how the .duck and .g commands (search.py) get results, and also how the xkcd module searches by comic title.

It seems DDG's Twitter ignored the bug report (sent two weeks ago) and other threads about this on their subreddit. While I'll continue trying to get their attention—and other users also contacting the service (via Twitter, Reddit) about the same issue will help—Sopel might need to change how it retrieves DuckDuckGo results, or start to warn users if they get no results from a query that looks like it would be affected by this issue.

Chances are the warning (along with switching the xkcd module's means of retrieving comics by title) will be done in Sopel 6.6.0 if the bug is not fixed on DuckDuckGo's end. Any changes to how search.py's duck_search() method works will come in Sopel 7, unless it breaks completely.

@dgw dgw added Bug Things to squish; generally used for issues Patches Welcome labels Nov 29, 2018
@dgw dgw added this to the 6.6.0 milestone Nov 29, 2018
@Hanicef
Copy link
Contributor

Hanicef commented Dec 1, 2018

I don't think Twitter is the way to go to report a bug to DuckDuckGo. I've reported the bug on DuckDuckGo's bug report page instead, and hopefully that will give us a better chance of them noticing the bug report.

I also left my email for them to contact me, and I'll report back whenever I get anything back from them.

@dgw
Copy link
Member Author

dgw commented Dec 1, 2018

Hmm… That feedback form looks familiar. I remember also reporting the bug there myself. Did you get any confirmation email, @Hanicef?

Honestly, I pinged DDG via every method I could find when I originally researched the bug a few weeks ago. Not one yielded any acknowledgement or reply. :/

@Hanicef
Copy link
Contributor

Hanicef commented Dec 1, 2018

Well, I just sent it, so I haven't got anything yet. I think we have to wait a few hours before we can expect to get an answer.

It's best to give it a few days tops, and if we haven't got an answer after that, then we can start assuming that they've either missed it or ignored it.

@dgw
Copy link
Member Author

dgw commented Dec 1, 2018

Just wondered if there was a confirmation I could look for in my own email to verify my memory of reporting it, haha. Guess there isn't one.

@dgw
Copy link
Member Author

dgw commented Dec 13, 2018

I think we're well into "start assuming that they've either missed it or ignored it" territory.

The question is, where do we go from here? Besides the contextual warning I started working on, that is.

@dgw dgw modified the milestones: 6.6.0, 7.0.0 Dec 26, 2018
kwaaak pushed a commit to kwaaak/sopel that referenced this issue Mar 25, 2019
The simple way to deal with sopel-irc#1415.

A rewrite isn't in the cards this late in the release cycle for 6.6.0,
so the next best option is to detect when a query is affected by the
bug and advise the user to modify it.
@dgw dgw removed this from the 7.0.0 milestone Nov 16, 2019
@dgw
Copy link
Member Author

dgw commented Nov 16, 2019

There's really nothing we can do about this, short of a rewrite to use an alternate method of accessing DuckDuckGo that isn't affected by the bug on their side. I'll leave this open for tracking, but without any milestone. The warning added in #1418 is Good Enough™.

@dgw
Copy link
Member Author

dgw commented Feb 7, 2022

Further testing during my work on #2247 indicates that this might not be a problem any more.

Using multiple site: operators fails on both DuckDuckGo and Bing, but I was able to use multiple -site: operators on both:

11:21:51 <dgw> .bing team chat  site:xkcd.com -site:almamater.xkcd.com -site:blog.xkcd.com -site:blag.xkcd.com
               -site:forums.xkcd.com -site:fora.xkcd.com -site:forums3.xkcd.com -site:store.xkcd.com
               -site:wiki.xkcd.com -site:what-if.xkcd.com
11:21:52 <Sopel> [search] https://m.xkcd.com/1782/
11:22:04 <dgw> .duck team chat site:xkcd.com -site:almamater.xkcd.com -site:blog.xkcd.com -site:blag.xkcd.com
               -site:forums.xkcd.com -site:fora.xkcd.com -site:forums3.xkcd.com -site:store.xkcd.com
               -site:wiki.xkcd.com -site:what-if.xkcd.com
11:22:05 <Sopel> [search] https://xkcd.com/1782/

Will add to 7.1.8 milestone for tracking. The last thing to do before closing this is probably to add the same warning from #1418 to Bing as well, since there's still inconsistent behavior depending on which search engine is used and how the site: query is written. site:domain1.tld site:domain2.tld fails on both, but (site:domain1.tld OR site:domain2.tld) does seem to work on DDG.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Things to squish; generally used for issues Low Priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants