Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setdefaulttimeout is causing issues with pyOpenSSL #62

Open
ejulio opened this issue Sep 28, 2021 · 1 comment
Open

setdefaulttimeout is causing issues with pyOpenSSL #62

ejulio opened this issue Sep 28, 2021 · 1 comment

Comments

@ejulio
Copy link
Contributor

ejulio commented Sep 28, 2021

This issue when running selenium-wire in scrapy cloud is caused because of a call to socket.setdefaulttimeout in sh_scrapy.crawl.

The related issue in pyOpenSSL pyca/pyopenssl#168

The cause of the issue is this call
https://github.com/scrapinghub/scrapinghub-entrypoint-scrapy/blob/master/sh_scrapy/crawl.py#L27

A minimal example to reproduce the issue

Clone https://github.com/pawelmhm/quotesbot/tree/selenium

Working case

cd quotesbot
docker built -t selenium-wire-scrapy-issue .
docker run selenium-wire-scrapy-issue scrapy crawl toscrape-css

It should finish properly scraping 100 items and no errors

Now, update settings.py and add the following to the top of the file

import socket
socket.setdefaulttimeout(60)

Then rebuild and rerun the container

docker built -t selenium-wire-scrapy-issue .
docker run selenium-wire-scrapy-issue scrapy crawl toscrape-css

Now it should fail, scraping only 10 items and showing an error net::ERR_SSL_PROTOCOL_ERROR in the logs

@Foxtrod89
Copy link

Foxtrod89 commented Jun 12, 2023

I'm using selenium wire which requires certificate + key. I just followed https://docs.zyte.com/misc/ca.html for certificate creation but key is out of discussion.
According to this https://github.com/wkeeling/selenium-wire#using-your-own-certificate, selenium Wire sets it mandatory for my own certificates.
And now I'm getting 127.0.0.1:61202: Invalid certificate, closing connection. Pass --ssl-insecure to disable validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants