We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
setdefaulttimeout
This issue when running selenium-wire in scrapy cloud is caused because of a call to socket.setdefaulttimeout in sh_scrapy.crawl.
selenium-wire
socket.setdefaulttimeout
sh_scrapy.crawl
The related issue in pyOpenSSL pyca/pyopenssl#168
pyOpenSSL
The cause of the issue is this call https://github.com/scrapinghub/scrapinghub-entrypoint-scrapy/blob/master/sh_scrapy/crawl.py#L27
A minimal example to reproduce the issue
Clone https://github.com/pawelmhm/quotesbot/tree/selenium
Working case
cd quotesbot docker built -t selenium-wire-scrapy-issue . docker run selenium-wire-scrapy-issue scrapy crawl toscrape-css
It should finish properly scraping 100 items and no errors
Now, update settings.py and add the following to the top of the file
settings.py
import socket socket.setdefaulttimeout(60)
Then rebuild and rerun the container
docker built -t selenium-wire-scrapy-issue . docker run selenium-wire-scrapy-issue scrapy crawl toscrape-css
Now it should fail, scraping only 10 items and showing an error net::ERR_SSL_PROTOCOL_ERROR in the logs
net::ERR_SSL_PROTOCOL_ERROR
The text was updated successfully, but these errors were encountered:
I'm using selenium wire which requires certificate + key. I just followed https://docs.zyte.com/misc/ca.html for certificate creation but key is out of discussion. According to this https://github.com/wkeeling/selenium-wire#using-your-own-certificate, selenium Wire sets it mandatory for my own certificates. And now I'm getting 127.0.0.1:61202: Invalid certificate, closing connection. Pass --ssl-insecure to disable validation.
127.0.0.1:61202: Invalid certificate, closing connection. Pass --ssl-insecure to disable validation.
Sorry, something went wrong.
No branches or pull requests
This issue when running
selenium-wire
in scrapy cloud is caused because of a call tosocket.setdefaulttimeout
insh_scrapy.crawl
.The related issue in
pyOpenSSL
pyca/pyopenssl#168The cause of the issue is this call
https://github.com/scrapinghub/scrapinghub-entrypoint-scrapy/blob/master/sh_scrapy/crawl.py#L27
A minimal example to reproduce the issue
Clone https://github.com/pawelmhm/quotesbot/tree/selenium
Working case
It should finish properly scraping 100 items and no errors
Now, update
settings.py
and add the following to the top of the fileThen rebuild and rerun the container
Now it should fail, scraping only 10 items and showing an error
net::ERR_SSL_PROTOCOL_ERROR
in the logsThe text was updated successfully, but these errors were encountered: