- fixed proxy authentication issue.
- fixed OverflowError during backoff computation.
- redirects with empty bodies are no longer considered bans (thanks Diga Widyaprana).
ROTATING_PROXY_BAN_POLICY
option allows to customize ban detection for all spiders.
max_proxies_to_try
request.meta key allows to overrideROTATING_PROXY_PAGE_RETRY_TIMES
option per-request.
- Update default ban detection rules: scrapy.exceptions.IgnoreRequest is not a ban.
- changed
ROTATING_PROXY_PAGE_RETRY_TIMES
default value - it is now 5.
- improved default ban detection rules;
- log ban stats.
Initial release