[scanner integration] Support landing page redirects on the same domain

[eloquence]

Some news organizations prefer to advertise URLs like https://nytimes.com/tips, which they'd also like us to use for their directory entries, and which redirect to the ultimate landing page destination. To avoid false scan results, the scanner should:

• follow 301/302 redirects on the same domain
• store the redirect destination in the scan result

This is part of epic #488 as non-200 statuses may otherwise trigger the delisting of a SecureDrop instance (#493).

[harrislapiroff]

What's the desired behavior if it redirects to a different domain? Should we count redirects to subdomains as the same domain?

[eloquence]

What's the desired behavior if it redirects to a different domain?

IMO this should be a hard failure, i.e. in the same category as, say, the site not being on HTTPS. We should not include such redirects in the directory.

Should we count redirects to subdomains as the same domain?

I think so. Of course we would penalize the use of subdomains as usual (#497).

[chigby]

I'm thinking that a good solution for this is:

1. The landing_page_url field remains as it is now. If the NYT wants to set this to https://nytimes.com/tips then that is fine.

2. We add a field to the Scan Result called, say, redirection_target.

3. When scanning a directory entry, we will follow redirects and save the URL from the final destination as the redirection target.

4. Use pshtt to scan the domain for the redirection target.

5. Replace the http_no_redirects scan result field with something like no_cross_domain_redirects which would be set to False if any of the intermediate redirections are not on the same top-level domain as the landing_page_url. This would also trigger a delisting.

Can anyone think of something I might be missing here?

[chigby]

This issue is resolved by #549.