[scanner integration] Support "delisted" state for directory entries

[eloquence]

As part of epic #488, we should support delisting individual instances from the directory in a manner which

• removes them from searches/dropdowns
• replaces an instance page with an "Under Review" page.
• associates the reason for delisting (from a fixed set including "other") with the directory entry (we may not with to display it for security reasons, but we need it for alerting and auditing)

It must be possible to set this state both in a manual and automated fashion, as some landing page scan results may trigger automated delisting.

[eloquence]

Was moved to "Dev" nearly a month ago but no PR open yet; not keeping this in the 7/25-8/8 sprint given Harris' workload.

[harrislapiroff]

Should we still display the onion url and landing page if an entry has been delisted? I can think of a few possible UIs here:

1. do not display the onion URL or landing page link if the entry has been delisted. We do not want people visiting.
2. display the onion URL and link highlighted in red below the warning to indicate the danger
3. provide a <details> disclosure saying basically "I understand the risk, show me instance details anyway"

[conorsch]

Strongly prefer 1). For instances that trigger the delisting flow, it's generally a problem like plain HTTP on the landing page—in which case we cannot trust the Onion URL displayed on the org's landing page, so omitting the information strikes me as safest.

cc @emkll @redshiftzero for further input.

[eloquence]

Agree with @conorsch - we don't want to display any information at all in these cases. These are severe cases, e.g., where the landing page is loading without HTTPS, is redirecting in unexpected ways, etc. Given this, I don't think it would be appropriate to show information at all if we have reason to suspect that it may have been compromised, or that source security would be significantly impacted by visiting that landing page.

[harrislapiroff]

Sounds great. That also happens to be the easiest option. 👍✨

[redshiftzero]

yep, since these are the most severe issues, I agree with not displaying at all 👍

[eloquence]

For quick reference, per current plan, these are the issues that would trigger an automatic de-listing:

• [scanner integration] Automatically delist instances with major landing page HTTPS issues #492 - no HTTPS or mixed-content warning (presuming we can detect the latter at some point)
• [scanner integration] Automatically delist instances with landing pages served with non-200 HTTP status #493 - non-200 status after legitimate redirects are resolved

Down the road we may also wish to consider:

• [scanner integration] Automatically delist instances with extended downtime #504 - extended downtime of the actual SD instance, pending integration of a tool like sdstatus

As noted top-level, there may be other reasons to manually de-list, especially until we have sdstatus integration.