Allow for subdomains when determining if to allow messages to appear on a thread set to authority_only

[RichardTaylor]

Requests to the University of Edinburgh on WhatDoTheyKnow are sent to an email address at the domain

@ed.ac.uk

The University of Edinburgh sends responses from a domain:

@informationrequest.ed.ac.uk

currently when a thread is set to authority_only responses don't get through.

[mdeuk]

+1 - The University of Edinburgh aren't the only public body to do this, and it can be a nuisance both to admins and to public bodies themselves.

Conversely however, there is a possible risk that this would allow for spoofing (e.g. some Universities provide student email on a sub-domain), but the risk of nefarious usage is perhaps unlikely and could be handled on a case-by-case basis.

[RichardTaylor]

there is a possible risk that this would allow for spoofing (e.g. some Universities provide student email on a sub-domain)

I too considered this, however in these cases the threads have previously been open to accept responses from anyone.

This closure to "authority only" is a spam reduction / spam incident mitigation measure, not the primary way we assure responses are from a public body and not spoofed. The secret part of the request address is one of the key ways spoofing is prevented.

An extreme way to resolve this ticket would be to always let .ac.uk and .gov.uk responses through when threads are set to authority_only. Perhaps that would be OK if we could use other means to prevent spoofing by spammers eg. if we had a DKIM / SPF authenticated .gov.uk or .ac.uk email we could let that through?

[garethrees]

I think the solution here is to create an association on PublicBody that stores "allowed domains" or "alias domains", or something to that effect.

Then in the AuthorityOnly gatekeeper we'd need to add some logic where we allow the message if the sender_domain is included in the "allowed domains" for that body.

diff --git a/app/models/info_request/response_gatekeeper/authority_only.rb b/app/models/info_request/response_gatekeeper/authority_only.rb
index 27d3b8e78..44e1b8293 100644
--- a/app/models/info_request/response_gatekeeper/authority_only.rb
+++ b/app/models/info_request/response_gatekeeper/authority_only.rb
@@ -27,6 +27,11 @@ def calculate_allow_reason(info_request, email)
           end
         end

+        if info_request.public_body.authority_domain?(sender_domain)
+          allow = true
+          reason = nil
+        end
+
         [allow, reason]
       end
     end

This would mean that there'd be no change in behaviour for authorities by default, but where we get notified of an edge case like this, we take human action to work around it if necessary and limit the risk of impersonation.

To make this a quicker change we could opt not to include an admin interface, unless we find ourselves needing to use the functionality repeatedly. Perhaps we'd want to at least render out any manually added allowed domains so that it's clear that they're in operation, but skip the editing forms to save a bit of time.

[garethrees]

Related to #427 and #5363.

[mdeuk]

I think the solution here is to create an association on PublicBody that stores "allowed domains" or "alias domains", or something to that effect.

I like this idea - it allows us to make a change (relatively) quickly where it is required, but it also provides us with adequate reassurance that we can protect against misuse in scenarios where an 'all or nothing' approach would be unsuitable.

To make this a quicker change we could opt not to include an admin interface, unless we find ourselves needing to use the functionality repeatedly. Perhaps we'd want to at least render out any manually added allowed domains so that it's clear that they're in operation, but skip the editing forms to save a bit of time.

It'd certainly be desirable if it was configurable via the admin console - and that it supported multiple locales; for example: we might wish to enable a body such as the Senedd to use both the English and Welsh variations of their domain simultaneously when setting an allowed sub-domain. That's another edge case perhaps, but it may help for internationalisation.

[RichardTaylor]

limit the risk of impersonation.

I don't think the authority_only setting is related to preventing impersonation. Requests set to authority_only will have previously open to responses from anyone (anyone with the request address).

My understanding is:

This closure to "authority only" is a spam reduction / spam incident mitigation measure, not the primary way we assure responses are from a public body and not spoofed. The secret part of the request address is one of the key ways spoofing is prevented.

[garethrees]

don't think the authority_only setting is related to preventing impersonation…

No, but allowing e.g. all .ac.uk domains to be considered "authority" domains in order to work around the described issue. would increase the risk of it for certain bodies.

[mdeuk]

There have been a couple of requests recently, where it would be helpful to allowlist ¹ domains used by orgs using case management systems, as the replies are issued off the vendor's domain / subdomain.

In this setup, the message typically takes the format of:

From:	[email protected]
Envelope:	[email protected]
Reply-to:	[email protected]

… meaning that we match authority_only against the From / envelope, and subsequently, send any bounce back to [email protected].

This isn't great, as it causes friction for the user, and extra admin for support teams ².

Ideally, being able to configure this at authority level would be preferable; but, I suppose a hacky interim solution of sorts could be to use a list similar to the ReplyToAddressValidator.invalid_reply_addresses list in model_patches.rb, as an interim step ³.

Related:

• Check capitalisation doesn't impact which domains are permitted to respond to authority_only threads #7354
• Allow users to receive a notification where a response to a request has bounced/been rejected #7589
• Add authority email aliases to improve holding pen guesses #4942

Footnotes

1. Neutral term - language matters! ↩

2. along with the potential that the organisation, depending on their vendor's setup, may have no knowledge of the fact that their email bounced, as they might end up not receiving the email at all. ↩

3. We would, of course, need to be careful with what was put here, so as not to create the hypothetical issue described above re whitelisting at top level domains (e.g. we'd probably never want to add *.gov.uk or *.com to a list) ↩

[garethrees]

We do have this capability of sorts, in that (at least as documented!) we allow responses from any domain that has already replied to a thread when "anybody" was allowed to reply:

https://github.com/mysociety/alaveteli/blob/0.43.0.0/app/models/info_request/response_gatekeeper/authority_only.rb#L20-L27

What's happened in these cases? They were so late in responding that we'd moved to authority_only before we'd received a response from the service provider domain?

being able to configure this at authority level would be preferable; but, I suppose a hacky interim solution of sorts could be to use a list similar to the ReplyToAddressValidator.invalid_reply_addresses list in model_patches.rb

I think we could do this in a patched way (and is preferable before we build it into a fully fleshed out admin UI, so that we can battle-test it and judge how frequely we find we want to do it)… something like:

# Aliases mapped to ID of PublicBody record
PUBLIC_BODY_DOMAIN_ALIASES = {
   # Department for Foobar
  1234 => %w[
    orgname.vendor.tld
    cy-orgname.vendor.tld
  ]
  # Department for Humpadinking
  987  => %w[
    other.tld
  ]
}

# our hacky method that patches something
# I think InfoRequest#who_can_followup_to is probably what we want…

def who_can_followup_to
  super

  # Find relevant domains: PUBLIC_BODY_DOMAIN_ALIASES[public_body.id]
  # Add them to the return array
end

# The gotcha here is that currently `InfoRequest#who_can_followup_to` is returning full
# email addresses, and then the `InfoRequest::ResponseGatekeeper::AuthorityOnly` is
# doing the work to find the domain. This might be fine as is, if 
# `PublicBody.extract_domain_from_email` works fine if given a domain. If not there'd
# be a bit more work to clean some of this up.

[HelenWDTK]

This issue has been automatically closed due to a lack of discussion or resolution for over 12 months.
Should we decide to revisit this issue in the future, it can be reopened.