Actually mail the terms notification

[chadwhitacre]

gratipay/gratipay.com#4232 → gratipay/gratipay.com#4282 → …

• Mail all of them, tracking bounces in SQS.
• Manually retrieve bad addresses from the queue—within 14 days!—and drop them to unverified in the database.

[chadwhitacre]

Testing with my account and with a test one that is bouncing. I'm looking for the bounce notification to queue up in SNS.

[chadwhitacre]

Well, SQS via SNS.

[chadwhitacre]

The queue is working! I see delivery notifications in it.

[chadwhitacre]

The subquery for recent exchanges is adding significant run time to queue-branch-email.

[chadwhitacre]

406, as expected. 👍

~ $ queue-branch-email all
Are you ready? [y/N]y
Are you REALLY ready? [y/N]y
... ? [y/N]y
Okay, you asked for it!
406
spotcheck: []
spotcheck: []
spotcheck: []
spotcheck: []
spotcheck: []
But really actually tho? I mean, ... seriously? [y/N]

[chadwhitacre]

Alright! I sent a final working test to myself. Ready? Nothing ventured, nothing gained! 😓

[dmk246]

Ready....set....GO!
You got this @whit537 !!

[chadwhitacre]

slack

[chadwhitacre]

Got mine!

[chadwhitacre]

Email queue is empty!

[chadwhitacre]

We just emailed a notice about our recent Terms of Service change. ✉️ In case you missed it: https://gratipay.news/from-teams-to-projects-45c46718507b. 👍

https://twitter.com/Gratipay/status/818586939850772480

[chadwhitacre]

Hopefully tomorrow I can download the info from the SQS queue and see how many bounces and complaints we got.

[chadwhitacre]

Seems like we're in pretty good shape. 👍

---

---

[chadwhitacre]

The spike back in December is interesting. Security researcher?

[chadwhitacre]

Branch file cleared in gratipay/gratipay.com@da2bd9c.

[chadwhitacre]

http://boto3.readthedocs.io/en/latest/reference/services/sqs.html

[chadwhitacre]

http://boto3.readthedocs.io/en/latest/guide/sqs.html#processing-messages

[chadwhitacre]

Downloading raw json with:

[gratipay] $ run defaults.env local.env ./download-email-status-queue.py
.............................................

---

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import json
import sys

import boto3

sqs = boto3.resource('sqs')
queue = sqs.Queue('https://sqs.us-west-2.amazonaws.com/482083521290/email-status-notifications')

nempty = 0
while nempty < 10:
    nprocessed = 0
    for message in queue.receive_messages():
        try:
            msg = json.loads(message.body)
        except:
            import pdb; pdb.set_trace()
        open('email_status_queue/{}.json'.format(msg['MessageId']), 'w+').write(message.body)
        message.delete()
        sys.stdout.write('.')
        sys.stdout.flush()
        nprocessed = 1
    if nprocessed == 0:
        nempty += 1

[chadwhitacre]

Oh that's right. ☺️

(Pdb) message.body
'Blah blah.'
(Pdb) message.delete()
{'ResponseMetadata': {'HTTPStatusCode': 200, 'RequestId': '41bcbb9e-50dc-566c-b606-eb71059f18ed'}}
(Pdb)

[chadwhitacre]

Got 'em! 👍

[gratipay] $ ls *.json|wc -l
     428
[gratipay] $

[chadwhitacre]

Committed to a local git repo just to be safe ...

[chadwhitacre]

Polled the queue through the AWS console. It's empty. 👍

Gonna leave that so we can keep collecting bounce messages ahead of gratipay/gratipay.com#4284.

[chadwhitacre]

I've deleted the secret key that I used for this.

[chadwhitacre]

Okay! Signed out of AWS. Now to inspect the files and see how many bounces and complaints we got!

[chadwhitacre]

[gratipay] $ ./process-emails.py 
423  Delivery
  4  Bounce
[gratipay] $

---

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import collections
import json
import os


counts = collections.defaultdict(int)


for filename in os.listdir('email_status_queue'):
    if not filename.endswith('.json'): continue
    try:
        raw = open('email_status_queue/'+filename).read()
        message = json.loads(raw)
        message = json.loads(message['Message'])  # Aaaah, Jeff Bezos! :)
    except Exception as exc:
        print(exc)
        import pdb; pdb.set_trace()
    counts[message['notificationType']] += 1

for nmessages, notification_type in reversed(sorted((v,k) for k,v in counts.items())):
    print('{:>3}  {}'.format(nmessages, notification_type))

[chadwhitacre]

Two were me testing, two were legit—we caught two bounces! 💃

[chadwhitacre]

Now I guess we just manually mark those as unverified, ya?

[chadwhitacre]

I'm working up a SQL script to do the unverifying. We don't have that API in Python or HTTP. I'm snagging an add_event example payload to include an unset event.

[chadwhitacre]

Actually, we don't have an event associated specifically with verification (only add and set primary), so I'm not going to worry about that.

[chadwhitacre]

Working with remove_email instead of unverifying ...

[chadwhitacre]

Tested locally with:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import sys

from gratipay import wireup
from gratipay.models.participant import Participant


bad_address = sys.argv[1]
db = wireup.db(wireup.env())

participant = db.one('select p.*::participants from participants p '
                     'where email_address=%s', (bad_address,))
db.run('update participants set email_address=null where email_address=%s', (bad_address,))
participant = Participant.from_id(participant.id)  # refresh self.email_address
participant.remove_email(bad_address)

[chadwhitacre]

Ooh! Even better. :-)

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import sys

from gratipay import wireup


bad_address = sys.argv[1]
db = wireup.db(wireup.env())

participant = db.one('update participants p set email_address=null '
                     'where email_address=%s returning p.*::participants', (bad_address,))
participant.remove_email(bad_address)

[chadwhitacre]

Okay!

heroku run bash

[chadwhitacre]

cat > remove-bad-address.py << EOF
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import sys

from gratipay import wireup


bad_address = sys.argv[1]
db = wireup.db(wireup.env())

participant = db.one('update participants p set email_address=null '
                     'where email_address=%s returning p.*::participants', (bad_address,))
participant.remove_email(bad_address)
EOF

[chadwhitacre]

chmod +x remove-bad-address.py
./remove-bad-address.py []
./remove-bad-address.py []

[chadwhitacre]

Confirmed in the UI for email and events for both participants.

Huzzah! 💃

[chadwhitacre]

I've thrown away the email status queue. We are done here.