Add script to fix stale cached values

[rohitpaulk]

Earlier (in Gratipay 1.0), we used to fund tips using the Gratipay balance that participants had. We used to refresh these values after every Payday.

Now that we've moved to charging in arrears and never using one's balance to fund tips - the old 'funded' values have to be corrected.

This was picked off #3876 (comment) (#3876 (comment) for reference)

[chadwhitacre]

This was picked off #3876 (comment) (#3876 (comment) for reference)

Pretty sure those are the same comment. 📦

[chadwhitacre]

sql/branch.py is a little goofy. :)

This doesn't strictly relate to #3981, correct? We could run this without touching that?

[chadwhitacre]

Anyway, running this script is low-risk, since all it does is call update_giving_and_teams over a subset of users. Running ...

[chadwhitacre]

Script started.

$ heroku config -s | run - ./env/bin/python sql/branch.py > fix-stale-cached-values.log

(run)

[chadwhitacre]

~8 minutes left

[chadwhitacre]

Blech. Network troubles, trying again ...

$ heroku run bash
Running bash on gratipay.... up, run.3297
~ $ ed
a
from gratipay import wireup

db = wireup.db(wireup.env())

participants = db.all("""
    SELECT p.*::participants
      FROM participants p
      JOIN payment_instructions pi ON pi.participant = p.username -- Only include those with tips
""")
total = len(participants)
counter = 0

for p in participants:
    print("Participant %s/%s" % (counter, total))
    p.update_giving_and_teams()
    counter += 1
.
w branch.py
404
q
~ $ python branch.py
...
Participant 1975/3508
Participant 1976/3508
Participant 1977/3508
Participant 1978/3508
Participant 1979/3508
Participant 1980/3508
Participant 1981/3508
Participant 1982/3508
Participant 1983/3508
Participant 1984/3508
Participant 1985/3508
Participant 1986/3508
Participant 1987/3508
Participant 1988/3508
Participant 1989/3508
Participant 1990/3508
Participant 1991/3508
Participant 1992/3508
Participant 1993/3508
Participant 1994/3508
Participant 1995/3508
Participant 1996/3508
Participant 1997/3508
Participant 1998/3508
Participant 1999/3508
Participant 2000/3508
Participant 2001/3508
Participant 2002/3508
Participant 2003/3508
Participant 2004/3508
Participant 2005/3508
Participant 2006/3508
Participant 2007/3508
Participant 2008/3508
Participant 2009/3508
Participant 2010/3508
Participant 2011/3508
Participant 2012/3508
Participant 2013/3508
Participant 2014/3508
Participant 2015/3508
Participant 2016/3508
Participant 2017/3508
Participant 2018/3508
Participant 2019/3508

[chadwhitacre]

Much zippier. :-)

[chadwhitacre]

Alright, script is done.

[chadwhitacre]

Actually, we still have 4 records matching this query (vs. 31 on a recent backup):

=> SELECT amount, due, is_funded FROM current_payment_instructions WHERE due > 10 AND amount > 0 AND is_funded;
┌────────┬───────┬───────────┐
│ amount │  due  │ is_funded │
├────────┼───────┼───────────┤
│   3.00 │ 90.00 │ t         │
│   1.00 │ 30.00 │ t         │
│   1.25 │ 37.50 │ t         │
│   1.00 │ 30.00 │ t         │
└────────┴───────┴───────────┘
(4 rows)

[rohitpaulk]

For all of the 4 records -

1. They have CCs attached at braintree, the exchange_routes entries have no errors.
2. They have never been charged (no exchanges entries).

[rohitpaulk]

Are we adding to these due values every week?

If no: When and why did we stop doing so? (#3980 (comment))
If yes: Why aren't we charging them?

[rohitpaulk]

Are we adding to these due values every week?

Yes. The last addition to due for each of those participants was 31st March (the last payday)

[rohitpaulk]

Looking at payday_participants for the last payday - has_credit_card is true for all users, and card_hold_ok is false.

[rohitpaulk]

Aha. is_suspicious is false NULL for all of these users. i.e. they hadn't been whitelisted through the review dashboard. They had tips set up, and they were ready to give. Time to hit #3937

[chadwhitacre]

!m @rohitpaulk

[rohitpaulk]

What's left:

1. Review and land Fix review dashboard, fixes #3937 #3984
2. Review the users on the dashboard (only 5 of them, should be quick)
3. Let the next payday run it's course - the users will be charged and the dues will settle down.

[rohitpaulk]

The high charges could be a bit of a surprise to these users though. Should we email them with a warning?

[mattbk]

Not super clear on what is happening, but I can send some emails. Do I know how much each person is about to be charged?

[chadwhitacre]

@mattbk Awesome, thank you. Check the "Giving" page for each user to see due.

[chadwhitacre]

@mattbk Maybe post a draft of the email you're planning to send here so that @rohitpaulk and I can review it?

[mattbk]

Subject: Gratipay charges will be larger than normal

Hi ~user,

We messed up. You've been giving for a while now, but we haven't been charging your credit card. This means you will be charged $___ next Thursday (April 14), and this money distributed to the teams that should have been receiving it all along.

We understand that this may be more than you're willing to give all at once. Please reply if you approve this charge. If you don't approve, we will reset your amount due to zero, and future charges will depend on your current giving, as they should have all along.

Sorry for the mistake, and thanks for using Gratipay!
Matt

[chadwhitacre]

!m @mattbk

Almost perfect—

Thanks for supporting Gratipay!

This makes it sound like they're giving money to Gratipay. How about "Thanks for using Gratipay!" instead?

[rohitpaulk]

yeah, there's only five of them

Note - I think it's only four... The fifth person should have a due < $10 if I'm not wrong.

[chadwhitacre]

We never actually say "Sorry"—how about "Sorry for the mistake, and thanks for using Gratipay!"

[mattbk]

Email sent to three out of four ~users. Fourth user (ID 467453) has no email on file, so I have reached out via Twitter for him to get in touch (doesn't follow us so can't DM).

[mattbk]

Fourth user (ID 467453) has no email on file, so I have reached out via Twitter for him to get in touch (doesn't follow us so can't DM).

He got back via email (FD), so I gave him the message.

[mattbk]

One user has requested a reset. Will update here if anyone requests that we do charge his or her card.

[chadwhitacre]

@mattbk Can you link the Freshdesk tickets here?

[mattbk]

See previous post. Same person.

[rohitpaulk]

• https://gratipay.freshdesk.com/helpdesk/tickets/4600 - Requested a reset. @rohitpaulk manually set due to zero in the DB
• https://gratipay.freshdesk.com/helpdesk/tickets/4599 - Looks like user canceled their tip..
• https://gratipay.freshdesk.com/helpdesk/tickets/4598
• https://gratipay.freshdesk.com/helpdesk/tickets/4597 - Fine with the charge. @rohitpaulk approved the user via the review dashboard.

[rohitpaulk]

I suggest that we keep this open till next payday - and if we don't receive a response by then, we'll reset the due to 0 for the user in question.

[mattbk]

I suggest that we keep this open till next payday - and if we don't receive a response by then, we'll reset the due to 0 for the user in question.

That's what I wrote in the email: no response means "reset me to zero."

[chadwhitacre]

I suggest that we keep this open till next payday - and if we don't receive a response by then, we'll reset the due to 0 for the user in question.

Any response here, @mattbk?

[mattbk]

No response.

[chadwhitacre]

@rohitpaulk Can we close the books here?

[chadwhitacre]

@rohitpaulk Bump. What's our status here?

[chadwhitacre]

Adding to my plate to get to the bottom of.

[chadwhitacre]

@rohitpaulk Do you remember what this was about? Are we good to close?

[rohitpaulk]

Nope, but I'll take a look at this soon.

[rohitpaulk]

Ran the same query again - there are matches, but all of these are users marked as suspicious = true (the issue here was with suspicious = NULL). Looks like that one rogue case got sorted out somehow?

The dues that match the current query will be marked as unfunded once #4420 lands.