Cap unfunded dues

[rohitpaulk]

#3818

[rohitpaulk]

Now to change the existing values...

[rohitpaulk]

Not ready to merge. I'm seeing weird data values in our DB, got to investigate more...

=> SELECT amount, due, is_funded FROM current_payment_instructions WHERE due > 10 AND amount > 0 AND is_funded;
 amount |  due   | is_funded
--------+--------+-----------
   1.00 |  14.00 | t
  10.00 | 140.00 | t
   5.00 |  70.00 | t
   1.00 |  11.00 | t
   1.00 |  14.00 | t
   1.00 |  14.00 | t
   1.00 |  14.00 | t
   1.00 |  14.00 | t
   5.00 |  70.00 | t
   1.00 |  14.00 | t
   1.00 |  14.00 | t
   1.00 |  14.00 | t
   1.00 |  14.00 | t
   2.25 |  31.50 | t
   8.00 |  80.00 | t
   2.00 |  28.00 | t
   1.00 |  14.00 | t
   1.00 |  14.00 | t
   5.00 |  70.00 | t
   1.00 |  14.00 | t
   1.00 |  11.00 | t
   1.00 |  12.00 | t
   0.75 |  10.50 | t
   1.00 |  14.00 | t
(24 rows)

How do funded payments get to a value greater than the minimum charge???

[rohitpaulk]

^ cc: @rorepo

[rohitpaulk]

Looks like none of these ~users have CCs attached... Do we not mark is_funded as false when a CC is removed?

[rohitpaulk]

Aha. These are people who have never attached a CC. The tips were funded because of their gratipay balance. We used to refresh these values after every Payday (before we moved to charging in arrears and never using one's balance to fund tips).

I guess running Participant.update_giving on all users should correct this problem. Going to try that on a backup...

[chadwhitacre]

Ick. We moved away from extra interpolation in ... b045fb1 (#3785). Can we avoid reintroducing that here? I guess that would mean putting minimum_charge in the database somewhere.

[rohitpaulk]

Aha, I'll see if I can get this done in a cleaner fashion.

[chadwhitacre]

How about the approach in 2493792, @rohitpaulk?

[rohitpaulk]

Looks good to me :)

[chadwhitacre]

Cool. A further improvement could be to call it payday_settings, and create it as a temporary table as part of payday.

[rohitpaulk]

I think having it as a permanent table makes sense - If we start to write heavy SQL outside of payday, it could be useful.

[chadwhitacre]

Initial punchlist:

• avoid Python-interpolating SQL—Cap unfunded dues #3876 (comment)
• update the minimum charge amount—Cap unfunded dues #3876 (comment)

I still need to slow down and read this code to understand what we're doing here ...

[chadwhitacre]

I'm not sure what's up with this Travis failure. I'm pretty sure it's present on master, too ...

[chadwhitacre]

Hoping to read and grok this PR tomorrow.

[chadwhitacre]

Build is failing on master: https://travis-ci.org/gratipay/gratipay.com/builds/98361516. Reticketed as #3885.

[chadwhitacre]

I've kicked Travis post-#3891.

[chadwhitacre]

Looks like there is still some breakage ...

[rohitpaulk]

/me kicks Travis again..

[rohitpaulk]

@whit537 - What's left here?

[chadwhitacre]

Not much ... looks like I pushed some commits that didn't end up on this branch?

[rohitpaulk]

Ohhhhhh

[rohitpaulk]

I might've forced pushed :/ Do you have the branch on your local? If so, please push it to a different branch and I'll merge the two manually

[chadwhitacre]

No worries. My commits are still in GitHub: https://github.com/gratipay/gratipay.com/commits/249379279a931a3ece8d29e2e9ec255665c997ad.

[chadwhitacre]

;-)

$ gf
remote: Counting objects: 57, done.
remote: Total 57 (delta 36), reused 36 (delta 36), pack-reused 21
Unpacking objects: 100% (57/57), done.
From github.com:gratipay/gratipay.com
 + 2493792...12057d5 cap-unfunded-dues -> origin/cap-unfunded-dues  (forced update)

[chadwhitacre]

Alright, brought those back. We'll see if Travis is okay with them.

[rohitpaulk]

Rebased on master again.

[chadwhitacre]

Rebased on master at ec2c732. Previous head was ad82f60.

[rohitpaulk]

The commit log on this is pretty confusing - makes it hard to review. I'm going to squash this down to 2-3 meaningful commits.

[rohitpaulk]

I'm seeing three major changes here, the rest is just noise.

1. Prevent dues from exceeding minimum_charge in payday
2. Add SQL script to repair old values that are above minimum charge
3. Add script to recalculate stale cached values

[rohitpaulk]

I'm going to cherry pick 1 into a separate PR - and retain 2 and 3 as part of this PR.

[rohitpaulk]

Split into #3981 and #3980