backfill exchanges

[chadwhitacre]

For #2779. This requires a little schema update: we want to link exchanges to transaction in foreign systems (Balanced, PayPal, etc.). Though with cash I guess we won't have a transaction to link against. Hmm ...

[rohitpaulk]

Though with cash I guess we won't have a transaction to link against.

What cash are you talking about?

[chadwhitacre]

@rohitpaulk In a few cases I made payouts to people by putting cold, hard cash in their grubby paws. :)

[rohitpaulk]

Aha :)

[chadwhitacre]

@kaguillera and I are trying to get the numbers from this latest version to line up.

known customer               47565
raise UnknownCustomer            5
----------------------------------
                             47570


exchange_id in transaction   17796
triangulated an exchange     29659
raise UnknownExchange           88
----------------------------------
                             47543


exchange has a route          2306
card matches a route         44576
created a route                427
raise UnknownRoute               6
----------------------------------
                             47315


Linking                      45003
Already linked                2306
raise UnknownCustomer            5
raise UnknownExchange           88
raise UnknownRoute               6
----------------------------------
                             47408


^201\d-                      47573
----------------------------------
                             47573

[chadwhitacre]

The out 3 in customers is because we have three debits that were not credit card charges but were rather escrow shuffles from New Alliance back over to Balanced (cf gratipay/inside.gratipay.com#308). Our filter isn't good enough!

[chadwhitacre]

Okay, we've weeded out five ACH debits: three escrow shuffles, and two user payins (one coming from Coinbase, and another a direct manual ACH debit).

[chadwhitacre]

Here's our current tally. We're added counts into the script itself with a check after each transaction to find the places where our numbers go out.

Hi-level
----------------------------------
Linking                      45079
Already linked                2309
raise UnknownCustomer            4
raise UnknownExchange            2
raise UnknownRoute               0
                             47394
Participants
----------------------------------
known customer               47564
raise UnknownCustomer            4
                             47568
Exchanges
----------------------------------
exchange_id in transaction   17797
triangulated an exchange     29765
raise UnknownExchange            2
                             47564
Routes
----------------------------------
exchange has a route          2307
card matches a route         45076
created a route                  0
raise UnknownRoute               0
                             47383
Check
----------------------------------
^201\d-                      47568
                             47568

[chadwhitacre]

Okay! We're getting good numbers now:

[gratipay] $ ./tally-backfill.py 
Hi-level
----------------------------------
Linking                      45076
Already linked                2306
raise                          186
                             47568
Participants
----------------------------------
known customer               47564
raise UnknownCustomer            4
                             47568
Exchanges
----------------------------------
exchange_id in transaction   17796
triangulated an exchange     29765
raise UnknownExchange            3
                             47564
Routes
----------------------------------
exchange has a route          2306
card matches a route         45076
created a route                  0
raise AmbiguousRoute           161
raise AmbiguousCustomer         18
raise UnknownRoute               0
                             47561
Check
----------------------------------
^201\d-                      47568
                             47568

[chadwhitacre]

We've got a bug where we're inserting routes out-of-band of our transaction. Note the 0 above and the 1091 below. The problem is that we're calling ExchangeRoutes.insert, which doesn't use our cursor.

Hi-level
----------------------------------
Linking                      44995
Already linked                2306
raise                          267
                             47568
Participants
----------------------------------
known customer               47564
raise UnknownCustomer            4
                             47568
Exchanges
----------------------------------
exchange_id in transaction   17796
triangulated an exchange     29765
raise UnknownExchange            3
                             47564
Routes
----------------------------------
exchange has a route          2306
card matches a route         43904
created a route               1091
raise AmbiguousRoute           161
raise AmbiguousCustomer         99
raise UnknownRoute               0
                             47561
Check
----------------------------------
^201\d-                      47568
                             47568

[chadwhitacre]

The goal of this PR is to resolve all of the raises under Hi-level.

[chadwhitacre]

The three UnknownExchanges are:

• Two of me testing Balanced (they're the first two credit card charges we ever did on Balanced).
• A special case where the manual exchange was recorded wrong, with a second exchange to correct. The amount of the first exchange is out relative to Balanced. (backfill the status, route, and ref columns of the exchanges table #2779 (comment))

[chadwhitacre]

For the first two test transactions, I suppose we should insert a new exchange record for each, since it is money I added into the system. Or we should refund them! Can we refund our first two transactions!? :D

For the third, I think we need to manually modify the exchanges table to fold those two into a single exchange. It should then match the single transaction in Balanced.

[chadwhitacre]

It appears that all duplicate routes per exchange are the result of duplicate customers (#3806).

[chadwhitacre]

With aaea493 (should be the same as #3807 (comment)):

Hi-level
----------------------------------
Linking                      44995
Already linked                2306
raise                          267
                             47568
Participants
----------------------------------
known customer               47564
raise UnknownCustomer            4
                             47568
Exchanges
----------------------------------
exchange_id in transaction   17796
triangulated an exchange     29765
raise UnknownExchange            3
                             47564
Routes
----------------------------------
exchange has a route          2306
card matches a route         43904
created a route               1091
raise AmbiguousRoute           161
raise AmbiguousCustomer         99
raise UnknownRoute               0
                             47561
Check
----------------------------------
^201\d-                      47568
                             47568

[chadwhitacre]

(And it is, afaict.)

[chadwhitacre]

Resolving ambiguous routes via the absorptions table (4731b5d) works:

Hi-level
----------------------------------
Linking                      45156
Already linked                2306
raise                          100
                             47562
Participants
----------------------------------
known customer               47564
raise UnknownCustomer            4
                             47568
Exchanges
----------------------------------
exchange_id in transaction   17796
triangulated an exchange     29765
raise UnknownExchange            3
                             47564
Routes
----------------------------------
exchange has a route          2306
card matches a route         44065
created a route               1091
raise AmbiguousRoute             0
raise AmbiguousCustomer         93
raise UnknownRoute               0
                             47555
Check
----------------------------------
^201\d-                      47568
                             47568

[chadwhitacre]

Looks like by modifying the source code while the script is running, I'm screwing up the tracebacks (which load their lines from source), and therefore throwing the counts off.

[chadwhitacre]

By using customer id to match in addition to card id (e236680), we reduce the number of routes we need to create by 55%, and we reduce AmbiguousCustomers by a bit, but at the cost of bringing back some AmbiguousRoutes. These are consternating, because a spot-check reveals that when a single user has multiple routes it can be because they invalidated an old one. When did they do that? It's hard to tell.

Hi-level
----------------------------------
Linking                      45115
Already linked                2306
raise                          147
                             47568
Participants
----------------------------------
known customer               47564
raise UnknownCustomer            4
                             47568
Exchanges
----------------------------------
exchange_id in transaction   17796
triangulated an exchange     29765
raise UnknownExchange            3
                             47564
Routes
----------------------------------
exchange has a route          2306
card matches                 37510
customer matches              7113
created a route                492
raise AmbiguousRoute            60
raise AmbiguousCustomer         80
raise UnknownRoute               0
                             47561
Check
----------------------------------
^201\d-                      47568
                             47568

[chadwhitacre]

Honestly, at this point we're pushing 95% of our credit card charges linked between exchanges and transactions. That's probably good enough for the purposes of getting some refunds out the door tomorrow.

[chadwhitacre]

Should've marked this Review earlier, sorry. Merging now to unblock #3810, which has to get done tonight if it's going to get done at all.

[chadwhitacre]

Application error?!

[chadwhitacre]

Fixed in #3813.