Fix totals for off-year special candidates on /candidates/totals endpoint

[lbeaufort]

Summary (required)

• Resolves Incorrect totals for /candidates/totals for some candidates #3196 and Resolves Candidates missing from totals endpoint (Ossoff) #3200. The /candidates/totals endpoint was doubling totals for odd-year special election candidates, and missing at least one odd-year special election candidate

1) #3196: Fixes incorrect totals for /candidates/totals/ for off-year special election candidates

Change to ofec_candidate_totals_mv:
Add a SELECT DISTINCT:

 ), link AS ( --Off-year special election candidates have > 1 row/cycle
         SELECT DISTINCT cand_id,
            cand_election_yr + cand_election_yr % 2 as rounded_election_yr,
            fec_election_yr,
            cmte_id,
            cmte_dsgn
          FROM ofec_cand_cmte_linkage_mv
          WHERE cmte_dsgn IN ('P', 'A')

cand_id	cand_election_yr	fec_election_yr	cmte_id	cmte_tp	cmte_dsgn
S0AL00156	2018	2018	C00640623	S	P
S0AL00156	2017	2018	C00640623	S	P

Performance

Before: 87037 rows, cost=(22821.52..22840.34)
After: 87037 rows, cost=(17454.39..17462.67)

Example: Doug Jones, Raised

Total raised	Total Receipts (from candidate page)
$46,956,378.00	$23,478,189.00

Incorrect totals in API call:
https://api.open.fec.gov/v1/candidates/totals/?api_key=DEMO_KEY&per_page=20&page=1&candidate_id=S0AL00156&cycle=2018

2) #3200 : Fixes missing off-year special candidates

Change to ofec_candidate_totals_with_0s_mv: Remove restrictive join on totals (ofec_candidate_totals_mv) that was preventing odd-year special candidates who didn't run again (H8GA06195 is an example) from showing up:
AND cand.candidate_election_year = totals.election_year

Missing candidate in API call (top raising candidate in 2018)
https://api.open.fec.gov/v1/candidates/totals/?api_key=DEMO_KEY&sort_hide_null=true&q=H8GA06195&cycle=2018&sort=-receipts&per_page=30&page=1&office=H

Data tracking: https://docs.google.com/spreadsheets/d/1NUlhWZ6aHvNNSugPk3BWemDQSZn3pimjbqW_4IC_cnw/edit#gid=1784243483

Background

• Resource is TotalsCandidateView
• MV is ofec_candidate_totals_with_0s_mv which relies on ofec_candidate_totals_mv
• ofec_candidate_totals_mv has incorrect totals for Doug Jones (S0AL00156)
• The problem is with the join on ofec_cand_cmte_linkage_mv - off-year special election candidates will have two entries for 2018 in there.

Impacted areas:

• https://www.fec.gov/data/candidates/senate/
• https://www.fec.gov/data/candidates/house/
• https://www.fec.gov/data/raising/
• https://www.fec.gov/data/spending
• Top raising candidates here: https://www.fec.gov/data/

How to test locally

• Testing migration: Drop and recreate your local cfdm_test db, then invoke create_sample_db

Testing real data:

• Connect your SQLA_CONN to the dev database.
  (There is an ofec_candidate_totals_mv_tmp and ofec_candidate_totals_with_0s_mv_tmp based off it for data testing.)
• To use these MV's, change CandidateTotal(db.Model) in webservices/common/models/candidates.py to point to ofec_candidate_totals_with_0s_mv_tmp
• Test totals for off-year special election candidates. S0AL00156 and H8GA06195 are a start, but extra eyes on other candidates are helpful. Compare totals to https://www.fec.gov/data/candidate// page. Can use "Impacted areas" to see where these appear on the front end https://docs.google.com/spreadsheets/d/1r9QtqGwWrK5ZZVz3tTBB8TpmDzPBgjzOvCmuHrbQ6tg/edit#gid=0

Outstanding items

• Figure out how to change this MV when other MV's depend on it <- You need to drop and re-create MV's that depend on it.
• Ask DB team about performance - this is slower than before <- Moved the SELECT DISTINCT for link into a subquery and performance beat the old query - thanks @ChingKuan!
• Once we can rename the MV/Indexes, check to make sure rename_indexes works <- I think this might be case sensitive? Will test if I can
• Jon Ossoff, the top raising candidate for 2017-2018, is missing from this endpoint: https://api.open.fec.gov/v1/candidates/totals/?api_key=DEMO_KEY&sort_hide_null=true&q=H8GA06195&cycle=2018&sort=-receipts&per_page=30&page=1&office=H
• Update comments with new changes
• Create _tmp for testing data
• Add tests for odd-year house/senate special candidates

[codecov-io]

Codecov Report

Merging #3198 into develop will decrease coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           develop   #3198      +/-   ##
==========================================
- Coverage    88.15%   88.1%   -0.05%     
==========================================
  Files           77      77              
  Lines         6145    6129      -16     
==========================================
- Hits          5417    5400      -17     
- Misses         728     729       +1

Impacted Files	Coverage Δ
webservices/common/models/candidates.py	100% <ø> (ø)	⬆️
webservices/flow.py	100% <ø> (ø)	⬆️
webservices/resources/candidate_aggregates.py	96.72% <100%> (-0.21%)	⬇️
webservices/resources/committees.py	84.09% <0%> (-1.14%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f95b78b...faa1e69. Read the comment docs.

[lbeaufort]

I updated the data tracking sheet with the new data using the temporary MV's on dev: https://docs.google.com/spreadsheets/d/1NUlhWZ6aHvNNSugPk3BWemDQSZn3pimjbqW_4IC_cnw/edit#gid=1104244016

[hcaofec]

Data looks good. One thing I noticed though is when running this query, each ID returns two rows, the only difference is is_election column. So one candidate has both False and True for is_election column in 2018. I am not sure if that conforms to business rule.

select *
from public.ofec_candidate_totals_mv_tmp
where candidate_id in ('H8MT01182', 'H8GA06286')

@lbeaufort

[fecjjeng]

select *
from public.ofec_candidate_totals_mv
where candidate_id in ('H8MT01182', 'H8GA06286')
would result in the same phenomenon. ie the same candidate has both False and True for is_election column in 2018 (the amount of money is good with the fix though).
So it seems the distinct solve one problem but not all problem?

[lbeaufort]

@fecjjeng and @hcaofec thanks for taking a look. @jwchumley is going to spend some time investigating the data because Jon Ossoff is missing (see outstanding item in PR). I don't follow the is_election logic but it might be part of the problem.

[lbeaufort]

@hcaofec @fecjjeng - Pat and I looked into is_election field and it really only makes sense for the Senate/Presidential candidates that need 2-year totals in an off-election year. is_election True means "This has the totals for the entire election cycle 4 or 6 years for Prez/Sen) and False means it's just the 2-year total within the larger 4 or 6 year cycle. S2MA00170 2018 is a good example.

[hcaofec]

By following the testing steps written in this PR, here are the results:

1. Dropped and recreated cfdm_test db locally, then ran ‘invoke create_sample_db’, migration went successfully and the creating script of ofec_candidate_totals_mv_tmp has the latest version
2. Set local SQLA_CONN to the dev database.
3. Change CandidateTotal(db.Model)  to point to ofec_candidate_totals_with_0s_mv_tmp
4. Test /candidate/:  + /candidates/totals/ endpoint, compared result with same endpoint in production

candidate_id	counts	before change	after change
S0AL00156	before: 2; after: 3	"receipts": 46956378,	"receipts": 23478189,
S8AL00340	1	"receipts": 12527334	"receipts": 6263667
H8GA06195		not found	"receipts": 31381944.8

The data comparison looks good. But the creation sql script can be refactored to simply the logic. ( see comment above )

[hcaofec]

The left join here ( line 66 - 69 ) may not be necessary here because of the distinct logic used in link subquery. And because ofec_candidate_election_mv is used on the right side of the left join, it will not have any effect on the result of cycle_totals subquery.
@vrajmohan @lbeaufort

[lbeaufort]

@hcaofec good catch, thanks for taking a look! I had to make more changes to ofec_candidate_totals_mv for future cycle work. Because this changeset is thoroughly tested, I will look at removing this join in #3219 and re-test if that sounds ok to you.

[hcaofec]

tested on local by running py.test. No errors.