[V0086] Fix candidate totals for future elections

[lbeaufort]

Summary (required)

• Resolves Presidential 2017 - 2020 data on adv data table not displaying properly #3214

Fix candidate totals for future elections (2020/2022 presidential/senate)

I double-checked totals and tracked them here: https://docs.google.com/spreadsheets/d/1NUlhWZ6aHvNNSugPk3BWemDQSZn3pimjbqW_4IC_cnw/edit#gid=0

How to test the changes locally

• Drop/create cfdm_test, invoke create_sample_db
• Point to dev DB, change class CandidateHistoryWithFuture(BaseCandidate)to ofec_candidate_history_with_future_election_mv_tmp and class CandidateTotal(db.Model) to ofec_candidate_totals_mv_2020_tmp
• I find it easiest to test these by running my CMS locally and pointing to my local API. Links:
• http://localhost:8000/data/candidates/president/?election_year=2020&cycle=2020&election_full=true
• http://localhost:8000/data/candidates/senate/?election_year=2022&cycle=2018&election_full=false
• You can test raising/spending breakdowns if you run the older branch feature/add-suite-to-committee-address

Impacted areas of the application

List general components of the application that this PR will affect:

• /candidates/totals endpoint
• fec.gov/data/candidates/president/
• fec.gov/data/candidates/house/ (should be no change)
• fec.gov/data/candidates/senate/
• Raising/Spending breakdowns when we bring them back

Before - no data 👎

After - data! 👍

Related PRs

Branched from feature/fix-special-candidate-totals, PR: #3198

[codecov-io]

Codecov Report

Merging #3219 into develop will increase coverage by 0.35%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop    #3219      +/-   ##
===========================================
+ Coverage    87.78%   88.14%   +0.35%     
===========================================
  Files           77       77              
  Lines         6167     6147      -20     
===========================================
+ Hits          5414     5418       +4     
+ Misses         753      729      -24

Impacted Files	Coverage Δ
webservices/utils.py	91.2% <100%> (+0.12%)	⬆️
webservices/common/models/candidates.py	100% <100%> (ø)	⬆️
webservices/resources/candidate_aggregates.py	96.72% <100%> (ø)	⬆️
webservices/flow.py	100% <100%> (ø)	⬆️
webservices/resources/audit.py	97.05% <0%> (-0.05%)	⬇️
webservices/rest.py	89.73% <0%> (-0.04%)	⬇️
webservices/tasks/__init__.py	68% <0%> (ø)	⬆️
webservices/tasks/legal_docs.py	0% <0%> (ø)	⬆️
webservices/docs.py	100% <0%> (ø)	⬆️
webservices/spec.py	100% <0%> (ø)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f920bdd...ef48865. Read the comment docs.

[ccostino]

This could be a good opportunity to try out Hypothesis and seeing how well it pairs with py.test's parameterized functions! I know there is a desire to get this out sooner rather than later, but I think these tests and this setup would make for an excellent candidate in trying this out in the future. :-)

[lbeaufort]

Thanks @ccostino! I forgot I put this note in here. I was thinking of using a get_current_cycle() + 2 but it looks like that function is only in the CMS.

[lbeaufort]

@ccostino I made these future cycles dynamic :)

[hcaofec]

the creation script of this ofec_candidate_history_with_future_elections_mv

1. is ofec_candidate_history_with_future_elections_mv going to replace ofec_candidate_history_mv eventually?
2. does ofec_candidate_detail_mv have extra data that ofec_candidate_history_mv does not have? ofec_candidate_history seems to have all the future candidate_election_yr. So the union to ofec_candidate_detail_mv may not necessary.
3. by rounding up future candidate_election_yr ( line 47 ), a few fec_election_year ( the ones in the middle) are missing. This could bring potential issue.
4. might want to remove plural in table name to follow naming conventions

@vrajmohan @lbeaufort @jwchumley

[lbeaufort]

Thanks so much for taking a look, @hcaofec!

1. is ofec_candidate_history_with_future_elections_mv going to replace ofec_candidate_history_mv eventually?

I'm not sure it should. As I understand the business logic of ofec_candidate_history_mv , it's similar to disclosure.cand_valid_fec_yr and shows 2-year periods where the candidate had financial activity. No one has had financial activity in 2020 yet.

2. does ofec_candidate_detail_mv have extra data that ofec_candidate_history_mv does not have? ofec_candidate_history seems to have all the future candidate_election_yr. So the union to ofec_candidate_detail_mv may not necessary.

I needed a way to bring in a row for a future election cycle, in order to join back to the totals. When I tried to union ofec_candidate_history to itself, it got more complicated, and I can't remember why.

3. by rounding up future candidate_election_yr ( line 47 ), a few fec_election_year ( the ones in the middle) are missing. This could bring potential issue.

I will investigate, thanks!

4. might want to remove plural in table name to follow naming conventions?

Yes, thanks!

[hcaofec]

ran this query in dev
select * from public.ofec_candidate_totals_mv_2020_tmp
where candidate_id in( 'S6CA00584')
order by cycle

fec_cycle 2020 is missing, and not sure if last row of financial data ( cycle=2022) makes sense or not

@jwchumley @lbeaufort

[lbeaufort]

@hcaofec that looks right to me, we wouldn't want 2020 in there because the candidate (S6CA00584) is running in 2022. There should be a cycle for every year there's been financial activity, plus one for the election year.

candidate_id	election_year	cycle	is_election	receipts
S6CA00584	2016	2016	True (6-year totals)	15349610.00
S6CA00584	2016	2016	False (2-year totals)	15349610.00
S6CA00584	2022	2018	False (2-year totals)	3499303.00
S6CA00584	2022	2022	True (6-year totals)	3499303.00

[PaulClark2]

@hcaofec @lbeaufort looks correct to me. We need the 2022 row to display the full 6-year period. There's no 2020 row because it's not 2019-20 yet.

candidate_id	election_year	cycle	is_election	receipts
S6CA00584	2022	2018	False (2-year totals)	3499303.00
S6CA00584	2022	2022	True (6-year totals)	3499303.00

@fec-jli when I click on the "details" link of security/snyk - data/flyway/build.gradle (fec-jli) above I get an error message: "organisation does not exist." Any ideas why?

[lbeaufort]

Update: I am now using ofec_candidate_history_mv as the 2nd union. @hcaofec and I tested and it looks good. I also updated the testing views.