IE missing from /schedule_e/by_candidate

[PaulClark2]

The IE from "A new promise" appears in /schedule_e/ but not /schedule_e/by_candidate.

https://api.open.fec.gov/v1/schedules/schedule_e/?api_key=DEMO_KEY&sort_hide_null=true&data_type=processed&is_notice=true&candidate_id=H8CT05245&sort=-expenditure_date&per_page=30

https://api.open.fec.gov/v1/schedules/schedule_e/by_candidate/?api_key=DEMO_KEY&sort_hide_null=true&sort=-total&per_page=10&page=1&cycle=2018&election_full=true&office=house&state=CT&stateFull=Connecticut&district=05

@fec-issues-bot commented on Tue Oct 02 2018

What were you trying to do and how can we improve it?

Find IEs in CT-5 race. I found an IE on the IE map in classic (https://classic.fec.gov/data/IndependentExpenditure.do?format=html&id=ieState&election_yr=2018&candidateOfficeState=CT&candOffice=H&candidateOfficeDistrict=05) but it doesn't show up on the election page on the new site: https://www.fec.gov/data/elections/house/CT/05/2018/.

I also found this lineup on the IE page of the new site: https://www.fec.gov/data/independent-expenditures/?data_type=processed&is_notice=true&candidate_id=H8CT05245.

That first IE doesn't show up either on Classic or on the election page on the new site.

General feedback?

The new election pages are supposed to provide a snapshot of the state of play in every district.

Tell us about yourself

Judy in the Press Office, interesting in bringing back the maps

Details

• URL: https://www.fec.gov/data/elections/house/CT/05/2018/
• User Agent: Amazon CloudFront

Completion criteria

• Investigate what caused this discrepancy and if it's a larger issue.
• Fix the problem
• Put in a new ticket for any future work

[lbeaufort]

@PaulClark2 it looks like the aggregate tables that feed into the /schedule_e/by_candidate/ endpoint that appears on the election profile pages are excluding transactions from 24/48 hour reports. Should the aggregates be made of quarterly filings or 24/48 hour notices?

ofec_sched_e_aggregate_candidate_mv:
https://github.com/fecgov/openFEC/blob/develop/data/migrations/V0026__more_public_schema_objects.sql#L4988

[PaulClark2]

@lbeaufort excluding the 24/48 hour reports makes sense to me because we don't have a good automated way to sort out news and amendments. The problem is we aren't including spending that has been reported on the 24/48 reports submitted before the next regular report. I'm not sure what our solution should be.

[lbeaufort]

@PaulClark2 were you exploring SQL for this issue, or was that for something else?

[lbeaufort]

This issue was actually caused by a misunderstanding of the functionality. The aggregate tables that feed into the /schedule_e/by_candidate/ endpoint that appears on the election profile pages are excluding transactions from 24/48 hour reports.

Maybe we need more user testing before designing a solution?
cc: @JonellaCulmer @PaulClark2

[lbeaufort]

One possible solution would be to have a toggle for IE 24/48 and regularly scheduled reports like we do on the data tables and on the "Individual contributions to candidates" section on the election profile page.

IE data table toggle

https://www.fec.gov/data/independent-expenditures/?data_type=processed&is_notice=true

Individual contributions to candidates toggle

https://www.fec.gov/data/elections/house/CT/05/2018/#individual-contributions

[lbeaufort]

Moving back to fec-cms since this is not an API bug.

[lbeaufort]

Issue moved to fecgov/fec-cms #2503 via ZenHub