Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes to get opportunity ELT process functioning in production #2429

Closed
2 tasks
chouinar opened this issue Oct 9, 2024 · 0 comments · Fixed by #2430
Closed
2 tasks

Fixes to get opportunity ELT process functioning in production #2429

chouinar opened this issue Oct 9, 2024 · 0 comments · Fixed by #2430
Assignees
@chouinar

Comments

@chouinar
Copy link
Collaborator

chouinar commented Oct 9, 2024

Summary

A catch-all ticket for various small ELT process issues encountered in testing the ELT process in prod:

  • Fix the default chunk size to actually use the 800 value we've configured
  • Account for ” ” as a value for opportunity category in the transform logic (make a general pass at other similar fields)
  • Handle removing the current opportunity summary connection when deleting the opportunity summary object

Acceptance criteria

  • Fixes made
  • Tests updated to verify fixes
@chouinar chouinar self-assigned this Oct 9, 2024
@chouinar chouinar moved this from Icebox to In Review in Simpler.Grants.gov Product Backlog Oct 9, 2024
chouinar added a commit that referenced this issue Oct 9, 2024
@chouinar
[Issue #2429] Various small fixes to get ELT process working in prod
fdcbba2
@chouinar chouinar mentioned this issue Oct 9, 2024
Merged
chouinar added a commit that referenced this issue Oct 9, 2024
@chouinar
[Issue #2429] Various small fixes to get ELT process working in prod (#…
96d864f 
…2430)

## Summary
Fixes #2429

### Time to review: __5 mins__

## Changes proposed
Actually set the default chunk size to 800

Handle cases where lookup values in the legacy system are ` `
(whitespace) and treat them the same as null/empty strings

Handle deleting the `current_opportunity_summary` record when an
`opportunity_summary` is deleted

## Context for reviewers
All issues found during testing in the prod ELT runs I was doing today

Chunk size affects how "big" the batches are when we do queries to the
Oracle foreign data wrapper as Oracle has a hard limit on the size of
queries that we were hitting.

For the lookup values, found a dozen or so opportunities where the
opportunity category was literally a ` ` (space) and we couldn't process
them. This fixes that and pre-emptively does it for other similar rows
(although I think most of them haven't had the issue).

The delete just takes advantage of SQLAlchemy's approach to deletes
which will automatically delete through relationships when you do
`db_session.delete(some_record)`. Now it'll know it needs to first
delete the current opportunity summary record as part of that.

## Additional information
Tests updated accordingly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chouinar chouinar

Labels
None yet
Projects
Simpler.Grants.gov Product Backlog
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Issue #2429] Various small fixes to get ELT process working in prod HHS/simpler-grants-gov
1 participant
@chouinar