Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task]: Create Script Entry Point for Transformation Steps #1810

Closed
acouch opened this issue Apr 23, 2024 · 1 comment · Fixed by #1950
Closed

[Task]: Create Script Entry Point for Transformation Steps #1810

acouch opened this issue Apr 23, 2024 · 1 comment · Fixed by #1950
Assignees
Labels
project: grants.gov Grants.gov Modernization tickets

Comments

@acouch
Copy link
Collaborator

acouch commented Apr 23, 2024

Summary

We want to create the entrypoint to the ecs task that handles copying data from the legacy Oracle database, running the transformations, and the running the set current opportunities logic in that order. Each of these steps of the process will be implemented as a class derived from the Task class.

Setting up a script with an entrypoint simply requires adding a bit of boilerplate. For example, you can see some of this boilerplate in how we setup the copy-oracle-data command: https://github.com/HHS/simpler-grants-gov/blob/main/api/src/data_migration/copy_oracle_data.py#L53

This should look something like:

@data_migration_blueprint.cli.command(
    "whatever-we-want-to-call-this", help="some help message"
)
@flask_db.with_db_session()
def copy_oracle_data(db_session: db.Session) -> None:
      logger.info("Starting")

      CopyFromLegacyTask(db_session).run()
      TransformOracleDataTask(db_session).run()
      SetCurrentOpportunitiesTask(db_session).run()

     logger.info("Done")

And the command can be run with poetry run flask data-migration whatever-we-want-to-call-this

Acceptance criteria

No response

@acouch acouch added project: grants.gov Grants.gov Modernization tickets refinement labels Apr 23, 2024
@acouch acouch added this to the Search API - ELT Implementation milestone Apr 23, 2024
@chouinar chouinar moved this from Icebox to Sprint Ready in Simpler.Grants.gov Product Backlog Apr 23, 2024
@acouch
Copy link
Collaborator Author

acouch commented Apr 26, 2024

This is a sub-task: #1858

@jamesbursa jamesbursa self-assigned this May 7, 2024
@sumiat sumiat removed the refinement label May 7, 2024
@jamesbursa jamesbursa moved this from Sprint Ready to In Progress in Simpler.Grants.gov Product Backlog May 8, 2024
@margaretspring margaretspring moved this from In Progress to In Review in Simpler.Grants.gov Product Backlog May 8, 2024
jamesbursa added a commit that referenced this issue May 8, 2024
## Summary
Fixes #1810

## Changes proposed
- Add an entrypoint that runs the 3 tasks for load and transform in
series.

## Context for reviewers
The regular load and transform job runs 3 tasks in order:

1.  LoadOracleDataTask
2. TransformOracleDataTask
3. SetCurrentOpportunitiesTask

This is implemented as a flask command and can be run locally using
`poetry run flask data-migration load-transform`

## Additional information
Options are available to turn on or off each of the 3 tasks.

```
Options:
  --load / --no-load              run LoadOracleDataTask
  --transform / --no-transform    run TransformOracleDataTask
  --set-current / --no-set-current
                                  run SetCurrentOpportunitiesTask
```

![Screenshot 2024-05-08 at 12 36
49](https://github.com/HHS/simpler-grants-gov/assets/3811269/488a2045-8209-4046-986d-d55405b394ac)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
project: grants.gov Grants.gov Modernization tickets
Projects
Development

Successfully merging a pull request may close this issue.

3 participants