Skip to content

Commit

Permalink
[Issue #1745] Setup transformation process structure & transform an o…
Browse files Browse the repository at this point in the history
…pportunity (#1794)

## Summary
Fixes #1745

### Time to review: __10 mins__

## Changes proposed
Setup the transformation process script structure

Implement some shared utilities that subsequent PRs will use

Implement the transformation logic for opportunities

## Context for reviewers
A lot of setup in this PR, a lot that can be reused in the subsequent
PRs to add transformations for the other sets of tables. Tried to make
sure those would require refactoring or pulling out implementation
details by setting up utils like the timestamp conversions + initial
query to the DB to fetch the transforming records.

As far as the implementation goes, determining what needs to be
transformed is pretty simple - the `transformed_at` column is null.
There is then a second column that says whether the record should be
deleted or not. When we query the staging tables, we also join with the
relevant table we'll transform to (opportunity in this case), that way
we already have the "source" and "destination" records and just need to
modify the destination record (or create it if its an insert).
  • Loading branch information
chouinar authored May 3, 2024
1 parent 3799be9 commit 480f395
Show file tree
Hide file tree
Showing 9 changed files with 600 additions and 45 deletions.
80 changes: 42 additions & 38 deletions api/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions api/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ pydot = "1.4.2"
sadisplay = "0.4.9"
ruff = "^0.4.0"
debugpy = "^1.8.1"
freezegun = "^1.5.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
Expand Down
Empty file.
Loading

0 comments on commit 480f395

Please sign in to comment.