Update test framework and support msgspec serialization.
Description of change
A major update to build and run this framework to be pep8 compliant as much as possible and to modernise the testing framework.
Changes:
- Moving from setup.py to pyproject.toml and poetry for installation
- Introducing tox for platform independent testing of pep8 compliance.
- Introducing black for formatting
- Introducing isort for compliant and order includes
- Introducing flake8 for linting alongside pylint
- Introducing mypy for type checking
- Updating github actions to run tox tests rather than directly running pytests.
- Code formatting updates and changes to pass / met the standards above.
- Deprecating python 3.7
Manual QA steps
- Ensuring all pytests pass
- Ensuring all introduced tox python pep compliant tests pass
- Running against tap-oracle which uses this variant of the singer-python framework
- Ensure performance is good from ORJSON -> MSGSPEC
Performance Results:
-
Table A: 455.4k rows, 22.3 MB
-
Table B: 848.9k rows, 31.1 MB
-
MSGSPEC with Singer Decimal on runtime:
Start Time 2024-07-08T05:06:48
End Time 2024-07-08T05:13:27Total Time: 6:39 (min/secs) - lag 1:19 (min/sec) before ingesting really kicks in.
-
ORJSON with Singer Decimal on runtime:
Start Time 2024-07-08T05:32:10
End Time 2024-07-08T05:39:40Total Time: 7:30 (min/secs) - lag 1:34 (min/sec) before ingesting really kicks in.
-
MSGSPEC with Singer Decimal off runtime:
Start Time 2024-07-08T05:47:19
End Time 2024-07-08T05:53:53Total Time: 6:34 (min/secs) - lag 0:51 (min/sec) before ingesting really kicks in.
-
ORJSON with Singer Decimal off runtime:
Start Time 2024-07-08T05:55:48
End Time 2024-07-08T06:03:18Total Time: 7:30 (min/secs) - lag 1:13 (min/sec) before ingesting really kicks in.
-
Overall performance gain running from tap-oracle to target-snowflake (real world scenario). Close to one minute.