Skip to content

Update test framework and support msgspec serialization.

Compare
Choose a tag to compare
@s7clarke10 s7clarke10 released this 08 Jul 07:00
· 16 commits to master since this release
59efc29

Description of change

A major update to build and run this framework to be pep8 compliant as much as possible and to modernise the testing framework.

Changes:

  1. Moving from setup.py to pyproject.toml and poetry for installation
  2. Introducing tox for platform independent testing of pep8 compliance.
  3. Introducing black for formatting
  4. Introducing isort for compliant and order includes
  5. Introducing flake8 for linting alongside pylint
  6. Introducing mypy for type checking
  7. Updating github actions to run tox tests rather than directly running pytests.
  8. Code formatting updates and changes to pass / met the standards above.
  9. Deprecating python 3.7

Manual QA steps

  • Ensuring all pytests pass
  • Ensuring all introduced tox python pep compliant tests pass
  • Running against tap-oracle which uses this variant of the singer-python framework
  • Ensure performance is good from ORJSON -> MSGSPEC

Performance Results:

  • Table A: 455.4k rows, 22.3 MB

  • Table B: 848.9k rows, 31.1 MB

  • MSGSPEC with Singer Decimal on runtime:

    Start Time 2024-07-08T05:06:48
    End Time 2024-07-08T05:13:27

    Total Time: 6:39 (min/secs) - lag 1:19 (min/sec) before ingesting really kicks in.

  • ORJSON with Singer Decimal on runtime:

    Start Time 2024-07-08T05:32:10
    End Time 2024-07-08T05:39:40

    Total Time: 7:30 (min/secs) - lag 1:34 (min/sec) before ingesting really kicks in.

  • MSGSPEC with Singer Decimal off runtime:

    Start Time 2024-07-08T05:47:19
    End Time 2024-07-08T05:53:53

    Total Time: 6:34 (min/secs) - lag 0:51 (min/sec) before ingesting really kicks in.

  • ORJSON with Singer Decimal off runtime:

    Start Time 2024-07-08T05:55:48
    End Time 2024-07-08T06:03:18

    Total Time: 7:30 (min/secs) - lag 1:13 (min/sec) before ingesting really kicks in.

  • Overall performance gain running from tap-oracle to target-snowflake (real world scenario). Close to one minute.