-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(targets): Targets now accept a batch_size_rows
setting to configure how many rows are loaded in each record batch
#2248
feat(targets): Targets now accept a batch_size_rows
setting to configure how many rows are loaded in each record batch
#2248
Conversation
CodSpeed Performance ReportMerging #2248 will not alter performanceComparing Summary
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2248 +/- ##
==========================================
+ Coverage 88.54% 88.56% +0.01%
==========================================
Files 54 54
Lines 4724 4730 +6
Branches 920 921 +1
==========================================
+ Hits 4183 4189 +6
Misses 380 380
Partials 161 161 ☔ View full report in Codecov by Sentry. |
@pnadolny13 and @edgarrmondragon I broke out the code from pr 1876 that gives the user of a target the ability to set the batch size in rows which triggers a drain. Please take a look and let me know what you think. |
Co-authored-by: Edgar Ramírez Mondragón <[email protected]>
ed51495
to
c247f82
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for taking the time for extracting this into its own PR @BuzzCutNorman!
@edgarrmondragon Thanks 🙏 for all of the comment and documentation edits. Loads better 🥳 than what I had. 🎉 |
batch_size_rows
setting to configure how many rows are loaded in each record batch
…igure how many rows are loaded in each record batch (#2248) * added batch_size_rows tests to target * add batch_size_rows test to test target sql * added batch size row to target capabilities * add batch_size_rows to target as a bulitin config * added code for batch_size_rows * Apply suggestions from code review Co-authored-by: Edgar Ramírez Mondragón <[email protected]> * applied documentaion update from review to max_size * Update singer_sdk/sinks/core.py * Update singer_sdk/sinks/core.py * chore: Link to `batch_size_rows` attribute docs * Update `batch_size_rows` docs --------- Co-authored-by: Edgar Ramírez Mondragón <[email protected]> Co-authored-by: Edgar Ramírez-Mondragón <[email protected]>
* refactor: Limit internal usage of pendulum * fix: Replace use of deprecated `jsonschema._RefResolver` with recommended `referencing` library (#2187) * refactor: Replace use of deprecated `jsonschema._RefResolver` with recommended `referencing` library * Fix types * Continue on error * refactor: Remove unused `logger` parameter from private catalog helper functions (#2188) * refactor: Use `functools.lru_cache` instead of the stale `memoization` library (#1981) (#2189) * chore(deps): bump referencing from 0.32.1 to 0.33.0 (#2194) Bumps [referencing](https://github.com/python-jsonschema/referencing) from 0.32.1 to 0.33.0. - [Release notes](https://github.com/python-jsonschema/referencing/releases) - [Changelog](https://github.com/python-jsonschema/referencing/blob/main/docs/changes.rst) - [Commits](python-jsonschema/referencing@v0.32.1...v0.33.0) --- updated-dependencies: - dependency-name: referencing dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump cryptography from 42.0.0 to 42.0.1 (#2193) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.0 to 42.0.1. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@42.0.0...42.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump pytz from 2023.3.post1 to 2023.4 (#2191) Bumps [pytz](https://github.com/stub42/pytz) from 2023.3.post1 to 2023.4. - [Commits](stub42/pytz@release_2023.3.post1...release_2023.4) --- updated-dependencies: - dependency-name: pytz dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(templates): Include empty `schemas` directory in REST tap cookiecutter (#2195) * docs: Added an example implemetation of JSON schema validation that uses `fastjsonschema` (#2231) Supersedes #2066 Co-authored-by: Dan Norman <[email protected]> * fix(templates): Test workflow job now fails for unsupported Python versions in cookiecutter templates (#2232) * Try to setup Poetry env for specific Python version before installing dependencies * Keep running tests for other Python versions if others fail * chore(deps): bump pre-commit from 3.6.0 to 3.6.1 in /.github/workflows (#2233) Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.6.0 to 3.6.1. - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v3.6.0...v3.6.1) --- updated-dependencies: - dependency-name: pre-commit dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: pre-commit autoupdate (#2234) updates: - [github.com/python-jsonschema/check-jsonschema: 0.27.4 → 0.28.0](python-jsonschema/check-jsonschema@0.27.4...0.28.0) - [github.com/astral-sh/ruff-pre-commit: v0.2.0 → v0.2.1](astral-sh/ruff-pre-commit@v0.2.0...v0.2.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * chore(deps): bump sqlalchemy from 2.0.25 to 2.0.26 (#2235) Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.25 to 2.0.26. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump sqlalchemy from 2.0.26 to 2.0.27 (#2236) Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.26 to 2.0.27. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump duckdb from 0.9.2 to 0.10.0 (#2237) Bumps [duckdb](https://github.com/duckdb/duckdb) from 0.9.2 to 0.10.0. - [Release notes](https://github.com/duckdb/duckdb/releases) - [Changelog](https://github.com/duckdb/duckdb/blob/main/tools/release-pip.py) - [Commits](duckdb/duckdb@v0.9.2...v0.10.0) --- updated-dependencies: - dependency-name: duckdb dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(taps): Hard and soft deletes for handling `ACTIVATE_VERSION` messages now use the same `WHERE` clause (#2242) fix: using same logic for both hard and soft deletes Soft delete is deleting only versions smaller than the latest version Hard delete is deleting smaller and equal. Making both strategies delete only smaller versions. * fix: Instances of `oneOf` are now handled by null-appending logic (#2245) fix: Instance of `oneOf` are now handled by null-appending logic * chore: Cleaned up some unused files (#2240) chore: Clean up unused files * fix: Force flattened record according to provided flattened schema (#2243) * Force flattened record according to provided flattened schema * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix import * fix * revert collections change * ruff linter fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ruff linter fixes * fix * fix * add test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removing None option for flatten schema and max level * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert typing * fix check * Add a short comment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Edgar Ramírez Mondragón <[email protected]> Co-authored-by: Edgar Ramírez-Mondragón <[email protected]> * docs: Linked reference docs to source code (#2239) docs: Link to source code Closes #149 * chore: Release v0.35.1 (#2247) * chore: Bump package version * Apply suggestions from code review --------- Co-authored-by: edgarrmondragon <[email protected]> * chore(deps): bump cryptography from 42.0.2 to 42.0.3 (#2249) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.2 to 42.0.3. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@42.0.2...42.0.3) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump pre-commit from 3.6.1 to 3.6.2 in /.github/workflows (#2251) Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.6.1 to 3.6.2. - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v3.6.1...v3.6.2) --- updated-dependencies: - dependency-name: pre-commit dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump faker from 23.1.0 to 23.2.1 (#2250) Bumps [faker](https://github.com/joke2k/faker) from 23.1.0 to 23.2.1. - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](joke2k/faker@v23.1.0...v23.2.1) --- updated-dependencies: - dependency-name: faker dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: Null type is now correctly appended as `{"type": "null"}` to `oneOf` types (#2252) * chore: Release v0.35.2 (#2253) chore: Bump package version Co-authored-by: edgarrmondragon <[email protected]> * chore: pre-commit autoupdate (#2254) updates: - [github.com/astral-sh/ruff-pre-commit: v0.2.1 → v0.2.2](astral-sh/ruff-pre-commit@v0.2.1...v0.2.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * chore(deps-dev): bump types-simplejson from 3.19.0.2 to 3.19.0.20240218 (#2255) Bumps [types-simplejson](https://github.com/python/typeshed) from 3.19.0.2 to 3.19.0.20240218. - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-simplejson dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump pytest from 8.0.0 to 8.0.1 (#2256) Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.0.0 to 8.0.1. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@8.0.0...8.0.1) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump cryptography from 42.0.3 to 42.0.4 (#2261) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.3 to 42.0.4. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@42.0.3...42.0.4) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump coverage from 7.4.1 to 7.4.2 (#2260) Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.4.1 to 7.4.2. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@7.4.1...7.4.2) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * refactor: Deprecated `singer_sdk.authenticators.BasicAuthenticator` in favor of `requests.auth.HTTPBasicAuth` (#2220) * refactor: `Deprecate singer_sdk.authenticators.BasicAuthenticator` in favor of `requests.auth.HTTPBasicAuth` Closes #2040 * Emit deprecation warning * feat(targets): Targets now accept a `batch_size_rows` setting to configure how many rows are loaded in each record batch (#2248) * added batch_size_rows tests to target * add batch_size_rows test to test target sql * added batch size row to target capabilities * add batch_size_rows to target as a bulitin config * added code for batch_size_rows * Apply suggestions from code review Co-authored-by: Edgar Ramírez Mondragón <[email protected]> * applied documentaion update from review to max_size * Update singer_sdk/sinks/core.py * Update singer_sdk/sinks/core.py * chore: Link to `batch_size_rows` attribute docs * Update `batch_size_rows` docs --------- Co-authored-by: Edgar Ramírez Mondragón <[email protected]> Co-authored-by: Edgar Ramírez-Mondragón <[email protected]> * fix: Database disconnects are now handled via SQLAlchemy `pool_pre_ping` parameter (#2258) feat: Database disconnects are now handled via SQLAlchemy `pool_pre_ping` parameter * feat: Nested settings are now documented in the output of `--about --format=markdown` (#2263) * feat: JSON schema keywords such as `maxLength` are now supported in `StringType`, `IntegerType` and `NumberType` JSON schema helpers (#2241) feat: Support `maxLength` and similar keywords in string, integer and number JSON schema helpers * chore(deps): bump poetry from 1.7.1 to 1.8.0 in /.github/workflows (#2265) Bumps [poetry](https://github.com/python-poetry/poetry) from 1.7.1 to 1.8.0. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](python-poetry/poetry@1.7.1...1.8.0) --- updated-dependencies: - dependency-name: poetry dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump typing-extensions from 4.9.0 to 4.10.0 (#2266) Bumps [typing-extensions](https://github.com/python/typing_extensions) from 4.9.0 to 4.10.0. - [Release notes](https://github.com/python/typing_extensions/releases) - [Changelog](https://github.com/python/typing_extensions/blob/main/CHANGELOG.md) - [Commits](https://github.com/python/typing_extensions/commits) --- updated-dependencies: - dependency-name: typing-extensions dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump cryptography from 42.0.4 to 42.0.5 (#2268) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.4 to 42.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@42.0.4...42.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump importlib-resources from 6.1.1 to 6.1.2 (#2267) Bumps [importlib-resources](https://github.com/python/importlib_resources) from 6.1.1 to 6.1.2. - [Release notes](https://github.com/python/importlib_resources/releases) - [Changelog](https://github.com/python/importlib_resources/blob/main/NEWS.rst) - [Commits](python/importlib_resources@v6.1.1...v6.1.2) --- updated-dependencies: - dependency-name: importlib-resources dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump coverage from 7.4.2 to 7.4.3 (#2269) Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.4.2 to 7.4.3. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@7.4.2...7.4.3) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump pytest from 8.0.1 to 8.0.2 (#2270) Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.0.1 to 8.0.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@8.0.1...8.0.2) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Update locked dependencies (#2271) * chore: Release v0.36.0 (#2272) chore: Bump package version Co-authored-by: edgarrmondragon <[email protected]> * chore(deps): bump poetry from 1.8.0 to 1.8.1 in /.github/workflows (#2273) Bumps [poetry](https://github.com/python-poetry/poetry) from 1.8.0 to 1.8.1. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](python-poetry/poetry@1.8.0...1.8.1) --- updated-dependencies: - dependency-name: poetry dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(templates): Test workflow now fails for any Python matrix job failure in cookiecutter template (#2274) Let test runs complete for all specified Python versions and fail the workflow if any do not succeed * chore(deps): bump faker from 23.2.1 to 23.3.0 (#2275) Bumps [faker](https://github.com/joke2k/faker) from 23.2.1 to 23.3.0. - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](joke2k/faker@v23.2.1...v23.3.0) --- updated-dependencies: - dependency-name: faker dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump python-dateutil from 2.8.2 to 2.9.0 (#2276) Bumps [python-dateutil](https://github.com/dateutil/dateutil) from 2.8.2 to 2.9.0. - [Release notes](https://github.com/dateutil/dateutil/releases) - [Changelog](https://github.com/dateutil/dateutil/blob/master/NEWS) - [Commits](dateutil/dateutil@2.8.2...2.9.0) --- updated-dependencies: - dependency-name: python-dateutil dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump the actions group with 1 update (#2280) Bumps the actions group with 1 update: [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish). Updates `pypa/gh-action-pypi-publish` from 1.8.11 to 1.8.12 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.8.11...v1.8.12) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump python-dateutil from 2.9.0 to 2.9.0.post0 (#2279) Bumps [python-dateutil](https://github.com/dateutil/dateutil) from 2.9.0 to 2.9.0.post0. - [Release notes](https://github.com/dateutil/dateutil/releases) - [Changelog](https://github.com/dateutil/dateutil/blob/master/NEWS) - [Commits](dateutil/dateutil@2.9.0...2.9.0.post0) --- updated-dependencies: - dependency-name: python-dateutil dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump poetry from 1.8.1 to 1.8.2 in /.github/workflows (#2278) Bumps [poetry](https://github.com/python-poetry/poetry) from 1.8.1 to 1.8.2. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](python-poetry/poetry@1.8.1...1.8.2) --- updated-dependencies: - dependency-name: poetry dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump nox from 2023.4.22 to 2024.3.2 in /.github/workflows (#2277) Bumps [nox](https://github.com/wntrblm/nox) from 2023.4.22 to 2024.3.2. - [Release notes](https://github.com/wntrblm/nox/releases) - [Changelog](https://github.com/wntrblm/nox/blob/main/CHANGELOG.md) - [Commits](wntrblm/nox@2023.04.22...2024.03.02) --- updated-dependencies: - dependency-name: nox dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: The `state_partition_context` dictionary is now correctly interpolated in the error message when duplicate partitions/contexts are detected in the input state (#2281) * test: Made the `test_sync_timer` test faster by using `time_machine` (#2282) * chore(deps): bump faker from 23.3.0 to 24.0.0 (#2284) Bumps [faker](https://github.com/joke2k/faker) from 23.3.0 to 24.0.0. - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](joke2k/faker@v23.3.0...v24.0.0) --- updated-dependencies: - dependency-name: faker dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump sqlalchemy from 2.0.27 to 2.0.28 (#2286) Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.27 to 2.0.28. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump duckdb-engine from 0.11.1 to 0.11.2 (#2285) Bumps [duckdb-engine](https://github.com/Mause/duckdb_engine) from 0.11.1 to 0.11.2. - [Release notes](https://github.com/Mause/duckdb_engine/releases) - [Changelog](https://github.com/Mause/duckdb_engine/blob/main/CHANGELOG.md) - [Commits](Mause/duckdb_engine@v0.11.1...v0.11.2) --- updated-dependencies: - dependency-name: duckdb-engine dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Make deptry happy --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dan Norman <[email protected]> Co-authored-by: ReubenFrankel <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Vitor Avancini <[email protected]> Co-authored-by: Joao Amaral <[email protected]> Co-authored-by: MeltyBot <[email protected]>
This feature will allow target users the ability to set the batch size used to triggers a drain via the target's meltano confiig.
batch_size_rows: 100000
When a batch is full and a drain is triggered, the message logged will now state the size of the batch so the users has a visual confirmation that the batch size they configured is being utilized.
The code for this PR was pulled out of PR 1876.
Closes #2155
📚 Documentation preview 📚: https://meltano-sdk--2248.org.readthedocs.build/en/2248/