Deprecate PART_SIZE_MB in connectors using S3/GCS storage #13753

VitaliiMaltsev · 2022-06-14T10:38:57Z

What

Following up on work from #10260, we changed the buffering logic to set some upper limits on the buffer size while buffering a stream before invoking the upload operations using StreamTransferManager.

How

Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager

🚨 User Impact 🚨

There should not be any user impact

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

VitaliiMaltsev · 2022-06-14T10:40:02Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2494598833
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2494598833
No Python unittests run

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-06-14T10:40:14Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2494599595
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2494599595
🐛 https://gradle.com/s/tiqcjm6nme6su

Build Failed

Test summary info:

Could not find result summary

VitaliiMaltsev · 2022-06-14T10:40:33Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2494601262
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2494601262
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 545    354    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1338    564    58%

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-06-14T10:40:52Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2494602739
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2494602739
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 545    354    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1338    564    58%

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-06-14T10:41:58Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2494608322
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2494608322
🐛 https://gradle.com/s/prf6saoqjcs7c

Build Failed

Test summary info:

Could not find result summary

VitaliiMaltsev · 2022-06-14T11:31:16Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2494877791
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2494877791
No Python unittests run

Build Passed

Test summary info:

All Passed

alexandr-shegeda

LGTM. we can proceed with publishing and merging without airbyte review

VitaliiMaltsev · 2022-06-17T10:36:06Z

/publish connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2514960707
🚀 Successfully published connectors/destination-gcs
🚀 Auto-bumped version for connectors/destination-gcs
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2514960707

VitaliiMaltsev · 2022-06-17T10:36:28Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2514962661
🚀 Successfully published connectors/destination-s3
🚀 Auto-bumped version for connectors/destination-s3
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2514962661

VitaliiMaltsev · 2022-06-17T10:36:52Z

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2514964744
🚀 Successfully published connectors/destination-bigquery
🚀 Auto-bumped version for connectors/destination-bigquery
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2514964744

VitaliiMaltsev · 2022-06-19T13:41:24Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2524160524
🚀 Successfully published connectors/destination-redshift
🚀 Auto-bumped version for connectors/destination-redshift
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2524160524

VitaliiMaltsev · 2022-06-19T14:28:46Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2524291063
❌ Failed to publish connectors/destination-snowflake
❌ Couldn't auto-bump version for connectors/destination-snowflake

…ze' into vmaltsev/11389-deprecate-part-size

VitaliiMaltsev · 2022-06-20T10:01:32Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2528154179
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2528154179
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-06-20T10:56:57Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2528448406
🚀 Successfully published connectors/destination-snowflake
🚀 Auto-bumped version for connectors/destination-snowflake
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2528448406

danieldiamond · 2022-06-21T08:37:21Z

FYI snowflake destination version 0.4.29 is failing. confirmed that 0.4.28 still works:

Verification error(s) occurred for CHECK. Errors: [$.loading_method: should be valid to one and only one of the schemas ]

note: we hit the destination API to update the snowflake credentials programmatically. could that be the reason why this version if failing?

danieldiamond · 2022-06-21T08:38:00Z

cc @VitaliiMaltsev

Airbyte Version: 0.39.21-alpha
Source: MySQL (0.5.12)
Destination: Snowflake (0.4.29)

VitaliiMaltsev · 2022-06-21T08:53:44Z

@danieldiamond this message is due to the part_size parameter being removed from the spec.

Snowflake credentials should not contain part_size parameter also

edgao · 2022-06-22T15:29:04Z

@VitaliiMaltsev I think we should remove the additionalProperties: false entries from the specs. Currently we don't have a way to migrate old-style specs (which contain the part_size property), so preexisting connections will fail until they're manually fixed.

VitaliiMaltsev · 2022-06-23T11:51:31Z

@edgao do you want me to create a PR for this?
Which connectors require additionalProperties: false field to be removed?

edgao · 2022-06-23T15:13:16Z

let's just do the ones in this PR for now (bigquery, bigquery-denormalized, gcs, redshift, snowflake) so that users can upgrade them safely. The other ones can be handled later.

VitaliiMaltsev · 2022-06-24T10:02:45Z

let's just do the ones in this PR for now (bigquery, bigquery-denormalized, gcs, redshift, snowflake) so that users can upgrade them safely. The other ones can be handled later.

@edgao please review #14114

* source-notion: Updated docs as per new format (#13708) * Keep doc up-to-date with changes (#13667) `airbyte-db/lib` has been renamed to `airbyte-db/db-lib` * Updated BigQuery, Google Sheets, Facebook Marketing, Salesforce docs (#13717) * initial changes * Edited google Sheets doc * More edits * edited the intro and prereqs for BigQuery * edited the data loading section * more edits * Grammatical edits * Formatting edits * Adds zombie removal tool (#13718) * Adds zombie removal tool * Corrects endpoint adds comments * Adds API links * Changes search logic, fixes escape character * Corrects help text * Fix connection read object building (#13568) Make sure the sourceCatalogId field is set with the value stored in the database. * fix modal overlay problem (#13724) * Improve Gradle build and fix storybook (#13719) * Improve Gradle build and fix storybook * Run all copy tasks after copyDocker * fix stream descriptor typo (#13726) * Save streams to reset in job config when creating reset job (#13703) * save streams to reset in job config when creating reset job * change streamDescriptors to streamsToReset * :bug: Postgres Source: fixed unsupported date-time datatypes during incremental sync (#13655) * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * add timestamp datatype test * Bump version in Dockerfile * auto-bump connector version Co-authored-by: grishick <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Bump version in Dockerfile to match source-postgres version (#13732) * Fail the connection state instead of throwing an exception (#13728) * Fail the connection state instead of throwin exception * Fix test * rm import * Update airbyte-workers/src/main/java/io/airbyte/workers/general/DefaultCheckConnectionWorker.java Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Evan Tahler <[email protected]> * [low-code connectors] add a few unit tests (#13666) * add a couple of unit tests * refactor tests * 🎉 Destination databricks: rename to databricks lakehouse (#13722) * Update Databricks naming * Update destination_spec * Update BOOTSTRAP.md * Update Dockerfile * Update README.md * Update spec.json * Update databricks.md * Update databricks.md * Update airbyte-integrations/connectors/destination-databricks/BOOTSTRAP.md Co-authored-by: LiRen Tu <[email protected]> Co-authored-by: LiRen Tu <[email protected]> * Fallback to parsing datetime and time strings w/ and w/o timezones in case DateTimeParseException is thrown (#13745) * Fall back to parsing w/ or w/o TZ if parsing a date or a time string fails * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump source-postgres-strict-encrypt version to 0.4.23 to match source-postgres version (#13747) * S3 and GCS destinations: Updating processing data types for Avro/Parquet formats (#13483) * S3 destination: Updating processing data types for Avro/Parquet formats * S3 destination: handle comparing data types * S3 destination: clean code * S3 destination: clean code * S3 destination: handle case with unexpected json schema type * S3 destination: clean code * S3 destination: Extract the same logic for Avro/Parquet formats to separate parent class * S3 destination: clean code * S3 destination: clean code * GCS destination: Update data types processing for Avro/Parquet formats * GCS destination: clean redundant code * S3 destination: handle case with numbers inside array * S3 destination: clean code * S3 destination: add unit test * S3 destination: update unit test cases with number types. * S3 destination: update unit tests. * S3 destination: bump version for s3 and gcs * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source Salesforce: fix sync capped streams with more records than page size (#13658) * change logic for counting records * update doc * correct unit test * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Docs: update normalization doc (#13739) * update dbt version in custom normalization (#13742) * Kube fix and Docs for open telemetry integration (#13701) * Create interface, factory for metric client * remove unused func * change count val to use long * PR fix * otel metric client implementation * merge conflicts resolve * build fix * add a test, moved version into deps catalog * fix test * add docs for open telemetry * fix kube setting for otel, and add doc * helm related fields update for opentel * cdk: fix typo in build.gradle.hbs (#13761) * Bump Airbyte version from 0.39.17-alpha to 0.39.18-alpha (#13759) Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> * Update destinations to handle new state messages (#13670) * 🐛 Postgres Source: fixed truncated precision if the value of the milliseconds or seconds is 0 (#13549) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Postgres source strict encrypt trucnated precision bump version (#13769) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version * Postgres source strict encrypt bump version Co-authored-by: Octavia Squidington III <[email protected]> * Migrate StateDB to support per stream states (#13731) * Update StateDB to support per Stream states. * Add `StateType` type * Add `steam_name`, `namespace` and `type` to `state` table. * Set the default StateType to LEGACY * Update getting-started-with-airbyte-cloud.md (#13741) * Update getting-started-with-airbyte-cloud.md Edited Getting Started with Airbyte Cloud guide to match the updated Cloud UI. * Update getting-started-with-airbyte-cloud.md Updated based on Amruta's suggestions. * 🐛 Normalization correctly propagates deletions to the final tables (#12846) * Update Airbyte Protocol Docs (#13709) * Source Facebook marketing: fix `not syncing any data` issue (#13749) * #253 oncall source fb marketing: fix issue * #253 oncall: upd changelog * Update docs/integrations/sources/facebook-marketing.md * auto-bump connector version Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Ignore custom connectors during connectors update (#13581) * Ignore Custom connectors during connectors update * Skip any connector flagged as custom as well * Format * [low-code connectors] Support for list-based stream slicers (#13668) * list slicer * Add comment * test * ast eval * refactor * fix * remove print * rename * return a generator * smaller fix and reset to master * union type hint * assert * comment * format * 📝 Postgres source: add troubleshooting for postgres hot standby (#13784) * Add troubleshooting section in postgres doc * Update doc * Add one more solution * upgrade debezium version for postgres to 1.9.2 (#13368) * upgrade debezium version for postgres to 1.9.2 * fix test+build * fix build * address review comments * update link to docs * fix test * 🐛 Source Amplitude: Fix an infinite loop (#13638) * Fix infinite loop when fetching Amplitude data * Update changelog and version * Address review * fix: unit tests were failing * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * octavia-cli: allow user to set custom API HTTP headers(#12893) * 🎉 Source Github: Fix API sorting, fix `get_starting_point` caching (#13707) Signed-off-by: Sergey Chvalyuk <[email protected]> * 🐛 Source Zendesk Support: fixed 429 error for `TicketMetrics` stream (#13757) * Add some dev-facing normalization docs (#13780) * Fix mysql cdc acceptance test (#13795) * fix mysql cdc acceptance test * remove comments * doc: add link to forum in plopfile (#13560) * Make connector adaptable based on deployment mode (#13522) * Add deployment mode to env shared with jobs * Add adaptive runners * Migrate postgres source to use adaptive runner * Add an array of specs in docker image spec definition * Add copyright * Parse docker image spec with specs list * Update spec yaml files * Pass in DEPLOYMENT_MODE to docker compose file * Revert "Parse docker image spec with specs list" This reverts commit 8fe41dd3b7fa5306ba8b62d660c002420adb9b23. * Revert changes in docker image spec * Read cloud specific spec files based on deployment mode * Revert "Update spec yaml files" This reverts commit 059f326432dba1f8da4ad4dfd8d5ac7a14de66c3. * Publish cloud spec file if necessary * Fix upload script * Move test files * Update docker compose file * Format code * Add comment about spec filename * Add unit tests * Remove redundant jdbc acceptance test When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog: - `wakeup_at` - `last_visited_at` - `last_comment_at` These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally. The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test. Given that the JDBC strict encrypt test is basically redundant, it is removed. * Update color palette, add scss colors, borders, and spacing (#13802) * Add updated color palette to SCSS and theme * Migrate current color palette to new colors * Add border and spacing variables * Remove legacy color mapping from scss side * Add style guide to front end repo (#13418) * Add .md from google doc draft * update per review * cleanup * Update STYLEGUIDE.md Co-authored-by: Tim Roes <[email protected]> * Edited Stripe, HubSpot, Intercom docs (#13810) * Initial edits * Updated the Stripe connector doc * Updated the Stripe doc * Updated HubSpot and Intercom docs * Minor sidebar edit * Remove the catalog from the sidebar since the file has been deleted in Update Airbyte Protocol Docs (#13709) (#13813) * Fix broken link (#13815) * fixing broken links * docusaurus updates * Add METRIC_CLIENT and OTEL_COLLECTOR_ENDPOINT to .env files to fix kube deployments (#13817) * octavia-cli: per workspace state (#13070) * Fix LabeledSwitch component width (#13798) * Fix LabeledSwitch component width * Switch to new color palette * Replace color * Fix color variable * Update 4-connection-checking.md (#13812) * Update 4-connection-checking.md As discussed in https://airbytehq-team.slack.com/archives/C02UQ9MJ4GG/p1655312684160609 - this example is misleading, as it does not actually connect to any API to validate connectivity. * Update 4-connection-checking.md Added a link to OneSignal check_connection, and move the "Note" to below the example * Bump Airbyte version from 0.39.18-alpha to 0.39.19-alpha (#13820) Co-authored-by: edgao <[email protected]> * Normalization: Upgrade MySQL to dbt 1.0.0 (#11470) * Restart workflow after activity failure instead of quarantining (#13779) * use SHORT_ACTIVITY_OPTIONS on check connection activity so that it has retries * retry workflow after delay instead of quarantining * allow activity env vars to be configured in docker and kube * add env var for workflow restart delay and refactor slightly * update tests to handle new restart behavior * update test name * add empty env var values to .env files * fail attempt before job in cleanJobState to prevent state machine failure * change default value of max activity attempt retries from 10 to 5 * Bump Airbyte version from 0.39.19-alpha to 0.39.20-alpha (#13829) Co-authored-by: lmossman <[email protected]> * 🎉 Postgres source: prepare for removing strict-encrypt variant (#13823) * Bump postgres version * Update doc * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * update readme (#13836) * add evan, jonathan, charles as protocol reviews in codeowners (#13776) * 📚 update MSSQL docs to reflect that we support normalization (#13826) * [low-code connectors] Cartesian product stream slicer (#13740) * list slicer * Add comment * product stream slicer * comment * rename * format * Update comment * split on 2 lines for readability * Revert "rename" This reverts commit e801f2d1f859306c6a2b0d66d265753ee9e32a54. * updated stacktrace format in java trace messages (#13847) * updated stacktrace format in java trace messages * test checks specifically on stacktrace in trace message * remove unused import * source-facebook-marketing: 🎉 Add fields to the ads_set stream (#13623) * 🎉 Source Github: use GraphQL for `pull_request_stats` stream (#13763) Signed-off-by: Sergey Chvalyuk <[email protected]> * Tweak schema validation code for readability. (#13721) * Refactor schema validation for better reading. * Better comments. * Format. * source-facebook-marketing: bump to 0.2.53 (#13853) * Source Hubspot - fix missing data issue (#13837) * #278 oncall: source hubspot - fix missing data issue * #278 oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source Amplitude: try-catch BadZipFile exception (#13846) * #272 oncall: source Amplitude - try-catch BadZipFile exception * #272-oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🐛 Server can display streams with the same name in different namespaces (#13855) * Updated connector catalog (#13862) * Fix the running job counter (#13833) * Add StateType and StateWrapper objects to the model (#13858) * Emit orphan running jobs metrics (#13865) That metrics show the number of jobs currently flagged as running that are associated to inactive or deprecated connections. * Add meaningful tests for OTEL client (#13831) * Add meaningful tests for OTEL client * PR comment fix * code clean ups * Persist state to reset connection job config (#13867) * add current connection state to JobResetConnectionConfig * pass state from reset connection config to job sync config in generate input activity * format * Source Google Analytics: Add threshold_days option, uncomment statement for incremental tests (#13871) * Harshith/test pr 12536 (#13874) * 🎉 New Source: Firebolt (#1) * fix: Boolean type cast * test: Improve format testing and doc * refactor: Move some db functionality * docs: Adding types doc link in utils * feat: Use future-proof Auth in SDK * fix: integration tests are failing * chore: update seed file Co-authored-by: Petro Tiurin <[email protected]> Co-authored-by: ptiurin <[email protected]> * :bug: Source Relational DB : remove unicode nulls from cursors (#13854) * Make sure that cursor value doesn't contain unicode null. * Test that cursors doesn't contain unicode null * incr version * enableReinitialize on account form (#13622) * #13888 source instagram: fix build (#13889) * Fix md and lg spacing scss vars (#13886) * Publishing java connectors with better formatted stacktrace in Trace messages (#13852) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Fix md and lg spacing scss vars (#13886) * Delete TEMP-testing-command.yml * auto-bump connector version * Delete publish-command.yml * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * Update _variables.scss Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * octavia-cli: fix workspace not having anonymous_data_collection property (#13869) * Update connection update calls to use central utility to ensure connection update has all data (#13564) * Update connection updates with build update utility * Add buildConnectionUpdate utility * Update components that update the connection to use utility when necessary * Use conection name when saving connection from replication view to prevent override from refreshed catalog * Improve connection check on ReplicationView onSubmit function * Display connection state in connection setting page (#13394) * Display Connection State in Setting page * memoize callback * rendering and confirmaton * setState API * Input validation * remove JSON step * rename apiMethod to `updateState` * test and adjust route * skip if sync is running * prevent state update when sync is running * code editor component * errors fixed * scss style * make linter happy * Back to monaco editor * Remove ability to edit state * Adjust FE code * Fix CSS problem * Update airbyte-webapp/src/locales/en.json Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * just use PRE to render state for now Co-authored-by: Tim Roes <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * update api for per stream (#13835) * Update airbyte-protocol.md (#13892) * Update airbyte-protocol.md * Fix typo * Fix prose * Add protocol reviewers for protocol documentation * Remove duplicate * Edited Amplitude, Mailchimp, and Zendesk Support docs (#13897) * deleting SUMMARY.md since we don't need it for docusaurus builds (#13901) * Do not hide unexpected errors in the check connection (#13903) * Do not hide unexpected errors in the check connection * Fix test * Common code to deserialize a state message in the new format (#13772) * Common code to deserialize a state message in the new format * PR comments and type changed to typed * Format * Add StateType and StateWrapper objects to the model * Use state wrapper instead of Either * Switch to optional * PR comments * Support array legacy state * format Co-authored-by: Jimmy Ma <[email protected]> * 🐛 Source Amazon Seller Partner: handle start date for financial stream (#13633) * start and end date for finacial stream should not be more than 180 days apart * improve unit tests * make changes to start date for finance stream * update tests * lint changes * update version to 0.2.22 for source-amazon-seller-partner * Normalization: Fix incorrect jinja2 macro `json_extract_array` call (#13894) Signed-off-by: Sergey Chvalyuk <[email protected]> * Docs: fixed the broken links (#13915) * 0.2.5 -> 0.2.6 (#13924) Signed-off-by: Sergey Chvalyuk <[email protected]> * 13546 Fix integration tests source-postgres Mac OS (#13872) * 13546 Fix integration tests source-postgres Mac OS * 13548 Fixed integration tests source-tidb Mac OS (#13927) * Source MsSql : incr ver to include changes #13854 (#13887) * incr version * put PR id * docker ver * connectors that published (#13932) * Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753) * Removed part_size from connectors that use StreamTransferManager * fixed S3DestinationConfigTest * fixed S3JsonlFormatConfigTest * upadate changelog and bump version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * upadate changelog and bump version for Redshift and Snowflake destinations * auto-bump connector version * fix GCS staging test * fix GCS staging test * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Reverted changes in SshBastionContainer (#13934) * 🎉 New Source Dockerhub (#13931) * init * implement working source + tests * add docs * add docs * fix bad comments * Update airbyte-integrations/connectors/source-dockerhub/acceptance-test-config.yml * Update airbyte-integrations/connectors/source-dockerhub/Dockerfile * Update airbyte-integrations/connectors/source-dockerhub/.dockerignore * Apply suggestions from code review * Update docs/integrations/sources/dockerhub.md * Update airbyte-integrations/connectors/source-dockerhub/integration_tests/acceptance.py Co-authored-by: George Claireaux <[email protected]> * address @Phlair's feedback * address @Phlair's feedback * each record is now a Docker image rather than response page * format * fix unit tests * fix acceptance tests * add icon, definition and generate seed spec * add requests to requirements Co-authored-by: sw-yx <[email protected]> * commented out non-relevant tests (#13940) * Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha (#13938) Co-authored-by: alafanechere <[email protected]> * newaction (#13942) * remove test action (#13944) * 🎉Source-mysql: aligned datatype test (#13945) * [13607] source-mysql: aligned datatype tests for regular and CDC ways + added CHAR fix to CDC processing * #13958 Source Stripe: fix configured catalogs (#13959) * 🐛 Source: Typeform - Update schema for Responses stream (#13935) * Upd responses schema * Upd docs * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :window: Updated email invitation flow that enables invited users to set name and create password (#12788) * First pass accepting email link invitation * Update Auth service with signInWithEmailLink calls * Add AcceptEmailInvite component * Update FirebaseActionRoute to handle sign in mode * Rename ResetPasswordAction to FirebseActionRoute * Add create password setp to AcceptEmailInvite component * Remove continueURL from invite fetch * Update accept email invite for user to enter both email and password together * Set name during email link signup * Update AcceptEmailInvite to send name * Add updateName to UserService * Update AuthService to set name during sign up * Remove steps from AcceptEmailInvite component Remove setPassword from AuthService * Add header and title to accept invite page * Move invite error messages to en file * For invite link pages, show login link instead of sign up * Disable name update on sign in via email lnk * Resend email invite when the invite link is expired * Fix status message in accept email invite page * Re-enable set user's name during sign up email invite * Update signUpWithEmailLink so that sign up is successful even if we fail to update the user's name * Update comments on GoogleAuthService signInWithEmailLink * Add newsletter and accept terms checkboxes to accept email invite component * Extract signup form from signup page * Extract fields from signup form * Update accept email invite component to use field components from signup form * Ensure that sign up button is disable until form is valid and security checkbox is checked * Make error status text color in accept email link red * Update workspace check in DefaultView so that user lands in workspace selector when there are no workspaces * Add coment around continueUrl param usage in UserService * Remove usless default case in GoogleAuthService * Source Marketo: process fail during creation of an export job (#13930) * #9322 source Marketo: process fail during creation of an export job * #9322 source marketo: upd changelog * #9322 source marketo: fix unit test * #9322 source marketo: fix SATs * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :window: :wrench: Add eslint rules for CSS modules (#13952) * add eslint-plugin-css-modules rules * Fixes: - turn on eslint css modules rule as error - remove unused styles * add warning message if styled components is used * Revert "add warning message if styled components is used" This reverts commit 4e92b8b2110142bb679f15aeb034e377e0dcc69c. * replace rule severity with words * Update salesforce.md Fixed broken link * :window: 🔧 Add auto-fixable linting rules to webapp (#13462) * Add new eslint rules that fit with our code style and downgrade rules to warn * allowExpressions in fragment eslint rule * Enable function-component-definition in eslint and fix styles * Cleanup lint file * Fix react/function-component-definition warnings manually * Add more auto-fixable rules and fix * Fix functions that require usless returns * Update array-type rule to array-simple * Fix eslint errors manually disable assignmentExpression for arrays in prefer-destructuring rule * Auto fix new linting issues after rebase * Enhance /publish to allow for multiple connectors and parallel execution (#13864) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Delete TEMP-testing-command.yml * auto-bump connector version * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * destinations * + snowflake * saved * auto-bump connector version * auto-bump connector version * java source bumps * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * remove twice-defined methods * label things * revert action * using the new test action * point at action * wrong tag on action * update pool label * update to use new ec2-github-runner fork * this needs to be more generic than publisher * change publish to run on pool * add comment about runner-pool usage * updated publish command docs for multi & parallel connector runs * auto-bump connector version * auto-bump connector version * auto-bump connector version * unbump failed publish versions * missed dockerfiles * remove failed docs * mssql fix * overhauled the git comment output * bumping a test connector that should work * slight order switcheroo * output connectors properly in first message * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump Airbyte version from 0.39.21-alpha to 0.39.22-alpha (#13979) Co-authored-by: Phlair <[email protected]> * Parker/temporal cloud (#13243) * switch to temporal cloud client for now * format * use client cert/key env secret instead of path to secret * add TODO comments * format * add logging to debug timeout issue * add more logging * change workflow task timeout * PR feedback: consolidate as much as possible, add missing javadoc * fix acceptance test, needs to specify localhost * add internal-use only comments * format * refactor to clean up TemporalClient and prepare it for future dependency injection framework * remove extraneous log statements * PR feedback * fix test * return isInitialized true in test * 📄 Postgres source: fix CDC setup order in docs (#13949) * postgres source: fix CDC setup order docs * Update docs/integrations/sources/postgres.md Co-authored-by: Liren Tu <[email protected]> * Per-stream state support for Postgres source (#13609) * WIP Per-stream state support for Postgres source * Fix failing test * Improve code coverage * Make global the default state manager * Add legacy adapter state manager * Formatting * Include legacy state for backwards compatibility * Add global state manager * Implement Global/CDC state handling * Fix test issues * Fix issue with updated method signature * Handle empty state case in global state manager * Adjust to protocol changes * Fix failing acceptance tests * Fix failing test * Fix unmodifiable list issue * Fix unmodifiable exception * PR feedback * Abstract global state manager selection * Handle conversion between different state types * Handle invalid conversion * Rename parameter * Refactor state manager creation * Fix failing tests * Fix failing integration tests * Add CDC test * Fix failing integration test * Revert change * Fix failing integration test * Use per-stream for postgres tests * Formatting * Correct stream descriptor validation * Correct permalink * PR feedback * Bump Airbyte version from 0.39.22-alpha to 0.39.23-alpha (#13984) Co-authored-by: pmossman <[email protected]> * Adds test for new workflow (#13986) * Adds test for new workflow * Adds airbyte repo * remove testing line * Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472) * write out new request options provider and refactor components and parts of the YAML config * fix formatting * pr feedback to consolidate body_data_provider to simplify the code * pr feedback get rid of extraneous optional * publish oss for cloud (#13978) workflow to publish oss artifacts that cloud needs to build against use docker buildx to create arm images for local development * skip debezium engine startup in case no table is in INCREMENTAL mode (#13870) * 🎉 Source Github: break point added for workflows_runs stream (#13926) Signed-off-by: Sergey Chvalyuk <[email protected]> * 6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866) * 6339: debug info * 6339: not using 'USE' on Azure SQL servers * 6339: cleanup * 6339: cleanup2 * 6339: cleanup3 * 6339: versions/changelogs updated * 6339: merge from master (consolidation issue) * 6339: dev connector version (for testing in airbyte cloud) * 6339: code review implementation * 6339: apply formatting * in case runners fail to spin up, this needs to run on github-hosted (#13996) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * :tada: New Source: Webflow (#13617) * Added webflow code * Updated readme * Updated README * Added webflow to source_definitions.yaml * Enhanced documentation for the Webflow source connector * Improved webflow source connector instructions * Moved Site ID to before API token in Spec.yaml (for presentation in the UI) * Addressed comments in PR. * Changes to address requests in PR review * Removed version from config * Minor udpate to spec.yaml for clarity * Updated to pass the accept-version as a constant rather than parameter * Updated check_connection to hit the collections API that requires both site id and the authentication token. * Fixed the test_check_connection to use the new check_connection function * Added a streams test for generate_streams * Re-named "autentication" object to "auth" to be more consistent with the way it is created by the CDK * Added in an explict line to instantiante an "auth" object from WebflowTokenAuthenticator, to make it easier to describe in the blog * Fixed a typo in a comment * Renamed some classes to be more intuitive * Renamed class to be more intuitive * Minor change to an internal method name * Made _get_collection_name_to_id_dict staticmethod * Fixed a unit-test error that only appeared when running " python -m pytest -s unit_tests". This was caused by Mocked settings from test_source.py leaking into test_streams.py * format: add double quotes and remove unused import * readme: remove semantic version naming of connector in build commands * Updated spec.yaml * auto-bump connector version * format files * add changelog * update dockerfile * auto-bump connector version Co-authored-by: sajarin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: marcosmarxm <[email protected]> * Source-oracle: fixed tests + checkstyle (#13997) * Source-oracle: fixed tests + checkstyle * 🐛Destination-mysql: fixed integration test and build process (#13302) * [13180] destination-mysql: fixed integration test * update changelog to include debezium version upgrade (#13844) * make table headers look less like successes (#13999) * source-twilio: implement lookback windows (#13896) * Revert "12708: Add an option to use encryption with staging in Redshift Destination (#13675)" (#14010) This reverts commit aa28d448d820df9d79c2c0d06b38978d1108fb2c. * Revert "6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866)" (#14011) This reverts commit 0d870bd37bc3b5cd798b92115d73bcc45a42d8f7. * [low-code connectors] BasicHttpAuthenticator (#13733) * implement basichttpauthenticator * add optional refresh access token authenticator * remove prints * type hints * Fix and unit test * missing test * Add class to __init__ file * Add comment * migrate JsonSchemas to use basic path instead of JSONPath (#13917) * scaffold for catalog diff, needs fixing on type handling and tests (#13786) * Prepare release of JDBC connectors (#13987) * Prepare release of JDBC connectors * Update source definitions manually * use built in check for if path is definite (#13834) * 13535 Fixed bastion network for integration tests (#14007) * doc: add error troubleshooting `docker-compose up` (#13765) * fix: duplicate resource allocations in `airbyte-temporal` deployment (#13816) * helm-chart: Fix worker deployment format error (#13839) * add catalog diff connection read (#13918) * doc: fix small typo on Shopify documentation (#13992) * add streams to reset to job info (#13919) * Generate api for changes in #13370 and make code compatible (#14014) * Generate api for per-stream updates #13835 (#14021) * Revert "Prepare release of JDBC connectors (#13987)" (#14029) This reverts commit df759b30778082508e2872513800fac34d98ff7c. * Fix per stream state protocol backward compatibility (#14032) * rename state type field to fix backwards compatibility issue * replace usages of stateType with type * support semi incremental by adding extractor record filter (#13520) * support semi incremental by adding extractor record filter * refactor extractor into a record_selector that supports extraction and filtering of response records * Remove pydantic spec from amazon ads and use YAML spec (#13988) * add EdDSA support in SSH tunnel (#9494) * add EdDSA support * verify EdDSA support works correct Co-authored-by: Yurii Bidiuk <[email protected]> * 🎉New source connector: source-metabase (#13752) * Add docs * Close metabase session when sync finishes * Close session in check_connection * Add source definition to seed * Add icon * improve cdc check for connectors (#14005) * improve should use cdc check * Revert "improve should use cdc check" This reverts commit 7d01727279d21d33a6c18ed3227ee94432636120. * improve should use cdc check * add unit test * Update webflow.md * Update webflow.md * Update webflow.md * Remove legacy sentry code from cdk (#14016) * rip sentry out of cdk * remove sentry dsn from gsc * Update webflow.md * Update webflow.md * Fixed broken links (#14071) * 🪟Persist unsaved changes on schema refresh (#13895) * add form values tracker context * add clarifying comment * add same functionality to create connection * Update airbyte-webapp/src/components/CreateConnectionContent/CreateConnectionContent.tsx Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * Fixes broken links so we can deploy again (#14075) also adds better error message for when this happens to others * Adds symmary.md to gitignore (#14078) * Added webflow icon (#14069) * Added webflow icon * Added icon * Build create connection form build failure (#14081) * Fix CDK obfuscation of nested secrets (#14035) * Added Buy Credits section to Managing Airbyte Cloud (#13905) * Added Buy Credits section to Managing Airbyte Cloud * Made some style changes * Made edits based on Natalie's suggestions * Deleted link * Deleted line * Edited email address * Updated reaching out to sales sentence * disable es-lit to fix build (#14087) * Release source connectors (#14077) * Release source connectors * Fix issue with database connection in test * Fix failing tests due to authentication * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump Airbyte version from 0.39.23-alpha to 0.39.24-alpha (#14094) Co-authored-by: jdpgrailsdev <[email protected]> * Emit the state to remove in the airbyte empty source (#13725) What This PR updates the EmptyAirbyteSource in order to perform a partial update and handle the new state message format. How The empty will now emit different messages based on the type of state being provided: Per stream: it will emit one message per stream that have been reset Global: It will emit one global message that will contain null for the stream that have been reset including the shared state Co-authored-by: Jimmy Ma <[email protected]> * Add StatePersistence object (#13900) Add a StatePersistence object that supports Read/Writes of States to the DB with StreamDescriptor fields The only migrations that is supported are * moving from LEGACY to GLOBAL * moving from LEGACY to STREAM * All other state type migrations are expected to go through an explicit reset beforehand. * secret-persistence: Hashicorp Vault Secret Store (#13616) Co-authored-by: Amanda Murphy <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> * 🐛 Source Hubspot: remove `AirbyteSentry` dependency (#14102) * fixed * updated changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * fix: format VaultSecretPersistenceTest.java (#14110) * Source Hubspot: extend error logging (#14054) * #291 incall - source Hubspot: extend error logging * huspot: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Update webflow.md (#14083) * Update webflow.md Removed a description that is only applicable to people that are writing connector code, not to _users_ of the connector. * Update webflow.md * Update webflow.md * Update webflow.md * Update webflow.md * 12708: Add an option to use encryption with staging in Redshift Desti… (#14013) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * 12708: specs updated * 12708: specs updated * 12708: removing autogenerated files from PR * 12708: changelog updated * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source PayPal Transaction: Update Transaction Schema (#13682) * Update transaction schema. * Transform money values from strings to floats or integers. Co-authored-by: nataly <[email protected]> Co-authored-by: Augustin <[email protected]> * fix(jsonSchemas): raise error when items property not provided (#14018) * fix stream name in stream transformation update (#14044) * 🐛 Destination Redshift: Improved discovery for redshift-destination not SUPER streams (#13690) airbyte-12843: Improved discovery for redshift-destination not SUPER tables, excluded views from discovery. * Remove skiptests option (#14100) * update sentry release script (#14123) * Remove "additionalProperties": false from specs for connectors with staging (#14114) * Remove "additionalProperties": false from spec for connectors with staging * Remove "additionalProperties": false from spec for Redshift destination * bump versions * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * [14003] source-oracle: added custom jdbc field (#14092) * [14003] source-oracle: added custom jdbc field * Add JobErrorReporter for sending sync job connector failures to Sentry (#13899) * skeleton for reporting connector errors to sentry * report on job failures instead of attempt failures * report sync job failures with relevant metadata using JobErrorReporter * send stack traces from python connectors to sentry * test JobCreationAndStatusUpdate and JobErrorReporter * logs * refactor into helper, initial tests * using sentry * run format * load reporting client from env * load sentry dsn from env * send java stack traces to sentry * test sentryclient, refactor to use Hub instance * ErrorReportingClient.report -> .reportJobFailureReason * inject exception helper, test stack trace parse error tagging * rm logs * more stack trace tests * remove logs * fix failing tests * rename ErrorReportingClient to JobErrorReportingClient * rename vars in docker-compose * Return an Optional instead of null when parsing stack traces * dont remove airbyte prefix when setting release name * from_trace_message static * remove failureSummary from jobfailure input, get from Job * send stacktrace string if we weren't able to parse * set deployment mode tag * update .env * just log if something goes wrong * Use StateMessageHelper in source (#14125) * Use StateMessageHelper in source * PR feedback and formatting * More PR feedback * Revert change * Revert changes * Bump Airbyte version from 0.39.24-alpha to 0.39.25-alpha (#14124) Co-authored-by: brianjlai <[email protected]> * Refactor acceptance tests and utils (#13950) * Refactor Basic acceptance tests and utils * Refactor Advanced acceptance tests and utils * Remove unused code * Clear destination db data during cleanup * Cleanup comments * cleanup init code * test creating new desintation db for each test * cleanup desintation db init * Allow to edit api client * pull in temporal cloud changes * Rename helper to harness; set some funcs to private; turn init into constructor * add func to set env vars instead of using static vars and move some functionality out of init into acceptance tests * update javadoc Co-authored-by: Davin Chia <[email protected]> * fix javadoc formatting * fix var naming Co-authored-by: Davin Chia <[email protected]> * Bump Airbyte version from 0.39.25-alpha to 0.39.26-alpha (#14141) Co-authored-by: terencecho <[email protected]> * 🎉 octavia-cli: Add ability to get existing resources (#13254) * 13541 Fixed integration tests source-db2 Mac OS (#14133) * 13523 Fix integration tests destination-cassandra Mac OS (#14134) * 🐛 Source Hubspot: fixed SAT test, commented out expected_records (#14140) * :bug: Source Intercom: extend `Contacts` schema with new properties (#14099) * Source Twilio: adopt best practices (#14000) * #1946 Source twilio: aopt best practices - tune tests * #1946 add expected_records to acceptance-test-config.yml * #1946 source twilio - upd schema and changelog * #1946 fix expected_records * #1946 source twilio: rm alerts from expected records as they expire in 30 days * #1946 source twilio: bump version * 🎉 Source BingAds: expose hourly/daily/weekly/monthly options from configuration (#13801) * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: config settings for hourly/daily/weekly/monthly reports added: default value for all periodic reports to True * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused class variables, if-statement * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused variables from config * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * remove VersionMismatchServer (#14076) * remove VersionMismatchServer * remove VersionMismatchServerTest * revert intended changes * Increase instance termination time limit to 3 hours to accommodate connector builds. (#14181) * Use correct bash comment symbol. (#14183) * 🎉 New Source: Orbit.love (#13390) * source-orbit: add definition and specs (#14189) * 🎉 Base Norrmalization: clean-up Redshift `tmp_schemas` after SAT (#14015) Now after `base-normalization` SAT the Destination Redshift will be automatically cleaned up from test leftovers. Other destinations are not covered yet. * Source Salesforce: fix customIntegrationTest for SAT (#14172) * Source Amazon Ads: increase timeout for SAT (#14167) * 🎉 Introduce Google Analytics Data API source (#12701) * Introduce Google Analytics Data API source https://developers.google.com/analytics/devguides/reporting/data/v1 * Add Google Analytics Data API source PR link * Add `client` class for Google Analytics Data API * Move dimensions and metrics extraction to the `client` class In the Google Analytics Data API * Change the copyright date to 2022 in Google Analytics Data API * fix: removing incremental syncs * fix: change project_id to string * fix: flake check is failing * chore: added it to source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * 🐛 Destination Redshift: use s3 bucket path for s3 staging operations (#13916) * Publish acceptance test utils maven artifact (#14142) * Fix StatePersistence Legacy read/write (#14129) StatePersistence will wrap/unwrap legacy state on write/read to ensure compatibility with the old behavior/data. * 🎉 Destination connectors: Improved "SecondSync" checks in Standard Destination Acceptance tests (#14184) * [11731] Improved "SecondSync" checks in Standard Destination Acceptance tests * 🐛 Source Zendesk Support: fixed "Retry-After" non integer value (#14112) Signed-off-by: Sergey Chvalyuk <[email protected]> * Source Tiktok Marketing: Videometrics (#13650) * added video metrics in streams.py * common metrics list updated. * updated streams.py with extended metrics required. * updated stream_test * updated configured_catalog * video metrics required list updated. * chore: formatting * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * 🎉 Source Github: secondary rate limits has to retry (#13955) Signed-off-by: Sergey Chvalyuk <[email protected]> * Harshith/test pr 13118 (#14192) * Firebolt destination * feat: Write method dropdown * feat: Use future-proof Auth in SDK * refactor: Move writer instantiation * fix: tests are failing * fix: tests are failing * fix: tests are failing * chore: added connector to definitions * fix: formatting and spec * fix: formatting for orbit Co-authored-by: ptiurin <[email protected]> * 🪟 :art: Show credit usage on chart's specific day (#13503) * add tooltip to chart * Fixes: - update main chart color; - change onHover background color * change chart color pallet to grey 500 * update color reference * remove opacity from UsageCell * 🐛 destination-redshift: use s3 bucket path for s3 cleanup (#14190) * Improve documentation for Postgres Source (#13830) * Improve documentation for Postgres Source * add information about additional JDBC params * add anchors for doc sections * fix link to CDC on Bare Metal * add more details about parsing date/time values * add doc link to SSH fields * Handle null reset source config (#14202) * handle null reset source config * format * Wait indefinitely if connection is not active (#14200) * also wait indefinitely if connection is deleted * fix test * Bump Airbyte version from 0.39.26-alpha to 0.39.27-alpha (#14204) Co-authored-by: lmossman <[email protected]> * Bmoric/feature flag for state deserialization (#14127) * Add Feature flag * Add default feature flag value * Update test * remove unsused * tmp * Update tests * rm unwanted change * PR comments * [low-code connectors] default types and default values (#14004) * default types and default values * cleanup * fixes so read works * remove prints and trycatch * comment * remove unused param * split file * extract method * extract methods * comment * optional * fix test * cleanup * delete interpolated request header provider * simplify next page url paginator interface * comment * format * add state type endpoint (#14111) * Bump Airbyte version from 0.39.27-alpha to 0.39.28-alpha (#14210) Co-authored-by: sherifnada <[email protected]> * 🐛 source-orbit: remove workspace_old.json (#14208) * Fix: Docs plural login redirecting to wrong URL (#14207) * [docs] fix numbering and incorrect filename in CDK docs (#13045) * [docs] fix numbering in CDK docs * Update 5-declare-schema.md * Update 5-declare-schema.md * Update 6-read-data.md * Update 8-test-your-connector.md * Remove the old scheduler from HelmCharts helper (#14187) * Remove the old scheduler from HelmCharts helper The old scheduler was removed as part of https://github.com/airbytehq/airbyte/pull/13400 * Remove legacy `scheduler` comment in HelmCharts * Source Gitlab: add GroupIssueBoards stream (#13252) * GitLab Source: add GroupIssueBoards stream * Address stream schema comments * Address comments * Bump version * Add as empty stream * run seed file source (#14215) * fix 'cannot reach server' error on demo instance (#10020) * Update CODEOWNERS (#14209) * 🎉 Source Github: use GraphQL for `reviews` stream (#13989) Signed-off-by: Sergey Chvalyuk <[email protected]> * workflow for publishing artifacts for cloud (#14199) * fix sentry org slug change (#14218) * Source File: correct spec json to match json format (#13738) * Upgrade spotless version and remove jvmargs workaround (#13705) * Source Zendesk Chat: Process large amount of data in batches for incremental (#14214) * increased the limit of itens in request * Configuration for max api pages on requests * included api_pagination_limit in sample * included api_pagination_limit in invalid_config * creating new table for chat_session * reverted api_pagination_limit approach * removed api_pagination_limit from TimeIncrementalStream * correct chat json * bump connector version * add changelog * run format * auto-bump connector version Co-authored-by: Roberto Bonnet <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Remove all @ts-ignore (#14221) * Bump hadoop to use version 3.3.3 (#14182) * Change the persistence activity to use the new persistence layer (#14205) * Change the persistence activity to use the new persistence layer * Use lombok * format * Use new State message helper * Fix build (#14225) * Fix build * Fix test * Use new state persistence for state reads (#14126) * Inject StatePersistence into DefaultJobCreator * Read the state from StatePersistence instead of ConfigRepository * Add a conversion helper to convert StateWrapper to State * Remove unused ConfigRepository.getConnectionState * Temporal per stream resets (#13990) * remove reset flags from workflow state + refactor * bring back cancelledForReset, since we need to distinguish between that case and a normal cancel * delete reset job streams on cancel or success * extract isResetJob to method * merge with master * set sync modes on streams in reset job correctly * format * Add test for getAllStreamsForConnection * fix tests * update more tests * add StreamResetActivityTests * fix tests for default job creator * remove outdated comment * remove debug lines * remove unused enum value * fix tests * fix constant equals ordering * make job mock not static * DRY and add comments * add comment about deleted streams * Remove io.airbyte.config.StreamDescriptor * regisster stream reset activity impl * refetch connection workflow when checking job id, since it may have been restarted * only cancel if workflow is running, to allow reset signal to always succeed even if batched with a workflow start * fix reset signal to use new doneWaiting workflow state prop * try to fix tests * fix reset cancel case * add acceptance test for resetting while sync is running * format * fix new acceptance test * lower sleep on test * raise sleep * increase sleep and timeout, and remove repeated test * use CatalogHelpers to extract stream descriptors * raise sleep and timeout to prevent transient failures * format Co-authored-by: alovew <[email protected]> Co-authored-by: Luis Gomez <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Amruta Ranade <[email protected]> Co-authored-by: Topher Lubaway <[email protected]> Co-authored-by: Malik Diarra <[email protected]> Co-authored-by: Teal Larson <[email protected]> Co-authored-by: Tim Roes <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: VitaliiMaltsev <[email protected]> Co-authored-by: grishick <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Greg Solovyev <[email protected]> Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: Andy <[email protected]> Co-authored-by: LiRen Tu <[email protected]> Co-authored-by: oneshcheret <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Xiaohan Song <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Charles <[email protected]> Co-authored-by: Sophia Wiley <[email protected]> Co-authored-by: Edward Gao <[email protected]> Co-authored-by: Denis Davydov <[email protected]> Co-authored-by: Subodh Kant Chaturvedi <[email protected]> Co-authored-by: Bastien Gandouet <[email protected]> Co-authored-by: Harshith Mullapudi <[email protected]> Co-authored-by: Paweł Kociński <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Baz <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: Alexander Marquardt <[email protected]> Co-authored-by: edgao <[email protected]> Co-authored-by: lmossman <[email protected]> Co-authored-by: George Claireaux <[email protected]> Co-authored-by: mutt-santiago-stachuk <[email protected]> Co-authored-by: Davin Chia <[email protected]> Co-authored-by: Arsen Losenko <[email protected]> Co-authored-by: Petro Tiurin <[email protected]> Co-authored-by: ptiurin <[email protected]> Co-authored-by: Andrii Leonets <[email protected]> Co-authored-by: Jonathan Pearlin <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Ganpat Agarwal <[email protected]> Co-authored-by: Rajakavitha Kodhandapani <[email protected]> Co-authored-by: Yevhen Sukhomud <[email protected]> Co-authored-by: sw-yx <[email protected]> Co-authored-by: alafanechere <[email protected]> Co-authored-by: Eugene <[email protected]> Co-authored-by: Anna Lvova <[email protected]> Co-authored-by: Vladimir <[email protected]> Co-authored-by: Phlair <[email protected]> Co-authored-by: Parker Mossman <[email protected]> Co-authored-by: Adam <[email protected]> Co-authored-by: Liren Tu <[email protected]> Co-authored-by: pmossman <[email protected]> Co-authored-by: Brian Lai <[email protected]> Co-authored-by: Peter Hu <[email protected]> Co-authored-by: Tuhai Maksym <[email protected]> Co-authored-by: sajarin <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: steve withington <[email protected]> Co-authored-by: Leo Sussan <[email protected]> Co-authored-by: cenegd <[email protected]> Co-authored-by: Tomas Perez Alvarez <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Yurii Bidiuk <[email protected]> Co-authored-by: Christophe Duong <[email protected]> Co-authored-by: jdpgrailsdev <[email protected]> Co-authored-by: Stella Chung <[email protected]> Co-authored-by: Amanda Murphy <[email protected]> Co-authored-by: Mohamed Magdy <[email protected]> Co-authored-by: nataly <[email protected]> Co-authored-by: Tyler Russell <[email protected]> Co-authored-by: Alexander Tsukanov <[email protected]> Co-authored-by: Pedro S. Lopez <[email protected]> Co-authored-by: brianjlai <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Daniel Diamond <[email protected]> Co-authored-by: drrest <[email protected]> Co-authored-by: Abhi Vaidyanatha <[email protected]> Co-authored-by: Zawar Khan <[email protected]> Co-authored-by: sherifnada <[email protected]> Co-authored-by: Sachin Jangid <[email protected]> Co-authored-by: Chris Wu <[email protected]> Co-authored-by: Jared Rhizor <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Roberto Bonnet <[email protected]> Co-authored-by: alovew <[email protected]>

* Adds zombie removal tool (#13718) * Adds zombie removal tool * Corrects endpoint adds comments * Adds API links * Changes search logic, fixes escape character * Corrects help text * Fix connection read object building (#13568) Make sure the sourceCatalogId field is set with the value stored in the database. * fix modal overlay problem (#13724) * Improve Gradle build and fix storybook (#13719) * Improve Gradle build and fix storybook * Run all copy tasks after copyDocker * fix stream descriptor typo (#13726) * Save streams to reset in job config when creating reset job (#13703) * save streams to reset in job config when creating reset job * change streamDescriptors to streamsToReset * :bug: Postgres Source: fixed unsupported date-time datatypes during incremental sync (#13655) * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * add timestamp datatype test * Bump version in Dockerfile * auto-bump connector version Co-authored-by: grishick <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Bump version in Dockerfile to match source-postgres version (#13732) * Fail the connection state instead of throwing an exception (#13728) * Fail the connection state instead of throwin exception * Fix test * rm import * Update airbyte-workers/src/main/java/io/airbyte/workers/general/DefaultCheckConnectionWorker.java Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Evan Tahler <[email protected]> * [low-code connectors] add a few unit tests (#13666) * add a couple of unit tests * refactor tests * 🎉 Destination databricks: rename to databricks lakehouse (#13722) * Update Databricks naming * Update destination_spec * Update BOOTSTRAP.md * Update Dockerfile * Update README.md * Update spec.json * Update databricks.md * Update databricks.md * Update airbyte-integrations/connectors/destination-databricks/BOOTSTRAP.md Co-authored-by: LiRen Tu <[email protected]> Co-authored-by: LiRen Tu <[email protected]> * Fallback to parsing datetime and time strings w/ and w/o timezones in case DateTimeParseException is thrown (#13745) * Fall back to parsing w/ or w/o TZ if parsing a date or a time string fails * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump source-postgres-strict-encrypt version to 0.4.23 to match source-postgres version (#13747) * S3 and GCS destinations: Updating processing data types for Avro/Parquet formats (#13483) * S3 destination: Updating processing data types for Avro/Parquet formats * S3 destination: handle comparing data types * S3 destination: clean code * S3 destination: clean code * S3 destination: handle case with unexpected json schema type * S3 destination: clean code * S3 destination: Extract the same logic for Avro/Parquet formats to separate parent class * S3 destination: clean code * S3 destination: clean code * GCS destination: Update data types processing for Avro/Parquet formats * GCS destination: clean redundant code * S3 destination: handle case with numbers inside array * S3 destination: clean code * S3 destination: add unit test * S3 destination: update unit test cases with number types. * S3 destination: update unit tests. * S3 destination: bump version for s3 and gcs * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source Salesforce: fix sync capped streams with more records than page size (#13658) * change logic for counting records * update doc * correct unit test * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Docs: update normalization doc (#13739) * update dbt version in custom normalization (#13742) * Kube fix and Docs for open telemetry integration (#13701) * Create interface, factory for metric client * remove unused func * change count val to use long * PR fix * otel metric client implementation * merge conflicts resolve * build fix * add a test, moved version into deps catalog * fix test * add docs for open telemetry * fix kube setting for otel, and add doc * helm related fields update for opentel * cdk: fix typo in build.gradle.hbs (#13761) * Bump Airbyte version from 0.39.17-alpha to 0.39.18-alpha (#13759) Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> * Update destinations to handle new state messages (#13670) * 🐛 Postgres Source: fixed truncated precision if the value of the milliseconds or seconds is 0 (#13549) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Postgres source strict encrypt trucnated precision bump version (#13769) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version * Postgres source strict encrypt bump version Co-authored-by: Octavia Squidington III <[email protected]> * Migrate StateDB to support per stream states (#13731) * Update StateDB to support per Stream states. * Add `StateType` type * Add `steam_name`, `namespace` and `type` to `state` table. * Set the default StateType to LEGACY * Update getting-started-with-airbyte-cloud.md (#13741) * Update getting-started-with-airbyte-cloud.md Edited Getting Started with Airbyte Cloud guide to match the updated Cloud UI. * Update getting-started-with-airbyte-cloud.md Updated based on Amruta's suggestions. * 🐛 Normalization correctly propagates deletions to the final tables (#12846) * Update Airbyte Protocol Docs (#13709) * Source Facebook marketing: fix `not syncing any data` issue (#13749) * #253 oncall source fb marketing: fix issue * #253 oncall: upd changelog * Update docs/integrations/sources/facebook-marketing.md * auto-bump connector version Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Ignore custom connectors during connectors update (#13581) * Ignore Custom connectors during connectors update * Skip any connector flagged as custom as well * Format * [low-code connectors] Support for list-based stream slicers (#13668) * list slicer * Add comment * test * ast eval * refactor * fix * remove print * rename * return a generator * smaller fix and reset to master * union type hint * assert * comment * format * 📝 Postgres source: add troubleshooting for postgres hot standby (#13784) * Add troubleshooting section in postgres doc * Update doc * Add one more solution * upgrade debezium version for postgres to 1.9.2 (#13368) * upgrade debezium version for postgres to 1.9.2 * fix test+build * fix build * address review comments * update link to docs * fix test * 🐛 Source Amplitude: Fix an infinite loop (#13638) * Fix infinite loop when fetching Amplitude data * Update changelog and version * Address review * fix: unit tests were failing * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * octavia-cli: allow user to set custom API HTTP headers(#12893) * 🎉 Source Github: Fix API sorting, fix `get_starting_point` caching (#13707) Signed-off-by: Sergey Chvalyuk <[email protected]> * 🐛 Source Zendesk Support: fixed 429 error for `TicketMetrics` stream (#13757) * Add some dev-facing normalization docs (#13780) * Fix mysql cdc acceptance test (#13795) * fix mysql cdc acceptance test * remove comments * doc: add link to forum in plopfile (#13560) * Make connector adaptable based on deployment mode (#13522) * Add deployment mode to env shared with jobs * Add adaptive runners * Migrate postgres source to use adaptive runner * Add an array of specs in docker image spec definition * Add copyright * Parse docker image spec with specs list * Update spec yaml files * Pass in DEPLOYMENT_MODE to docker compose file * Revert "Parse docker image spec with specs list" This reverts commit 8fe41dd3b7fa5306ba8b62d660c002420adb9b23. * Revert changes in docker image spec * Read cloud specific spec files based on deployment mode * Revert "Update spec yaml files" This reverts commit 059f326432dba1f8da4ad4dfd8d5ac7a14de66c3. * Publish cloud spec file if necessary * Fix upload script * Move test files * Update docker compose file * Format code * Add comment about spec filename * Add unit tests * Remove redundant jdbc acceptance test When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog: - `wakeup_at` - `last_visited_at` - `last_comment_at` These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally. The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test. Given that the JDBC strict encrypt test is basically redundant, it is removed. * Update color palette, add scss colors, borders, and spacing (#13802) * Add updated color palette to SCSS and theme * Migrate current color palette to new colors * Add border and spacing variables * Remove legacy color mapping from scss side * Add style guide to front end repo (#13418) * Add .md from google doc draft * update per review * cleanup * Update STYLEGUIDE.md Co-authored-by: Tim Roes <[email protected]> * Edited Stripe, HubSpot, Intercom docs (#13810) * Initial edits * Updated the Stripe connector doc * Updated the Stripe doc * Updated HubSpot and Intercom docs * Minor sidebar edit * Remove the catalog from the sidebar since the file has been deleted in Update Airbyte Protocol Docs (#13709) (#13813) * Fix broken link (#13815) * fixing broken links * docusaurus updates * Add METRIC_CLIENT and OTEL_COLLECTOR_ENDPOINT to .env files to fix kube deployments (#13817) * octavia-cli: per workspace state (#13070) * Fix LabeledSwitch component width (#13798) * Fix LabeledSwitch component width * Switch to new color palette * Replace color * Fix color variable * Update 4-connection-checking.md (#13812) * Update 4-connection-checking.md As discussed in https://airbytehq-team.slack.com/archives/C02UQ9MJ4GG/p1655312684160609 - this example is misleading, as it does not actually connect to any API to validate connectivity. * Update 4-connection-checking.md Added a link to OneSignal check_connection, and move the "Note" to below the example * Bump Airbyte version from 0.39.18-alpha to 0.39.19-alpha (#13820) Co-authored-by: edgao <[email protected]> * Normalization: Upgrade MySQL to dbt 1.0.0 (#11470) * Restart workflow after activity failure instead of quarantining (#13779) * use SHORT_ACTIVITY_OPTIONS on check connection activity so that it has retries * retry workflow after delay instead of quarantining * allow activity env vars to be configured in docker and kube * add env var for workflow restart delay and refactor slightly * update tests to handle new restart behavior * update test name * add empty env var values to .env files * fail attempt before job in cleanJobState to prevent state machine failure * change default value of max activity attempt retries from 10 to 5 * Bump Airbyte version from 0.39.19-alpha to 0.39.20-alpha (#13829) Co-authored-by: lmossman <[email protected]> * 🎉 Postgres source: prepare for removing strict-encrypt variant (#13823) * Bump postgres version * Update doc * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * update readme (#13836) * add evan, jonathan, charles as protocol reviews in codeowners (#13776) * 📚 update MSSQL docs to reflect that we support normalization (#13826) * [low-code connectors] Cartesian product stream slicer (#13740) * list slicer * Add comment * product stream slicer * comment * rename * format * Update comment * split on 2 lines for readability * Revert "rename" This reverts commit e801f2d1f859306c6a2b0d66d265753ee9e32a54. * updated stacktrace format in java trace messages (#13847) * updated stacktrace format in java trace messages * test checks specifically on stacktrace in trace message * remove unused import * source-facebook-marketing: 🎉 Add fields to the ads_set stream (#13623) * 🎉 Source Github: use GraphQL for `pull_request_stats` stream (#13763) Signed-off-by: Sergey Chvalyuk <[email protected]> * Tweak schema validation code for readability. (#13721) * Refactor schema validation for better reading. * Better comments. * Format. * source-facebook-marketing: bump to 0.2.53 (#13853) * Source Hubspot - fix missing data issue (#13837) * #278 oncall: source hubspot - fix missing data issue * #278 oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source Amplitude: try-catch BadZipFile exception (#13846) * #272 oncall: source Amplitude - try-catch BadZipFile exception * #272-oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🐛 Server can display streams with the same name in different namespaces (#13855) * Updated connector catalog (#13862) * Fix the running job counter (#13833) * Add StateType and StateWrapper objects to the model (#13858) * Emit orphan running jobs metrics (#13865) That metrics show the number of jobs currently flagged as running that are associated to inactive or deprecated connections. * Add meaningful tests for OTEL client (#13831) * Add meaningful tests for OTEL client * PR comment fix * code clean ups * Persist state to reset connection job config (#13867) * add current connection state to JobResetConnectionConfig * pass state from reset connection config to job sync config in generate input activity * format * Source Google Analytics: Add threshold_days option, uncomment statement for incremental tests (#13871) * Harshith/test pr 12536 (#13874) * 🎉 New Source: Firebolt (#1) * fix: Boolean type cast * test: Improve format testing and doc * refactor: Move some db functionality * docs: Adding types doc link in utils * feat: Use future-proof Auth in SDK * fix: integration tests are failing * chore: update seed file Co-authored-by: Petro Tiurin <[email protected]> Co-authored-by: ptiurin <[email protected]> * :bug: Source Relational DB : remove unicode nulls from cursors (#13854) * Make sure that cursor value doesn't contain unicode null. * Test that cursors doesn't contain unicode null * incr version * enableReinitialize on account form (#13622) * #13888 source instagram: fix build (#13889) * Fix md and lg spacing scss vars (#13886) * Publishing java connectors with better formatted stacktrace in Trace messages (#13852) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Fix md and lg spacing scss vars (#13886) * Delete TEMP-testing-command.yml * auto-bump connector version * Delete publish-command.yml * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * Update _variables.scss Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * octavia-cli: fix workspace not having anonymous_data_collection property (#13869) * Update connection update calls to use central utility to ensure connection update has all data (#13564) * Update connection updates with build update utility * Add buildConnectionUpdate utility * Update components that update the connection to use utility when necessary * Use conection name when saving connection from replication view to prevent override from refreshed catalog * Improve connection check on ReplicationView onSubmit function * Display connection state in connection setting page (#13394) * Display Connection State in Setting page * memoize callback * rendering and confirmaton * setState API * Input validation * remove JSON step * rename apiMethod to `updateState` * test and adjust route * skip if sync is running * prevent state update when sync is running * code editor component * errors fixed * scss style * make linter happy * Back to monaco editor * Remove ability to edit state * Adjust FE code * Fix CSS problem * Update airbyte-webapp/src/locales/en.json Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * just use PRE to render state for now Co-authored-by: Tim Roes <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * update api for per stream (#13835) * Update airbyte-protocol.md (#13892) * Update airbyte-protocol.md * Fix typo * Fix prose * Add protocol reviewers for protocol documentation * Remove duplicate * Edited Amplitude, Mailchimp, and Zendesk Support docs (#13897) * deleting SUMMARY.md since we don't need it for docusaurus builds (#13901) * Do not hide unexpected errors in the check connection (#13903) * Do not hide unexpected errors in the check connection * Fix test * Common code to deserialize a state message in the new format (#13772) * Common code to deserialize a state message in the new format * PR comments and type changed to typed * Format * Add StateType and StateWrapper objects to the model * Use state wrapper instead of Either * Switch to optional * PR comments * Support array legacy state * format Co-authored-by: Jimmy Ma <[email protected]> * 🐛 Source Amazon Seller Partner: handle start date for financial stream (#13633) * start and end date for finacial stream should not be more than 180 days apart * improve unit tests * make changes to start date for finance stream * update tests * lint changes * update version to 0.2.22 for source-amazon-seller-partner * Normalization: Fix incorrect jinja2 macro `json_extract_array` call (#13894) Signed-off-by: Sergey Chvalyuk <[email protected]> * Docs: fixed the broken links (#13915) * 0.2.5 -> 0.2.6 (#13924) Signed-off-by: Sergey Chvalyuk <[email protected]> * 13546 Fix integration tests source-postgres Mac OS (#13872) * 13546 Fix integration tests source-postgres Mac OS * 13548 Fixed integration tests source-tidb Mac OS (#13927) * Source MsSql : incr ver to include changes #13854 (#13887) * incr version * put PR id * docker ver * connectors that published (#13932) * Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753) * Removed part_size from connectors that use StreamTransferManager * fixed S3DestinationConfigTest * fixed S3JsonlFormatConfigTest * upadate changelog and bump version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * upadate changelog and bump version for Redshift and Snowflake destinations * auto-bump connector version * fix GCS staging test * fix GCS staging test * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Reverted changes in SshBastionContainer (#13934) * 🎉 New Source Dockerhub (#13931) * init * implement working source + tests * add docs * add docs * fix bad comments * Update airbyte-integrations/connectors/source-dockerhub/acceptance-test-config.yml * Update airbyte-integrations/connectors/source-dockerhub/Dockerfile * Update airbyte-integrations/connectors/source-dockerhub/.dockerignore * Apply suggestions from code review * Update docs/integrations/sources/dockerhub.md * Update airbyte-integrations/connectors/source-dockerhub/integration_tests/acceptance.py Co-authored-by: George Claireaux <[email protected]> * address @Phlair's feedback * address @Phlair's feedback * each record is now a Docker image rather than response page * format * fix unit tests * fix acceptance tests * add icon, definition and generate seed spec * add requests to requirements Co-authored-by: sw-yx <[email protected]> * commented out non-relevant tests (#13940) * Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha (#13938) Co-authored-by: alafanechere <[email protected]> * newaction (#13942) * remove test action (#13944) * 🎉Source-mysql: aligned datatype test (#13945) * [13607] source-mysql: aligned datatype tests for regular and CDC ways + added CHAR fix to CDC processing * #13958 Source Stripe: fix configured catalogs (#13959) * 🐛 Source: Typeform - Update schema for Responses stream (#13935) * Upd responses schema * Upd docs * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :window: Updated email invitation flow that enables invited users to set name and create password (#12788) * First pass accepting email link invitation * Update Auth service with signInWithEmailLink calls * Add AcceptEmailInvite component * Update FirebaseActionRoute to handle sign in mode * Rename ResetPasswordAction to FirebseActionRoute * Add create password setp to AcceptEmailInvite component * Remove continueURL from invite fetch * Update accept email invite for user to enter both email and password together * Set name during email link signup * Update AcceptEmailInvite to send name * Add updateName to UserService * Update AuthService to set name during sign up * Remove steps from AcceptEmailInvite component Remove setPassword from AuthService * Add header and title to accept invite page * Move invite error messages to en file * For invite link pages, show login link instead of sign up * Disable name update on sign in via email lnk * Resend email invite when the invite link is expired * Fix status message in accept email invite page * Re-enable set user's name during sign up email invite * Update signUpWithEmailLink so that sign up is successful even if we fail to update the user's name * Update comments on GoogleAuthService signInWithEmailLink * Add newsletter and accept terms checkboxes to accept email invite component * Extract signup form from signup page * Extract fields from signup form * Update accept email invite component to use field components from signup form * Ensure that sign up button is disable until form is valid and security checkbox is checked * Make error status text color in accept email link red * Update workspace check in DefaultView so that user lands in workspace selector when there are no workspaces * Add coment around continueUrl param usage in UserService * Remove usless default case in GoogleAuthService * Source Marketo: process fail during creation of an export job (#13930) * #9322 source Marketo: process fail during creation of an export job * #9322 source marketo: upd changelog * #9322 source marketo: fix unit test * #9322 source marketo: fix SATs * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :window: :wrench: Add eslint rules for CSS modules (#13952) * add eslint-plugin-css-modules rules * Fixes: - turn on eslint css modules rule as error - remove unused styles * add warning message if styled components is used * Revert "add warning message if styled components is used" This reverts commit 4e92b8b2110142bb679f15aeb034e377e0dcc69c. * replace rule severity with words * Update salesforce.md Fixed broken link * :window: 🔧 Add auto-fixable linting rules to webapp (#13462) * Add new eslint rules that fit with our code style and downgrade rules to warn * allowExpressions in fragment eslint rule * Enable function-component-definition in eslint and fix styles * Cleanup lint file * Fix react/function-component-definition warnings manually * Add more auto-fixable rules and fix * Fix functions that require usless returns * Update array-type rule to array-simple * Fix eslint errors manually disable assignmentExpression for arrays in prefer-destructuring rule * Auto fix new linting issues after rebase * Enhance /publish to allow for multiple connectors and parallel execution (#13864) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Delete TEMP-testing-command.yml * auto-bump connector version * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * destinations * + snowflake * saved * auto-bump connector version * auto-bump connector version * java source bumps * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * remove twice-defined methods * label things * revert action * using the new test action * point at action * wrong tag on action * update pool label * update to use new ec2-github-runner fork * this needs to be more generic than publisher * change publish to run on pool * add comment about runner-pool usage * updated publish command docs for multi & parallel connector runs * auto-bump connector version * auto-bump connector version * auto-bump connector version * unbump failed publish versions * missed dockerfiles * remove failed docs * mssql fix * overhauled the git comment output * bumping a test connector that should work * slight order switcheroo * output connectors properly in first message * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump Airbyte version from 0.39.21-alpha to 0.39.22-alpha (#13979) Co-authored-by: Phlair <[email protected]> * Parker/temporal cloud (#13243) * switch to temporal cloud client for now * format * use client cert/key env secret instead of path to secret * add TODO comments * format * add logging to debug timeout issue * add more logging * change workflow task timeout * PR feedback: consolidate as much as possible, add missing javadoc * fix acceptance test, needs to specify localhost * add internal-use only comments * format * refactor to clean up TemporalClient and prepare it for future dependency injection framework * remove extraneous log statements * PR feedback * fix test * return isInitialized true in test * 📄 Postgres source: fix CDC setup order in docs (#13949) * postgres source: fix CDC setup order docs * Update docs/integrations/sources/postgres.md Co-authored-by: Liren Tu <[email protected]> * Per-stream state support for Postgres source (#13609) * WIP Per-stream state support for Postgres source * Fix failing test * Improve code coverage * Make global the default state manager * Add legacy adapter state manager * Formatting * Include legacy state for backwards compatibility * Add global state manager * Implement Global/CDC state handling * Fix test issues * Fix issue with updated method signature * Handle empty state case in global state manager * Adjust to protocol changes * Fix failing acceptance tests * Fix failing test * Fix unmodifiable list issue * Fix unmodifiable exception * PR feedback * Abstract global state manager selection * Handle conversion between different state types * Handle invalid conversion * Rename parameter * Refactor state manager creation * Fix failing tests * Fix failing integration tests * Add CDC test * Fix failing integration test * Revert change * Fix failing integration test * Use per-stream for postgres tests * Formatting * Correct stream descriptor validation * Correct permalink * PR feedback * Bump Airbyte version from 0.39.22-alpha to 0.39.23-alpha (#13984) Co-authored-by: pmossman <[email protected]> * Adds test for new workflow (#13986) * Adds test for new workflow * Adds airbyte repo * remove testing line * Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472) * write out new request options provider and refactor components and parts of the YAML config * fix formatting * pr feedback to consolidate body_data_provider to simplify the code * pr feedback get rid of extraneous optional * publish oss for cloud (#13978) workflow to publish oss artifacts that cloud needs to build against use docker buildx to create arm images for local development * skip debezium engine startup in case no table is in INCREMENTAL mode (#13870) * 🎉 Source Github: break point added for workflows_runs stream (#13926) Signed-off-by: Sergey Chvalyuk <[email protected]> * 6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866) * 6339: debug info * 6339: not using 'USE' on Azure SQL servers * 6339: cleanup * 6339: cleanup2 * 6339: cleanup3 * 6339: versions/changelogs updated * 6339: merge from master (consolidation issue) * 6339: dev connector version (for testing in airbyte cloud) * 6339: code review implementation * 6339: apply formatting * in case runners fail to spin up, this needs to run on github-hosted (#13996) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * :tada: New Source: Webflow (#13617) * Added webflow code * Updated readme * Updated README * Added webflow to source_definitions.yaml * Enhanced documentation for the Webflow source connector * Improved webflow source connector instructions * Moved Site ID to before API token in Spec.yaml (for presentation in the UI) * Addressed comments in PR. * Changes to address requests in PR review * Removed version from config * Minor udpate to spec.yaml for clarity * Updated to pass the accept-version as a constant rather than parameter * Updated check_connection to hit the collections API that requires both site id and the authentication token. * Fixed the test_check_connection to use the new check_connection function * Added a streams test for generate_streams * Re-named "autentication" object to "auth" to be more consistent with the way it is created by the CDK * Added in an explict line to instantiante an "auth" object from WebflowTokenAuthenticator, to make it easier to describe in the blog * Fixed a typo in a comment * Renamed some classes to be more intuitive * Renamed class to be more intuitive * Minor change to an internal method name * Made _get_collection_name_to_id_dict staticmethod * Fixed a unit-test error that only appeared when running " python -m pytest -s unit_tests". This was caused by Mocked settings from test_source.py leaking into test_streams.py * format: add double quotes and remove unused import * readme: remove semantic version naming of connector in build commands * Updated spec.yaml * auto-bump connector version * format files * add changelog * update dockerfile * auto-bump connector version Co-authored-by: sajarin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: marcosmarxm <[email protected]> * Source-oracle: fixed tests + checkstyle (#13997) * Source-oracle: fixed tests + checkstyle * 🐛Destination-mysql: fixed integration test and build process (#13302) * [13180] destination-mysql: fixed integration test * update changelog to include debezium version upgrade (#13844) * make table headers look less like successes (#13999) * source-twilio: implement lookback windows (#13896) * Revert "12708: Add an option to use encryption with staging in Redshift Destination (#13675)" (#14010) This reverts commit aa28d448d820df9d79c2c0d06b38978d1108fb2c. * Revert "6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866)" (#14011) This reverts commit 0d870bd37bc3b5cd798b92115d73bcc45a42d8f7. * [low-code connectors] BasicHttpAuthenticator (#13733) * implement basichttpauthenticator * add optional refresh access token authenticator * remove prints * type hints * Fix and unit test * missing test * Add class to __init__ file * Add comment * migrate JsonSchemas to use basic path instead of JSONPath (#13917) * scaffold for catalog diff, needs fixing on type handling and tests (#13786) * Prepare release of JDBC connectors (#13987) * Prepare release of JDBC connectors * Update source definitions manually * use built in check for if path is definite (#13834) * 13535 Fixed bastion network for integration tests (#14007) * doc: add error troubleshooting `docker-compose up` (#13765) * fix: duplicate resource allocations in `airbyte-temporal` deployment (#13816) * helm-chart: Fix worker deployment format error (#13839) * add catalog diff connection read (#13918) * doc: fix small typo on Shopify documentation (#13992) * add streams to reset to job info (#13919) * Generate api for changes in #13370 and make code compatible (#14014) * Generate api for per-stream updates #13835 (#14021) * Revert "Prepare release of JDBC connectors (#13987)" (#14029) This reverts commit df759b30778082508e2872513800fac34d98ff7c. * Fix per stream state protocol backward compatibility (#14032) * rename state type field to fix backwards compatibility issue * replace usages of stateType with type * support semi incremental by adding extractor record filter (#13520) * support semi incremental by adding extractor record filter * refactor extractor into a record_selector that supports extraction and filtering of response records * Remove pydantic spec from amazon ads and use YAML spec (#13988) * add EdDSA support in SSH tunnel (#9494) * add EdDSA support * verify EdDSA support works correct Co-authored-by: Yurii Bidiuk <[email protected]> * 🎉New source connector: source-metabase (#13752) * Add docs * Close metabase session when sync finishes * Close session in check_connection * Add source definition to seed * Add icon * improve cdc check for connectors (#14005) * improve should use cdc check * Revert "improve should use cdc check" This reverts commit 7d01727279d21d33a6c18ed3227ee94432636120. * improve should use cdc check * add unit test * Update webflow.md * Update webflow.md * Update webflow.md * Remove legacy sentry code from cdk (#14016) * rip sentry out of cdk * remove sentry dsn from gsc * Update webflow.md * Update webflow.md * Fixed broken links (#14071) * 🪟Persist unsaved changes on schema refresh (#13895) * add form values tracker context * add clarifying comment * add same functionality to create connection * Update airbyte-webapp/src/components/CreateConnectionContent/CreateConnectionContent.tsx Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * Fixes broken links so we can deploy again (#14075) also adds better error message for when this happens to others * Adds symmary.md to gitignore (#14078) * Added webflow icon (#14069) * Added webflow icon * Added icon * Build create connection form build failure (#14081) * Fix CDK obfuscation of nested secrets (#14035) * Added Buy Credits section to Managing Airbyte Cloud (#13905) * Added Buy Credits section to Managing Airbyte Cloud * Made some style changes * Made edits based on Natalie's suggestions * Deleted link * Deleted line * Edited email address * Updated reaching out to sales sentence * disable es-lit to fix build (#14087) * Release source connectors (#14077) * Release source connectors * Fix issue with database connection in test * Fix failing tests due to authentication * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump Airbyte version from 0.39.23-alpha to 0.39.24-alpha (#14094) Co-authored-by: jdpgrailsdev <[email protected]> * Emit the state to remove in the airbyte empty source (#13725) What This PR updates the EmptyAirbyteSource in order to perform a partial update and handle the new state message format. How The empty will now emit different messages based on the type of state being provided: Per stream: it will emit one message per stream that have been reset Global: It will emit one global message that will contain null for the stream that have been reset including the shared state Co-authored-by: Jimmy Ma <[email protected]> * Add StatePersistence object (#13900) Add a StatePersistence object that supports Read/Writes of States to the DB with StreamDescriptor fields The only migrations that is supported are * moving from LEGACY to GLOBAL * moving from LEGACY to STREAM * All other state type migrations are expected to go through an explicit reset beforehand. * secret-persistence: Hashicorp Vault Secret Store (#13616) Co-authored-by: Amanda Murphy <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> * 🐛 Source Hubspot: remove `AirbyteSentry` dependency (#14102) * fixed * updated changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * fix: format VaultSecretPersistenceTest.java (#14110) * Source Hubspot: extend error logging (#14054) * #291 incall - source Hubspot: extend error logging * huspot: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Update webflow.md (#14083) * Update webflow.md Removed a description that is only applicable to people that are writing connector code, not to _users_ of the connector. * Update webflow.md * Update webflow.md * Update webflow.md * Update webflow.md * 12708: Add an option to use encryption with staging in Redshift Desti… (#14013) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * 12708: specs updated * 12708: specs updated * 12708: removing autogenerated files from PR * 12708: changelog updated * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source PayPal Transaction: Update Transaction Schema (#13682) * Update transaction schema. * Transform money values from strings to floats or integers. Co-authored-by: nataly <[email protected]> Co-authored-by: Augustin <[email protected]> * fix(jsonSchemas): raise error when items property not provided (#14018) * fix stream name in stream transformation update (#14044) * 🐛 Destination Redshift: Improved discovery for redshift-destination not SUPER streams (#13690) airbyte-12843: Improved discovery for redshift-destination not SUPER tables, excluded views from discovery. * Remove skiptests option (#14100) * update sentry release script (#14123) * Remove "additionalProperties": false from specs for connectors with staging (#14114) * Remove "additionalProperties": false from spec for connectors with staging * Remove "additionalProperties": false from spec for Redshift destination * bump versions * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * [14003] source-oracle: added custom jdbc field (#14092) * [14003] source-oracle: added custom jdbc field * Add JobErrorReporter for sending sync job connector failures to Sentry (#13899) * skeleton for reporting connector errors to sentry * report on job failures instead of attempt failures * report sync job failures with relevant metadata using JobErrorReporter * send stack traces from python connectors to sentry * test JobCreationAndStatusUpdate and JobErrorReporter * logs * refactor into helper, initial tests * using sentry * run format * load reporting client from env * load sentry dsn from env * send java stack traces to sentry * test sentryclient, refactor to use Hub instance * ErrorReportingClient.report -> .reportJobFailureReason * inject exception helper, test stack trace parse error tagging * rm logs * more stack trace tests * remove logs * fix failing tests * rename ErrorReportingClient to JobErrorReportingClient * rename vars in docker-compose * Return an Optional instead of null when parsing stack traces * dont remove airbyte prefix when setting release name * from_trace_message static * remove failureSummary from jobfailure input, get from Job * send stacktrace string if we weren't able to parse * set deployment mode tag * update .env * just log if something goes wrong * Use StateMessageHelper in source (#14125) * Use StateMessageHelper in source * PR feedback and formatting * More PR feedback * Revert change * Revert changes * Bump Airbyte version from 0.39.24-alpha to 0.39.25-alpha (#14124) Co-authored-by: brianjlai <[email protected]> * Refactor acceptance tests and utils (#13950) * Refactor Basic acceptance tests and utils * Refactor Advanced acceptance tests and utils * Remove unused code * Clear destination db data during cleanup * Cleanup comments * cleanup init code * test creating new desintation db for each test * cleanup desintation db init * Allow to edit api client * pull in temporal cloud changes * Rename helper to harness; set some funcs to private; turn init into constructor * add func to set env vars instead of using static vars and move some functionality out of init into acceptance tests * update javadoc Co-authored-by: Davin Chia <[email protected]> * fix javadoc formatting * fix var naming Co-authored-by: Davin Chia <[email protected]> * Bump Airbyte version from 0.39.25-alpha to 0.39.26-alpha (#14141) Co-authored-by: terencecho <[email protected]> * 🎉 octavia-cli: Add ability to get existing resources (#13254) * 13541 Fixed integration tests source-db2 Mac OS (#14133) * 13523 Fix integration tests destination-cassandra Mac OS (#14134) * 🐛 Source Hubspot: fixed SAT test, commented out expected_records (#14140) * :bug: Source Intercom: extend `Contacts` schema with new properties (#14099) * Source Twilio: adopt best practices (#14000) * #1946 Source twilio: aopt best practices - tune tests * #1946 add expected_records to acceptance-test-config.yml * #1946 source twilio - upd schema and changelog * #1946 fix expected_records * #1946 source twilio: rm alerts from expected records as they expire in 30 days * #1946 source twilio: bump version * 🎉 Source BingAds: expose hourly/daily/weekly/monthly options from configuration (#13801) * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: config settings for hourly/daily/weekly/monthly reports added: default value for all periodic reports to True * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused class variables, if-statement * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused variables from config * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * remove VersionMismatchServer (#14076) * remove VersionMismatchServer * remove VersionMismatchServerTest * revert intended changes * Increase instance termination time limit to 3 hours to accommodate connector builds. (#14181) * Use correct bash comment symbol. (#14183) * 🎉 New Source: Orbit.love (#13390) * source-orbit: add definition and specs (#14189) * 🎉 Base Norrmalization: clean-up Redshift `tmp_schemas` after SAT (#14015) Now after `base-normalization` SAT the Destination Redshift will be automatically cleaned up from test leftovers. Other destinations are not covered yet. * Source Salesforce: fix customIntegrationTest for SAT (#14172) * Source Amazon Ads: increase timeout for SAT (#14167) * 🎉 Introduce Google Analytics Data API source (#12701) * Introduce Google Analytics Data API source https://developers.google.com/analytics/devguides/reporting/data/v1 * Add Google Analytics Data API source PR link * Add `client` class for Google Analytics Data API * Move dimensions and metrics extraction to the `client` class In the Google Analytics Data API * Change the copyright date to 2022 in Google Analytics Data API * fix: removing incremental syncs * fix: change project_id to string * fix: flake check is failing * chore: added it to source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * 🐛 Destination Redshift: use s3 bucket path for s3 staging operations (#13916) * Publish acceptance test utils maven artifact (#14142) * Fix StatePersistence Legacy read/write (#14129) StatePersistence will wrap/unwrap legacy state on write/read to ensure compatibility with the old behavior/data. * 🎉 Destination connectors: Improved "SecondSync" checks in Standard Destination Acceptance tests (#14184) * [11731] Improved "SecondSync" checks in Standard Destination Acceptance tests * 🐛 Source Zendesk Support: fixed "Retry-After" non integer value (#14112) Signed-off-by: Sergey Chvalyuk <[email protected]> * Source Tiktok Marketing: Videometrics (#13650) * added video metrics in streams.py * common metrics list updated. * updated streams.py with extended metrics required. * updated stream_test * updated configured_catalog * video metrics required list updated. * chore: formatting * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * 🎉 Source Github: secondary rate limits has to retry (#13955) Signed-off-by: Sergey Chvalyuk <[email protected]> * Harshith/test pr 13118 (#14192) * Firebolt destination * feat: Write method dropdown * feat: Use future-proof Auth in SDK * refactor: Move writer instantiation * fix: tests are failing * fix: tests are failing * fix: tests are failing * chore: added connector to definitions * fix: formatting and spec * fix: formatting for orbit Co-authored-by: ptiurin <[email protected]> * 🪟 :art: Show credit usage on chart's specific day (#13503) * add tooltip to chart * Fixes: - update main chart color; - change onHover background color * change chart color pallet to grey 500 * update color reference * remove opacity from UsageCell * 🐛 destination-redshift: use s3 bucket path for s3 cleanup (#14190) * Improve documentation for Postgres Source (#13830) * Improve documentation for Postgres Source * add information about additional JDBC params * add anchors for doc sections * fix link to CDC on Bare Metal * add more details about parsing date/time values * add doc link to SSH fields * Handle null reset source config (#14202) * handle null reset source config * format * Wait indefinitely if connection is not active (#14200) * also wait indefinitely if connection is deleted * fix test * Bump Airbyte version from 0.39.26-alpha to 0.39.27-alpha (#14204) Co-authored-by: lmossman <[email protected]> * Bmoric/feature flag for state deserialization (#14127) * Add Feature flag * Add default feature flag value * Update test * remove unsused * tmp * Update tests * rm unwanted change * PR comments * [low-code connectors] default types and default values (#14004) * default types and default values * cleanup * fixes so read works * remove prints and trycatch * comment * remove unused param * split file * extract method * extract methods * comment * optional * fix test * cleanup * delete interpolated request header provider * simplify next page url paginator interface * comment * format * add state type endpoint (#14111) * Bump Airbyte version from 0.39.27-alpha to 0.39.28-alpha (#14210) Co-authored-by: sherifnada <[email protected]> * 🐛 source-orbit: remove workspace_old.json (#14208) * Fix: Docs plural login redirecting to wrong URL (#14207) * [docs] fix numbering and incorrect filename in CDK docs (#13045) * [docs] fix numbering in CDK docs * Update 5-declare-schema.md * Update 5-declare-schema.md * Update 6-read-data.md * Update 8-test-your-connector.md * Remove the old scheduler from HelmCharts helper (#14187) * Remove the old scheduler from HelmCharts helper The old scheduler was removed as part of https://github.com/airbytehq/airbyte/pull/13400 * Remove legacy `scheduler` comment in HelmCharts * Source Gitlab: add GroupIssueBoards stream (#13252) * GitLab Source: add GroupIssueBoards stream * Address stream schema comments * Address comments * Bump version * Add as empty stream * run seed file source (#14215) * fix 'cannot reach server' error on demo instance (#10020) * Update CODEOWNERS (#14209) * 🎉 Source Github: use GraphQL for `reviews` stream (#13989) Signed-off-by: Sergey Chvalyuk <[email protected]> * workflow for publishing artifacts for cloud (#14199) * fix sentry org slug change (#14218) * Source File: correct spec json to match json format (#13738) * Upgrade spotless version and remove jvmargs workaround (#13705) * Source Zendesk Chat: Process large amount of data in batches for incremental (#14214) * increased the limit of itens in request * Configuration for max api pages on requests * included api_pagination_limit in sample * included api_pagination_limit in invalid_config * creating new table for chat_session * reverted api_pagination_limit approach * removed api_pagination_limit from TimeIncrementalStream * correct chat json * bump connector version * add changelog * run format * auto-bump connector version Co-authored-by: Roberto Bonnet <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Remove all @ts-ignore (#14221) * Bump hadoop to use version 3.3.3 (#14182) * Change the persistence activity to use the new persistence layer (#14205) * Change the persistence activity to use the new persistence layer * Use lombok * format * Use new State message helper * Fix build (#14225) * Fix build * Fix test * Use new state persistence for state reads (#14126) * Inject StatePersistence into DefaultJobCreator * Read the state from StatePersistence instead of ConfigRepository * Add a conversion helper to convert StateWrapper to State * Remove unused ConfigRepository.getConnectionState * Temporal per stream resets (#13990) * remove reset flags from workflow state + refactor * bring back cancelledForReset, since we need to distinguish between that case and a normal cancel * delete reset job streams on cancel or success * extract isResetJob to method * merge with master * set sync modes on streams in reset job correctly * format * Add test for getAllStreamsForConnection * fix tests * update more tests * add StreamResetActivityTests * fix tests for default job creator * remove outdated comment * remove debug lines * remove unused enum value * fix tests * fix constant equals ordering * make job mock not static * DRY and add comments * add comment about deleted streams * Remove io.airbyte.config.StreamDescriptor * regisster stream reset activity impl * refetch connection workflow when checking job id, since it may have been restarted * only cancel if workflow is running, to allow reset signal to always succeed even if batched with a workflow start * fix reset signal to use new doneWaiting workflow state prop * try to fix tests * fix reset cancel case * add acceptance test for resetting while sync is running * format * fix new acceptance test * lower sleep on test * raise sleep * increase sleep and timeout, and remove repeated test * use CatalogHelpers to extract stream descriptors * raise sleep and timeout to prevent transient failures * format Co-authored-by: alovew <[email protected]> * Log Marketo response errors * Additional message if Marketo response has failed with empty errors Co-authored-by: Topher Lubaway <[email protected]> Co-authored-by: Malik Diarra <[email protected]> Co-authored-by: Teal Larson <[email protected]> Co-authored-by: Tim Roes <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: VitaliiMaltsev <[email protected]> Co-authored-by: grishick <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Greg Solovyev <[email protected]> Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: Andy <[email protected]> Co-authored-by: LiRen Tu <[email protected]> Co-authored-by: oneshcheret <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Xiaohan Song <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Charles <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Sophia Wiley <[email protected]> Co-authored-by: Edward Gao <[email protected]> Co-authored-by: Denis Davydov <[email protected]> Co-authored-by: Subodh Kant Chaturvedi <[email protected]> Co-authored-by: Bastien Gandouet <[email protected]> Co-authored-by: Harshith Mullapudi <[email protected]> Co-authored-by: Paweł Kociński <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Baz <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: Amruta Ranade <[email protected]> Co-authored-by: Alexander Marquardt <[email protected]> Co-authored-by: edgao <[email protected]> Co-authored-by: lmossman <[email protected]> Co-authored-by: George Claireaux <[email protected]> Co-authored-by: mutt-santiago-stachuk <[email protected]> Co-authored-by: Davin Chia <[email protected]> Co-authored-by: Arsen Losenko <[email protected]> Co-authored-by: Petro Tiurin <[email protected]> Co-authored-by: ptiurin <[email protected]> Co-authored-by: Andrii Leonets <[email protected]> Co-authored-by: Jonathan Pearlin <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Ganpat Agarwal <[email protected]> Co-authored-by: Rajakavitha Kodhandapani <[email protected]> Co-authored-by: Yevhen Sukhomud <[email protected]> Co-authored-by: sw-yx <[email protected]> Co-authored-by: alafanechere <[email protected]> Co-authored-by: Eugene <[email protected]> Co-authored-by: Anna Lvova <[email protected]> Co-authored-by: Vladimir <[email protected]> Co-authored-by: Phlair <[email protected]> Co-authored-by: Parker Mossman <[email protected]> Co-authored-by: Adam <[email protected]> Co-authored-by: Liren Tu <[email protected]> Co-authored-by: pmossman <[email protected]> Co-authored-by: Brian Lai <[email protected]> Co-authored-by: Peter Hu <[email protected]> Co-authored-by: Tuhai Maksym <[email protected]> Co-authored-by: sajarin <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: steve withington <[email protected]> Co-authored-by: Leo Sussan <[email protected]> Co-authored-by: cenegd <[email protected]> Co-authored-by: Tomas Perez Alvarez <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Yurii Bidiuk <[email protected]> Co-authored-by: Christophe Duong <[email protected]> Co-authored-by: jdpgrailsdev <[email protected]> Co-authored-by: Stella Chung <[email protected]> Co-authored-by: Amanda Murphy <[email protected]> Co-authored-by: Mohamed Magdy <[email protected]> Co-authored-by: nataly <[email protected]> Co-authored-by: Tyler Russell <[email protected]> Co-authored-by: Alexander Tsukanov <[email protected]> Co-authored-by: Pedro S. Lopez <[email protected]> Co-authored-by: brianjlai <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Daniel Diamond <[email protected]> Co-authored-by: drrest <[email protected]> Co-authored-by: Abhi Vaidyanatha <[email protected]> Co-authored-by: Zawar Khan <[email protected]> Co-authored-by: sherifnada <[email protected]> Co-authored-by: Sachin Jangid <[email protected]> Co-authored-by: Chris Wu <[email protected]> Co-authored-by: Jared Rhizor <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Roberto Bonnet <[email protected]> Co-authored-by: alovew <[email protected]>

* octavia-cli: fix workspace not having anonymous_data_collection property (#13869) * Update connection update calls to use central utility to ensure connection update has all data (#13564) * Update connection updates with build update utility * Add buildConnectionUpdate utility * Update components that update the connection to use utility when necessary * Use conection name when saving connection from replication view to prevent override from refreshed catalog * Improve connection check on ReplicationView onSubmit function * Display connection state in connection setting page (#13394) * Display Connection State in Setting page * memoize callback * rendering and confirmaton * setState API * Input validation * remove JSON step * rename apiMethod to `updateState` * test and adjust route * skip if sync is running * prevent state update when sync is running * code editor component * errors fixed * scss style * make linter happy * Back to monaco editor * Remove ability to edit state * Adjust FE code * Fix CSS problem * Update airbyte-webapp/src/locales/en.json Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * just use PRE to render state for now Co-authored-by: Tim Roes <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * update api for per stream (#13835) * Update airbyte-protocol.md (#13892) * Update airbyte-protocol.md * Fix typo * Fix prose * Add protocol reviewers for protocol documentation * Remove duplicate * Edited Amplitude, Mailchimp, and Zendesk Support docs (#13897) * deleting SUMMARY.md since we don't need it for docusaurus builds (#13901) * Do not hide unexpected errors in the check connection (#13903) * Do not hide unexpected errors in the check connection * Fix test * Common code to deserialize a state message in the new format (#13772) * Common code to deserialize a state message in the new format * PR comments and type changed to typed * Format * Add StateType and StateWrapper objects to the model * Use state wrapper instead of Either * Switch to optional * PR comments * Support array legacy state * format Co-authored-by: Jimmy Ma <[email protected]> * 🐛 Source Amazon Seller Partner: handle start date for financial stream (#13633) * start and end date for finacial stream should not be more than 180 days apart * improve unit tests * make changes to start date for finance stream * update tests * lint changes * update version to 0.2.22 for source-amazon-seller-partner * Normalization: Fix incorrect jinja2 macro `json_extract_array` call (#13894) Signed-off-by: Sergey Chvalyuk <[email protected]> * Docs: fixed the broken links (#13915) * 0.2.5 -> 0.2.6 (#13924) Signed-off-by: Sergey Chvalyuk <[email protected]> * 13546 Fix integration tests source-postgres Mac OS (#13872) * 13546 Fix integration tests source-postgres Mac OS * 13548 Fixed integration tests source-tidb Mac OS (#13927) * Source MsSql : incr ver to include changes #13854 (#13887) * incr version * put PR id * docker ver * connectors that published (#13932) * Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753) * Removed part_size from connectors that use StreamTransferManager * fixed S3DestinationConfigTest * fixed S3JsonlFormatConfigTest * upadate changelog and bump version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * upadate changelog and bump version for Redshift and Snowflake destinations * auto-bump connector version * fix GCS staging test * fix GCS staging test * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Reverted changes in SshBastionContainer (#13934) * 🎉 New Source Dockerhub (#13931) * init * implement working source + tests * add docs * add docs * fix bad comments * Update airbyte-integrations/connectors/source-dockerhub/acceptance-test-config.yml * Update airbyte-integrations/connectors/source-dockerhub/Dockerfile * Update airbyte-integrations/connectors/source-dockerhub/.dockerignore * Apply suggestions from code review * Update docs/integrations/sources/dockerhub.md * Update airbyte-integrations/connectors/source-dockerhub/integration_tests/acceptance.py Co-authored-by: George Claireaux <[email protected]> * address @Phlair's feedback * address @Phlair's feedback * each record is now a Docker image rather than response page * format * fix unit tests * fix acceptance tests * add icon, definition and generate seed spec * add requests to requirements Co-authored-by: sw-yx <[email protected]> * commented out non-relevant tests (#13940) * Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha (#13938) Co-authored-by: alafanechere <[email protected]> * newaction (#13942) * remove test action (#13944) * 🎉Source-mysql: aligned datatype test (#13945) * [13607] source-mysql: aligned datatype tests for regular and CDC ways + added CHAR fix to CDC processing * #13958 Source Stripe: fix configured catalogs (#13959) * 🐛 Source: Typeform - Update schema for Responses stream (#13935) * Upd responses schema * Upd docs * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :window: Updated email invitation flow that enables invited users to set name and create password (#12788) * First pass accepting email link invitation * Update Auth service with signInWithEmailLink calls * Add AcceptEmailInvite component * Update FirebaseActionRoute to handle sign in mode * Rename ResetPasswordAction to FirebseActionRoute * Add create password setp to AcceptEmailInvite component * Remove continueURL from invite fetch * Update accept email invite for user to enter both email and password together * Set name during email link signup * Update AcceptEmailInvite to send name * Add updateName to UserService * Update AuthService to set name during sign up * Remove steps from AcceptEmailInvite component Remove setPassword from AuthService * Add header and title to accept invite page * Move invite error messages to en file * For invite link pages, show login link instead of sign up * Disable name update on sign in via email lnk * Resend email invite when the invite link is expired * Fix status message in accept email invite page * Re-enable set user's name during sign up email invite * Update signUpWithEmailLink so that sign up is successful even if we fail to update the user's name * Update comments on GoogleAuthService signInWithEmailLink * Add newsletter and accept terms checkboxes to accept email invite component * Extract signup form from signup page * Extract fields from signup form * Update accept email invite component to use field components from signup form * Ensure that sign up button is disable until form is valid and security checkbox is checked * Make error status text color in accept email link red * Update workspace check in DefaultView so that user lands in workspace selector when there are no workspaces * Add coment around continueUrl param usage in UserService * Remove usless default case in GoogleAuthService * Source Marketo: process fail during creation of an export job (#13930) * #9322 source Marketo: process fail during creation of an export job * #9322 source marketo: upd changelog * #9322 source marketo: fix unit test * #9322 source marketo: fix SATs * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :window: :wrench: Add eslint rules for CSS modules (#13952) * add eslint-plugin-css-modules rules * Fixes: - turn on eslint css modules rule as error - remove unused styles * add warning message if styled components is used * Revert "add warning message if styled components is used" This reverts commit 4e92b8b2110142bb679f15aeb034e377e0dcc69c. * replace rule severity with words * Update salesforce.md Fixed broken link * :window: 🔧 Add auto-fixable linting rules to webapp (#13462) * Add new eslint rules that fit with our code style and downgrade rules to warn * allowExpressions in fragment eslint rule * Enable function-component-definition in eslint and fix styles * Cleanup lint file * Fix react/function-component-definition warnings manually * Add more auto-fixable rules and fix * Fix functions that require usless returns * Update array-type rule to array-simple * Fix eslint errors manually disable assignmentExpression for arrays in prefer-destructuring rule * Auto fix new linting issues after rebase * Enhance /publish to allow for multiple connectors and parallel execution (#13864) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Delete TEMP-testing-command.yml * auto-bump connector version * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * destinations * + snowflake * saved * auto-bump connector version * auto-bump connector version * java source bumps * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * remove twice-defined methods * label things * revert action * using the new test action * point at action * wrong tag on action * update pool label * update to use new ec2-github-runner fork * this needs to be more generic than publisher * change publish to run on pool * add comment about runner-pool usage * updated publish command docs for multi & parallel connector runs * auto-bump connector version * auto-bump connector version * auto-bump connector version * unbump failed publish versions * missed dockerfiles * remove failed docs * mssql fix * overhauled the git comment output * bumping a test connector that should work * slight order switcheroo * output connectors properly in first message * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump Airbyte version from 0.39.21-alpha to 0.39.22-alpha (#13979) Co-authored-by: Phlair <[email protected]> * Parker/temporal cloud (#13243) * switch to temporal cloud client for now * format * use client cert/key env secret instead of path to secret * add TODO comments * format * add logging to debug timeout issue * add more logging * change workflow task timeout * PR feedback: consolidate as much as possible, add missing javadoc * fix acceptance test, needs to specify localhost * add internal-use only comments * format * refactor to clean up TemporalClient and prepare it for future dependency injection framework * remove extraneous log statements * PR feedback * fix test * return isInitialized true in test * 📄 Postgres source: fix CDC setup order in docs (#13949) * postgres source: fix CDC setup order docs * Update docs/integrations/sources/postgres.md Co-authored-by: Liren Tu <[email protected]> * Per-stream state support for Postgres source (#13609) * WIP Per-stream state support for Postgres source * Fix failing test * Improve code coverage * Make global the default state manager * Add legacy adapter state manager * Formatting * Include legacy state for backwards compatibility * Add global state manager * Implement Global/CDC state handling * Fix test issues * Fix issue with updated method signature * Handle empty state case in global state manager * Adjust to protocol changes * Fix failing acceptance tests * Fix failing test * Fix unmodifiable list issue * Fix unmodifiable exception * PR feedback * Abstract global state manager selection * Handle conversion between different state types * Handle invalid conversion * Rename parameter * Refactor state manager creation * Fix failing tests * Fix failing integration tests * Add CDC test * Fix failing integration test * Revert change * Fix failing integration test * Use per-stream for postgres tests * Formatting * Correct stream descriptor validation * Correct permalink * PR feedback * Bump Airbyte version from 0.39.22-alpha to 0.39.23-alpha (#13984) Co-authored-by: pmossman <[email protected]> * Adds test for new workflow (#13986) * Adds test for new workflow * Adds airbyte repo * remove testing line * Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472) * write out new request options provider and refactor components and parts of the YAML config * fix formatting * pr feedback to consolidate body_data_provider to simplify the code * pr feedback get rid of extraneous optional * publish oss for cloud (#13978) workflow to publish oss artifacts that cloud needs to build against use docker buildx to create arm images for local development * skip debezium engine startup in case no table is in INCREMENTAL mode (#13870) * 🎉 Source Github: break point added for workflows_runs stream (#13926) Signed-off-by: Sergey Chvalyuk <[email protected]> * 6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866) * 6339: debug info * 6339: not using 'USE' on Azure SQL servers * 6339: cleanup * 6339: cleanup2 * 6339: cleanup3 * 6339: versions/changelogs updated * 6339: merge from master (consolidation issue) * 6339: dev connector version (for testing in airbyte cloud) * 6339: code review implementation * 6339: apply formatting * in case runners fail to spin up, this needs to run on github-hosted (#13996) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * :tada: New Source: Webflow (#13617) * Added webflow code * Updated readme * Updated README * Added webflow to source_definitions.yaml * Enhanced documentation for the Webflow source connector * Improved webflow source connector instructions * Moved Site ID to before API token in Spec.yaml (for presentation in the UI) * Addressed comments in PR. * Changes to address requests in PR review * Removed version from config * Minor udpate to spec.yaml for clarity * Updated to pass the accept-version as a constant rather than parameter * Updated check_connection to hit the collections API that requires both site id and the authentication token. * Fixed the test_check_connection to use the new check_connection function * Added a streams test for generate_streams * Re-named "autentication" object to "auth" to be more consistent with the way it is created by the CDK * Added in an explict line to instantiante an "auth" object from WebflowTokenAuthenticator, to make it easier to describe in the blog * Fixed a typo in a comment * Renamed some classes to be more intuitive * Renamed class to be more intuitive * Minor change to an internal method name * Made _get_collection_name_to_id_dict staticmethod * Fixed a unit-test error that only appeared when running " python -m pytest -s unit_tests". This was caused by Mocked settings from test_source.py leaking into test_streams.py * format: add double quotes and remove unused import * readme: remove semantic version naming of connector in build commands * Updated spec.yaml * auto-bump connector version * format files * add changelog * update dockerfile * auto-bump connector version Co-authored-by: sajarin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: marcosmarxm <[email protected]> * Source-oracle: fixed tests + checkstyle (#13997) * Source-oracle: fixed tests + checkstyle * 🐛Destination-mysql: fixed integration test and build process (#13302) * [13180] destination-mysql: fixed integration test * update changelog to include debezium version upgrade (#13844) * make table headers look less like successes (#13999) * source-twilio: implement lookback windows (#13896) * Revert "12708: Add an option to use encryption with staging in Redshift Destination (#13675)" (#14010) This reverts commit aa28d448d820df9d79c2c0d06b38978d1108fb2c. * Revert "6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866)" (#14011) This reverts commit 0d870bd37bc3b5cd798b92115d73bcc45a42d8f7. * [low-code connectors] BasicHttpAuthenticator (#13733) * implement basichttpauthenticator * add optional refresh access token authenticator * remove prints * type hints * Fix and unit test * missing test * Add class to __init__ file * Add comment * migrate JsonSchemas to use basic path instead of JSONPath (#13917) * scaffold for catalog diff, needs fixing on type handling and tests (#13786) * Prepare release of JDBC connectors (#13987) * Prepare release of JDBC connectors * Update source definitions manually * use built in check for if path is definite (#13834) * 13535 Fixed bastion network for integration tests (#14007) * doc: add error troubleshooting `docker-compose up` (#13765) * fix: duplicate resource allocations in `airbyte-temporal` deployment (#13816) * helm-chart: Fix worker deployment format error (#13839) * add catalog diff connection read (#13918) * doc: fix small typo on Shopify documentation (#13992) * add streams to reset to job info (#13919) * Generate api for changes in #13370 and make code compatible (#14014) * Generate api for per-stream updates #13835 (#14021) * Revert "Prepare release of JDBC connectors (#13987)" (#14029) This reverts commit df759b30778082508e2872513800fac34d98ff7c. * Fix per stream state protocol backward compatibility (#14032) * rename state type field to fix backwards compatibility issue * replace usages of stateType with type * support semi incremental by adding extractor record filter (#13520) * support semi incremental by adding extractor record filter * refactor extractor into a record_selector that supports extraction and filtering of response records * Remove pydantic spec from amazon ads and use YAML spec (#13988) * add EdDSA support in SSH tunnel (#9494) * add EdDSA support * verify EdDSA support works correct Co-authored-by: Yurii Bidiuk <[email protected]> * 🎉New source connector: source-metabase (#13752) * Add docs * Close metabase session when sync finishes * Close session in check_connection * Add source definition to seed * Add icon * improve cdc check for connectors (#14005) * improve should use cdc check * Revert "improve should use cdc check" This reverts commit 7d01727279d21d33a6c18ed3227ee94432636120. * improve should use cdc check * add unit test * Update webflow.md * Update webflow.md * Update webflow.md * Remove legacy sentry code from cdk (#14016) * rip sentry out of cdk * remove sentry dsn from gsc * Update webflow.md * Update webflow.md * Fixed broken links (#14071) * 🪟Persist unsaved changes on schema refresh (#13895) * add form values tracker context * add clarifying comment * add same functionality to create connection * Update airbyte-webapp/src/components/CreateConnectionContent/CreateConnectionContent.tsx Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * Fixes broken links so we can deploy again (#14075) also adds better error message for when this happens to others * Adds symmary.md to gitignore (#14078) * Added webflow icon (#14069) * Added webflow icon * Added icon * Build create connection form build failure (#14081) * Fix CDK obfuscation of nested secrets (#14035) * Added Buy Credits section to Managing Airbyte Cloud (#13905) * Added Buy Credits section to Managing Airbyte Cloud * Made some style changes * Made edits based on Natalie's suggestions * Deleted link * Deleted line * Edited email address * Updated reaching out to sales sentence * disable es-lit to fix build (#14087) * Release source connectors (#14077) * Release source connectors * Fix issue with database connection in test * Fix failing tests due to authentication * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump Airbyte version from 0.39.23-alpha to 0.39.24-alpha (#14094) Co-authored-by: jdpgrailsdev <[email protected]> * Emit the state to remove in the airbyte empty source (#13725) What This PR updates the EmptyAirbyteSource in order to perform a partial update and handle the new state message format. How The empty will now emit different messages based on the type of state being provided: Per stream: it will emit one message per stream that have been reset Global: It will emit one global message that will contain null for the stream that have been reset including the shared state Co-authored-by: Jimmy Ma <[email protected]> * Add StatePersistence object (#13900) Add a StatePersistence object that supports Read/Writes of States to the DB with StreamDescriptor fields The only migrations that is supported are * moving from LEGACY to GLOBAL * moving from LEGACY to STREAM * All other state type migrations are expected to go through an explicit reset beforehand. * secret-persistence: Hashicorp Vault Secret Store (#13616) Co-authored-by: Amanda Murphy <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> * 🐛 Source Hubspot: remove `AirbyteSentry` dependency (#14102) * fixed * updated changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * fix: format VaultSecretPersistenceTest.java (#14110) * Source Hubspot: extend error logging (#14054) * #291 incall - source Hubspot: extend error logging * huspot: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Update webflow.md (#14083) * Update webflow.md Removed a description that is only applicable to people that are writing connector code, not to _users_ of the connector. * Update webflow.md * Update webflow.md * Update webflow.md * Update webflow.md * 12708: Add an option to use encryption with staging in Redshift Desti… (#14013) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * 12708: specs updated * 12708: specs updated * 12708: removing autogenerated files from PR * 12708: changelog updated * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source PayPal Transaction: Update Transaction Schema (#13682) * Update transaction schema. * Transform money values from strings to floats or integers. Co-authored-by: nataly <[email protected]> Co-authored-by: Augustin <[email protected]> * fix(jsonSchemas): raise error when items property not provided (#14018) * fix stream name in stream transformation update (#14044) * 🐛 Destination Redshift: Improved discovery for redshift-destination not SUPER streams (#13690) airbyte-12843: Improved discovery for redshift-destination not SUPER tables, excluded views from discovery. * Remove skiptests option (#14100) * update sentry release script (#14123) * Remove "additionalProperties": false from specs for connectors with staging (#14114) * Remove "additionalProperties": false from spec for connectors with staging * Remove "additionalProperties": false from spec for Redshift destination * bump versions * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * [14003] source-oracle: added custom jdbc field (#14092) * [14003] source-oracle: added custom jdbc field * Add JobErrorReporter for sending sync job connector failures to Sentry (#13899) * skeleton for reporting connector errors to sentry * report on job failures instead of attempt failures * report sync job failures with relevant metadata using JobErrorReporter * send stack traces from python connectors to sentry * test JobCreationAndStatusUpdate and JobErrorReporter * logs * refactor into helper, initial tests * using sentry * run format * load reporting client from env * load sentry dsn from env * send java stack traces to sentry * test sentryclient, refactor to use Hub instance * ErrorReportingClient.report -> .reportJobFailureReason * inject exception helper, test stack trace parse error tagging * rm logs * more stack trace tests * remove logs * fix failing tests * rename ErrorReportingClient to JobErrorReportingClient * rename vars in docker-compose * Return an Optional instead of null when parsing stack traces * dont remove airbyte prefix when setting release name * from_trace_message static * remove failureSummary from jobfailure input, get from Job * send stacktrace string if we weren't able to parse * set deployment mode tag * update .env * just log if something goes wrong * Use StateMessageHelper in source (#14125) * Use StateMessageHelper in source * PR feedback and formatting * More PR feedback * Revert change * Revert changes * Bump Airbyte version from 0.39.24-alpha to 0.39.25-alpha (#14124) Co-authored-by: brianjlai <[email protected]> * Refactor acceptance tests and utils (#13950) * Refactor Basic acceptance tests and utils * Refactor Advanced acceptance tests and utils * Remove unused code * Clear destination db data during cleanup * Cleanup comments * cleanup init code * test creating new desintation db for each test * cleanup desintation db init * Allow to edit api client * pull in temporal cloud changes * Rename helper to harness; set some funcs to private; turn init into constructor * add func to set env vars instead of using static vars and move some functionality out of init into acceptance tests * update javadoc Co-authored-by: Davin Chia <[email protected]> * fix javadoc formatting * fix var naming Co-authored-by: Davin Chia <[email protected]> * Bump Airbyte version from 0.39.25-alpha to 0.39.26-alpha (#14141) Co-authored-by: terencecho <[email protected]> * 🎉 octavia-cli: Add ability to get existing resources (#13254) * 13541 Fixed integration tests source-db2 Mac OS (#14133) * 13523 Fix integration tests destination-cassandra Mac OS (#14134) * 🐛 Source Hubspot: fixed SAT test, commented out expected_records (#14140) * :bug: Source Intercom: extend `Contacts` schema with new properties (#14099) * Source Twilio: adopt best practices (#14000) * #1946 Source twilio: aopt best practices - tune tests * #1946 add expected_records to acceptance-test-config.yml * #1946 source twilio - upd schema and changelog * #1946 fix expected_records * #1946 source twilio: rm alerts from expected records as they expire in 30 days * #1946 source twilio: bump version * 🎉 Source BingAds: expose hourly/daily/weekly/monthly options from configuration (#13801) * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: config settings for hourly/daily/weekly/monthly reports added: default value for all periodic reports to True * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused class variables, if-statement * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused variables from config * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * remove VersionMismatchServer (#14076) * remove VersionMismatchServer * remove VersionMismatchServerTest * revert intended changes * Increase instance termination time limit to 3 hours to accommodate connector builds. (#14181) * Use correct bash comment symbol. (#14183) * 🎉 New Source: Orbit.love (#13390) * source-orbit: add definition and specs (#14189) * 🎉 Base Norrmalization: clean-up Redshift `tmp_schemas` after SAT (#14015) Now after `base-normalization` SAT the Destination Redshift will be automatically cleaned up from test leftovers. Other destinations are not covered yet. * Source Salesforce: fix customIntegrationTest for SAT (#14172) * Source Amazon Ads: increase timeout for SAT (#14167) * 🎉 Introduce Google Analytics Data API source (#12701) * Introduce Google Analytics Data API source https://developers.google.com/analytics/devguides/reporting/data/v1 * Add Google Analytics Data API source PR link * Add `client` class for Google Analytics Data API * Move dimensions and metrics extraction to the `client` class In the Google Analytics Data API * Change the copyright date to 2022 in Google Analytics Data API * fix: removing incremental syncs * fix: change project_id to string * fix: flake check is failing * chore: added it to source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * 🐛 Destination Redshift: use s3 bucket path for s3 staging operations (#13916) * Publish acceptance test utils maven artifact (#14142) * Fix StatePersistence Legacy read/write (#14129) StatePersistence will wrap/unwrap legacy state on write/read to ensure compatibility with the old behavior/data. * 🎉 Destination connectors: Improved "SecondSync" checks in Standard Destination Acceptance tests (#14184) * [11731] Improved "SecondSync" checks in Standard Destination Acceptance tests * 🐛 Source Zendesk Support: fixed "Retry-After" non integer value (#14112) Signed-off-by: Sergey Chvalyuk <[email protected]> * Source Tiktok Marketing: Videometrics (#13650) * added video metrics in streams.py * common metrics list updated. * updated streams.py with extended metrics required. * updated stream_test * updated configured_catalog * video metrics required list updated. * chore: formatting * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]> * 🎉 Source Github: secondary rate limits has to retry (#13955) Signed-off-by: Sergey Chvalyuk <[email protected]> * Harshith/test pr 13118 (#14192) * Firebolt destination * feat: Write method dropdown * feat: Use future-proof Auth in SDK * refactor: Move writer instantiation * fix: tests are failing * fix: tests are failing * fix: tests are failing * chore: added connector to definitions * fix: formatting and spec * fix: formatting for orbit Co-authored-by: ptiurin <[email protected]> * 🪟 :art: Show credit usage on chart's specific day (#13503) * add tooltip to chart * Fixes: - update main chart color; - change onHover background color * change chart color pallet to grey 500 * update color reference * remove opacity from UsageCell * 🐛 destination-redshift: use s3 bucket path for s3 cleanup (#14190) * Improve documentation for Postgres Source (#13830) * Improve documentation for Postgres Source * add information about additional JDBC params * add anchors for doc sections * fix link to CDC on Bare Metal * add more details about parsing date/time values * add doc link to SSH fields * Handle null reset source config (#14202) * handle null reset source config * format * Wait indefinitely if connection is not active (#14200) * also wait indefinitely if connection is deleted * fix test * Bump Airbyte version from 0.39.26-alpha to 0.39.27-alpha (#14204) Co-authored-by: lmossman <[email protected]> * Bmoric/feature flag for state deserialization (#14127) * Add Feature flag * Add default feature flag value * Update test * remove unsused * tmp * Update tests * rm unwanted change * PR comments * [low-code connectors] default types and default values (#14004) * default types and default values * cleanup * fixes so read works * remove prints and trycatch * comment * remove unused param * split file * extract method * extract methods * comment * optional * fix test * cleanup * delete interpolated request header provider * simplify next page url paginator interface * comment * format * add state type endpoint (#14111) * Bump Airbyte version from 0.39.27-alpha to 0.39.28-alpha (#14210) Co-authored-by: sherifnada <[email protected]> * 🐛 source-orbit: remove workspace_old.json (#14208) * Fix: Docs plural login redirecting to wrong URL (#14207) * [docs] fix numbering and incorrect filename in CDK docs (#13045) * [docs] fix numbering in CDK docs * Update 5-declare-schema.md * Update 5-declare-schema.md * Update 6-read-data.md * Update 8-test-your-connector.md * Remove the old scheduler from HelmCharts helper (#14187) * Remove the old scheduler from HelmCharts helper The old scheduler was removed as part of https://github.com/airbytehq/airbyte/pull/13400 * Remove legacy `scheduler` comment in HelmCharts * Source Gitlab: add GroupIssueBoards stream (#13252) * GitLab Source: add GroupIssueBoards stream * Address stream schema comments * Address comments * Bump version * Add as empty stream * run seed file source (#14215) * fix 'cannot reach server' error on demo instance (#10020) * Update CODEOWNERS (#14209) * 🎉 Source Github: use GraphQL for `reviews` stream (#13989) Signed-off-by: Sergey Chvalyuk <[email protected]> * workflow for publishing artifacts for cloud (#14199) * fix sentry org slug change (#14218) * Source File: correct spec json to match json format (#13738) * Upgrade spotless version and remove jvmargs workaround (#13705) * Source Zendesk Chat: Process large amount of data in batches for incremental (#14214) * increased the limit of itens in request * Configuration for max api pages on requests * included api_pagination_limit in sample * included api_pagination_limit in invalid_config * creating new table for chat_session * reverted api_pagination_limit approach * removed api_pagination_limit from TimeIncrementalStream * correct chat json * bump connector version * add changelog * run format * auto-bump connector version Co-authored-by: Roberto Bonnet <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Remove all @ts-ignore (#14221) * Bump hadoop to use version 3.3.3 (#14182) * Change the persistence activity to use the new persistence layer (#14205) * Change the persistence activity to use the new persistence layer * Use lombok * format * Use new State message helper * Fix build (#14225) * Fix build * Fix test * Use new state persistence for state reads (#14126) * Inject StatePersistence into DefaultJobCreator * Read the state from StatePersistence instead of ConfigRepository * Add a conversion helper to convert StateWrapper to State * Remove unused ConfigRepository.getConnectionState * Temporal per stream resets (#13990) * remove reset flags from workflow state + refactor * bring back cancelledForReset, since we need to distinguish between that case and a normal cancel * delete reset job streams on cancel or success * extract isResetJob to method * merge with master * set sync modes on streams in reset job correctly * format * Add test for getAllStreamsForConnection * fix tests * update more tests * add StreamResetActivityTests * fix tests for default job creator * remove outdated comment * remove debug lines * remove unused enum value * fix tests * fix constant equals ordering * make job mock not static * DRY and add comments * add comment about deleted streams * Remove io.airbyte.config.StreamDescriptor * regisster stream reset activity impl * refetch connection workflow when checking job id, since it may have been restarted * only cancel if workflow is running, to allow reset signal to always succeed even if batched with a workflow start * fix reset signal to use new doneWaiting workflow state prop * try to fix tests * fix reset cancel case * add acceptance test for resetting while sync is running * format * fix new acceptance test * lower sleep on test * raise sleep * increase sleep and timeout, and remove repeated test * use CatalogHelpers to extract stream descriptors * raise sleep and timeout to prevent transient failures * format Co-authored-by: alovew <[email protected]> * fix PostgresJdbcSourceAcceptanceTest by activating the feature flag (#14240) * fix PostgresJdbcSourceAcceptanceTest by activating the feature flag * fix AbstractJdbcSourceAcceptanceTest as well * fix expected_spec for strict encrypt * [13539] Fix integration tests source-clickhouse Mac OS (#14201) * [13539] Fix integration tests source-clickhouse Mac OS fixed unit tests * [13524] Fix integration tests destination-clickhouse Mac OS fixed unit tests * 6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#14121) * 6339: implementation * 6339: changelog updated * 6339: definitions updated * 6339: definitions reverted * 6339: still struggling with publishing * auto-bump connector version * 6339: definitions reverted - correct * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🪟 🎨 Update favicon and table row image styles (#14020) * style changes to favicon and imageblock * fix import * revert component and props names to block * Update airbyte-webapp/src/components/ImageBlock/ImageBlock.tsx Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * Update airbyte-webapp/src/components/ImageBlock/ImageBlock.module.scss Co-authored-by: Vladimir <[email protected]> * Update airbyte-webapp/src/components/ImageBlock/ImageBlock.tsx Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * Update airbyte-webapp/src/components/ImageBlock/ImageBlock.module.scss Co-authored-by: Vladimir <[email protected]> * add storybook Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: Vladimir <[email protected]> * upgrade potgresql version to fix default timestamp handling (#14211) * implement logic to trigger snapshot of new tables via debezium (#13994) * implement logic to trigger snapshot of new tables via debezium * format * improve test condition * fix build * BigQuery Denormalized "airbyte_type": "big_integer" to INT64 (#14079) * BigQuery Denormalized "airbyte_type": "big_integer" to INT64 * updated changelog * added unit test * removed star import * fixed checkstyle * bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Add Metrics section to Scaling Airbyte doc (#14224) * Added metrics section to scaling airbyte doc * Updated URL in doc * Deleted link * Added link * Added backslashes before brackets that aren't links * Edited note about tagged metrics * Changed list * Changed spacing * Changed spacing * Changed spacing * Deleted period * Fixed broken firebolt link * Added tables * Cleaned up wording in tables * Add ability to provide source/destination connector docker image (#14266) * Add ability to provide source/destination connector docker image * Make constant public * Bump Airbyte version from 0.39.28-alpha to 0.39.29-alpha (#14232) * disable flaky cmw test temporarily (#14269) * release new postgres source connector version 0.4.29 (#14265) * release new postgres source connector version 0.4.29 * add changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :tada: Source Tiktok marketing - remove granularity config option (#13890) * Removed granularity config option from spec, added corresponsing streams for each support granularity (hourly daily, lifetime), updated unittests, SAT * auto-formating * auto-formating * removed AdvertisersIds stream from list of exposed streams, updated docs * expose new style streams since 0.1.13, expose old streams for config for older version * update spec * fixed path to catalog * increased timeout * source bing-ads to ga (#13679) * Source Tiktok marketing - increase connector version (#14272) * increased connector version * increased connector version in seed * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Fix flaky connection manager workflow test (#14271) * try thread sleep instead of test env, and run 100 times * replace testEnv.sleep with Thread.sleep in several tests * replace RepeatedTest with Test * replace testEnv.sleep with Thread.sleep after signals are executed * run each test 100 times to see if any are flaky * add log * change repetitions to 100 to avoid out of memory * format * swap repeated test for normal test * 13532 Fixed integration tests destination-mssql Mac OS (#14252) * 13532 Fixed integration tests destination-mssql Mac OS * Source Google Analytics: Specify integer for dimension ga:dateHourMinute (#14298) * Specify integer for dimension ga:dateHourMinute * Update changelog * 🎉 Source Github: rename field `mergeable` to `is_mergeable` (#14274) Signed-off-by: Sergey Chvalyuk <[email protected]> * Update Airbyte Client (#14270) * #12668 #13198 enable full refresh, disable incremental and expected_records (#14191) * 🎉 Destination S3: update INSTANCE_PROFILE to use AWSDefaultProfileCredential (#14231) Co-authored-by: Mike Balmer <[email protected]> * Source Zendesk Support: pagination group membership (#14304) * add next_page_tooken and request * correct group_membership paginatin * update doc * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🪟 🐛 Fix OAuth validation not allowing to create source or destination (#14197) * Enable "Set up source/destination" button only if the form is valid * Update how ServiceForm initial values are patched so that it correctly patches the configuration with default values * Update initial values patching in service form to use initialValues to preserve already set values Update useOAuthFlowAdapter to correctly merge the values from the oauth response * Remove unused values var from ServiceForm * Add acceptance tests for per-stream state updates (#14263) * Add acceptance tests for per-stream state updates * PR feedback * Formatting * More PR feedback * PR feedback * Remove unused constant * Make sure that the feature flag is transfer to container (#14314) * Make sure that the feature flag is transfer to container * propagate the feature flags * Avoid propagating the feature flags * Fix tests * Source Postgres : use more simple and comprehensive query to get selectable tables (#14251) * use more simple and comprehensive query to get selectable tables * cover case when schema is not specified * add test to check discover with different ways of grants * format * incr ver * incr ver * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Fixed broken link * Fix for deleting stream resets (#14322) * Fix for deleting stream resets * Fix build by updating var (#14321) * Edited formatting (#14275) * Avoid error when creating dupl stream reset (#14328) * Bump Airbyte version from 0.39.29-alpha to 0.39.30-alpha (#14329) Co-authored-by: lmossman <[email protected]> * Release new postgres strict encrypt version (#14331) * Bump postgres strict encrypt version * Update changelogs * Update doc * Release new destination s3 version to pick up latest change (#14332) * Bump s3 version * Update pr id * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 13538 Fix integration tests destination-scylla Mac OS (#14308) * 13538 Fix integration tests destination-scylla Mac OS * Update cdk-speedrun.md (#14258) Added a link at the bottom of the article , so the user may find the more in-depth tutorial about building a real-world connector. * Update README.md (#14303) Added a link to https://airbyte.com/tutorials/extract-data-from-the-webflow-api in Webflow's README.md * Update building-a-python-source.md (#14262) * Update webflow.md (#14254) Added a link to the new blog - https://airbyte.com/tutorials/extract-data-from-the-webflow-api Co-authored-by: Simon Späti <[email protected]> * Alex/declarative stream incremental fix (#14268) * checkout files from test branch * read_incremental works * reset to master * remove dead code * comment * fix * Add test * comments * utc * format * small fix * Add test with rfc3339 * remove unused param * fix test * 🐛 SingerSource: Fix incompatibilities and typing issues (#14148) * Use logging.Logger in SingerSource * Fix SingerSource ConfigContainer This fixes typing issues with `ConfigContainer` and makes it compatible with `split_config`. Fixes #8710. * Fix SingerSource state and catalog typer issues * Rename SingerSource method args to match parent classes * Remove old comment about excluding Singer Co-authored-by: Alexandre Girard <[email protected]> * Update source postgres release stage to beta (#14326) * fix NPE (#14353) * fix NPE * Add test * Fix trailing * 🎉 octavia-cli: Add ability to import existing resources (#14137) * helm chart: Add Image Pull Secrets Param (#14031) * fix format (#14354) * Bump Airbyte version from 0.39.30-alpha to 0.39.31-alpha (#14355) Co-authored-by: benmoriceau <[email protected]> * tiktok to ga (#14358) * Update state.state type (#14360) * Run some DATs as part of base-normalization tests (#14312) * Revert "🎉 Source Github: rename field `mergeable` to `is_mergeable` (#14274)" (#14338) * Revert "🎉 Source Github: rename field `mergeable` to `is_mergeable` (#14274)" * Properly update the hasEmitted state (#14367) * Bmoric/state aggregator (#14364) * Update state.state type * Add state aggregator * Test and format * PR comments * Move to its own package * Update airbyte-workers/src/test/java/io/airbyte/workers/internal/state_aggregator/StateAggregatorTest.java Co-authored-by: Lake Mossman <[email protected]> * format * Update airbyte-workers/src/main/java/io/airbyte/workers/internal/state_aggregator/DefaultStateAggregator.java Co-authored-by: Lake Mossman <[email protected]> * format Co-authored-by: Lake Mossman <[email protected]> * Bump Airbyte version from 0.39.31-alpha to 0.39.32-alpha (#14383) Co-authored-by: alafanechere <[email protected]> * 🐛 Source Mixpanel: fix SAT tests (#14349) * Call the new revoke_user_session endpoint from the FE (#13165) * Source Instagram: change releaseStage to GA (#14162) * Source Google Analytics: Change releaseStage to GA (#13957) * source-outreach: fix record parsing and cursor field access (#14386) * Kustomize: Use `resources` since `bases` is deprecated (#14037) * fix: clone api doesn't take update configurations (#13592) * fix: clone api doesn't take update configurations * fix: you will be able to create clone in different workspace * fix: added description to source/destination body * cdk: Attach namespace to stream in catalog (#13923) * Source TiDB: correct jdbc string builder (#14243) * add icon for tidb-connector * Fix TiDB source connector * bump connector version * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Source Google Ads: use docsaurus feature for warn/note and udpdate doc (#14392) * use docsaurus feature for warn/note and udpdate doc * update description in supported streams * Source Facebook Marketing: allow configuration of MAX_BATCH_SIZE (#14267) * Add max batch size config * Bump semver * add changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 Source Github: add Retry for GraphQL API Resource limitations (#14376) Signed-off-by: Sergey Chvalyuk <[email protected]> * Add more metadata to the JobErrorReporter (#14395) * add workspace_id and connector_repository as tags * add tag for connection url * fix urls for job notifier * format * fix failing test * beta -> generally_available (#14315) Signed-off-by: Sergey Chvalyuk <[email protected]> * helm chart: Fix/double printing of extra volume mounts (#14091) * SentryJobErrorReporter: better handling of multiline chained java exceptions (#14398) * Docs: deploy on gcp use docusaurus tabs (#14401) * Revert "Kustomize: Use `resources` since `bases` is deprecated (#14037)" (#14415) This reverts commit 5c9a6a5fc655a9e597f755be8fc8ccf805a2537a. * Use Debezium Postgres image for CDC tests (#14318) * Use Debezium Postgres image for CDC tests * Formatting * 🎉 octavia-cli: Add ability to import all resources (#14374) * Bump Airbyte version from 0.39.32-alpha to 0.39.33-alpha (#14419) Co-authored-by: pedroslopez <[email protected]> * 📝 MySql source: clarify tinyint to number conversion when size > 1 (#14424) * 🪟 🐛 Fix Setup Source Button on OAuth Sources (#14413) * don't disable setup button * make eslint happy * one more cleanup * use the spec to decide how to create config object * Bump Airbyte version from 0.39.33-alpha to 0.39.34-alpha (#14428) Co-authored-by: timroes <[email protected]> * [low-code cdk] Enable configurable state checkpointing (#14317) * checkout files from test branch * read_incremental works * reset to master * remove dead code * comment * fix * Add test * comments * utc * format * small fix * Add test with rfc3339 * remove unused param * fix test * configurable state checkpointing * update test * fix type hints (#14352) * normalization: Do not return NULL for MySQL column values > 512 chars (#11694) Co-authored-by: Augustin <[email protected]> Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Tim Roes <[email protected]> Co-authored-by: Charles <[email protected]> Co-authored-by: Jonathan Pearlin <[email protected]> Co-authored-by: Amruta Ranade <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Ganpat Agarwal <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Rajakavitha Kodhandapani <[email protected]> Co-authored-by: Yevhen Sukhomud <[email protected]> Co-authored-by: Andrii Leonets <[email protected]> Co-authored-by: George Claireaux <[email protected]> Co-authored-by: VitaliiMaltsev <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: sw-yx <[email protected]> Co-authored-by: Baz <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: alafanechere <[email protected]> Co-authored-by: Eugene <[email protected]> Co-authored-by: Denis Davydov <[email protected]> Co-authored-by: Anna Lvova <[email protected]> Co-authored-by: Vladimir <[email protected]> Co-authored-by: Phlair <[email protected]> Co-authored-by: Parker Mossman <[email protected]> Co-authored-by: Adam <[email protected]> Co-authored-by: Liren Tu <[email protected]> Co-authored-by: pmossman <[email protected]> Co-authored-by: Topher Lubaway <[email protected]> Co-authored-by: Brian Lai <[email protected]> Co-authored-by: Peter Hu <[email protected]> Co-authored-by: Subodh Kant Chaturvedi <[email protected]> Co-authored-by: Tuhai Maksym <[email protected]> Co-authored-by: Alexander Marquardt <[email protected]> Co-authored-by: sajarin <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: steve withington <[email protected]> Co-authored-by: Leo Sussan <[email protected]> Co-authored-by: cenegd <[email protected]> Co-authored-by: Tomas Perez Alvarez <[email protected]> Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Edward Gao <[email protected]> Co-authored-by: Yurii Bidiuk <[email protected]> Co-authored-by: Christophe Duong <[email protected]> Co-authored-by: Teal Larson <[email protected]> Co-authored-by: Sophia Wiley <[email protected]> Co-authored-by: jdpgrailsdev <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Stella Chung <[email protected]> Co-authored-by: Amanda Murphy <[email protected]> Co-authored-by: Mohamed Magdy <[email protected]> Co-authored-by: nataly <[email protected]> Co-authored-by: Tyler Russell <[email protected]> Co-authored-by: Alexander Tsukanov <[email protected]> Co-authored-by: Pedro S. Lopez <[email protected]> Co-authored-by: brianjlai <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Davin Chia <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Daniel Diamond <[email protected]> Co-authored-by: drrest <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Abhi Vaidyanatha <[email protected]> Co-authored-by: Harshith Mullapudi <[email protected]> Co-authored-by: Zawar Khan <[email protected]> Co-authored-by: ptiurin <[email protected]> Co-authored-by: Greg Solovyev <[email protected]> Co-authored-by: lmossman <[email protected]> Co-authored-by: sherifnada <[email protected]> Co-authored-by: Sachin Jangid <[email protected]> Co-authored-by: Chris Wu <[email protected]> Co-authored-by: Jared Rhizor <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Roberto Bonnet <[email protected]> Co-authored-by: Malik Diarra <[email protected]> Co-authored-by: alovew <[email protected]> Co-authored-by: Oleksandr Sheheda <[email protected]> Co-authored-by: midavadim <[email protected]> Co-authored-by: Arsen Losenko <[email protected]> Co-authored-by: Ryan Lewon <[email protected]> Co-authored-by: Mike Balmer <[email protected]> Co-authored-by: Anne <[email protected]> Co-authored-by: Liren Tu <[email protected]> Co-authored-by: Simon Späti <[email protected]> Co-authored-by: Albin Skott <[email protected]> Co-authored-by: Caleb Fornari <[email protected]> Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Christian Martin <[email protected]> Co-authored-by: jordan-glitch <[email protected]> Co-authored-by: Daemonxiao <[email protected]> Co-authored-by: Keith Thompson <[email protected]> Co-authored-by: Leo Sussan <[email protected]> Co-authored-by: pedroslopez <[email protected]> Co-authored-by: timroes <[email protected]> Co-authored-by: Johannes Nicolai <[email protected]>

Removed part_size from connectors that use StreamTransferManager

c853af3

octavia-squidington-iii added connectors/destination/bigquery connectors/destination/bigquery-denormalized connectors/destination/snowflake connectors/destination/redshift connectors/destination/s3 connectors/destination/gcs connectors/destination/jdbc labels Jun 14, 2022

github-actions bot added the area/connectors Connector related issues label Jun 14, 2022

VitaliiMaltsev changed the title ~~Removed PART_SIZE_MB in connectors using S3/GCS storage~~ Deprecate PART_SIZE_MB in connectors using S3/GCS storage Jun 14, 2022

VitaliiMaltsev added 2 commits June 14, 2022 13:52

fixed S3DestinationConfigTest

416ae7a

fixed S3JsonlFormatConfigTest

00dbd77

VitaliiMaltsev added 2 commits June 14, 2022 14:39

Merge branch 'master' into vmaltsev/11389-deprecate-part-size

bb6fbbf

Merge branch 'master' into vmaltsev/11389-deprecate-part-size

189b276

VitaliiMaltsev requested review from alexandr-shegeda and suhomud June 16, 2022 10:39

alexandr-shegeda linked an issue Jun 16, 2022 that may be closed by this pull request

Connector UX: deprecate PART_SIZE_MB in connectors specs using S3/GCS storage #11389

Closed

alexandr-shegeda approved these changes Jun 16, 2022

View reviewed changes

alexandr-shegeda marked this pull request as ready for review June 16, 2022 13:35

upadate changelog and bump version

4e02f0c

github-actions bot added the area/documentation Improvements or additions to documentation label Jun 17, 2022

VitaliiMaltsev temporarily deployed to more-secrets June 19, 2022 13:42 Inactive

auto-bump connector version

0477041

octavia-squidington-iii temporarily deployed to more-secrets June 19, 2022 14:21 Inactive

VitaliiMaltsev added 3 commits June 20, 2022 11:11

Merge branch 'master' into vmaltsev/11389-deprecate-part-size

b2c7592

fix GCS staging test

e4a42e5

Merge remote-tracking branch 'origin/vmaltsev/11389-deprecate-part-si…

a63c43e

…ze' into vmaltsev/11389-deprecate-part-size

VitaliiMaltsev temporarily deployed to more-secrets June 20, 2022 09:41 Inactive

fix GCS staging test

738548d

VitaliiMaltsev temporarily deployed to more-secrets June 20, 2022 10:03 Inactive

auto-bump connector version

b4c72e8

VitaliiMaltsev merged commit c283d9d into master Jun 20, 2022

VitaliiMaltsev deleted the vmaltsev/11389-deprecate-part-size branch June 20, 2022 11:56

octavia-squidington-iii temporarily deployed to more-secrets June 20, 2022 11:57 Inactive

octavia-squidington-iii mentioned this pull request Jun 20, 2022

Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha #13938

Merged

VitaliiMaltsev mentioned this pull request Jun 24, 2022

Remove "additionalProperties": false from specs for connectors with staging #14114

Merged

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deprecate PART_SIZE_MB in connectors using S3/GCS storage #13753

Deprecate PART_SIZE_MB in connectors using S3/GCS storage #13753

VitaliiMaltsev commented Jun 14, 2022 •

edited

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda left a comment

VitaliiMaltsev commented Jun 17, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 17, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 17, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 19, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 19, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 20, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 20, 2022 •

edited by github-actions bot

Loading

danieldiamond commented Jun 21, 2022

danieldiamond commented Jun 21, 2022

VitaliiMaltsev commented Jun 21, 2022

edgao commented Jun 22, 2022

VitaliiMaltsev commented Jun 23, 2022

edgao commented Jun 23, 2022

VitaliiMaltsev commented Jun 24, 2022

Deprecate PART_SIZE_MB in connectors using S3/GCS storage #13753

Deprecate PART_SIZE_MB in connectors using S3/GCS storage #13753

Conversation

VitaliiMaltsev commented Jun 14, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

VitaliiMaltsev commented Jun 14, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Jun 14, 2022 • edited by github-actions bot Loading

Build Failed

VitaliiMaltsev commented Jun 14, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Jun 14, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Jun 14, 2022 • edited by github-actions bot Loading

Build Failed

VitaliiMaltsev commented Jun 14, 2022 • edited by github-actions bot Loading

Build Passed

alexandr-shegeda left a comment

Choose a reason for hiding this comment

VitaliiMaltsev commented Jun 17, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Jun 17, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Jun 17, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Jun 19, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Jun 19, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Jun 20, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Jun 20, 2022 • edited by github-actions bot Loading

danieldiamond commented Jun 21, 2022

danieldiamond commented Jun 21, 2022

VitaliiMaltsev commented Jun 21, 2022

edgao commented Jun 22, 2022

VitaliiMaltsev commented Jun 23, 2022

edgao commented Jun 23, 2022

VitaliiMaltsev commented Jun 24, 2022

VitaliiMaltsev commented Jun 14, 2022 •

edited

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 17, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 17, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 17, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 19, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 19, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 20, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jun 20, 2022 •

edited by github-actions bot

Loading