update some connectors to use YAML specs #12230

pedroslopez · 2022-04-20T23:45:00Z

What

#11668 introduced yaml spec support, now let's actually use it!

How

Update 5 connectors to use a spec.yaml:

I basically ran the old spec.json files through https://codebeautify.org/json-to-yaml and made minor tweaks (mainly change single quotes to double quotes)

Also updated some references to the old files and bumped connector versions.
Doc updates coming soon w/ references to new spec.yaml files.

pedroslopez · 2022-04-20T23:53:46Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2198945566
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2198945566
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_stripe/__init__.py       2      0   100%
source_stripe/streams.py      235     66    72%
source_stripe/source.py        22     11    50%
-----------------------------------------------
TOTAL                         259     77    70%

pedroslopez · 2022-04-20T23:54:12Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2198946783
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2198946783
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               150     19    87%
source_salesforce/streams.py           282     60    79%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             75     33    56%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  547    126    77%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             75      6    92%
source_salesforce/api.py               150     14    91%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           282     41    85%
--------------------------------------------------------
TOTAL                                  547     65    88%

pedroslopez · 2022-04-21T00:15:32Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199015565
❌ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199015565
🐛 https://gradle.com/s/3qyv5nydwzd3g
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_oneof_usage[inputs0] - AssertionError: Th...
=================== 1 failed, 26 passed in 118.06s (0:01:58) ===================

pedroslopez · 2022-04-21T00:23:22Z

/test connector=connectors/source-google-sheets

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199043935
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199043935
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
google_sheets_source/models/spreadsheet_values.py      12      0   100%
google_sheets_source/models/spreadsheet.py             34      0   100%
google_sheets_source/models/__init__.py                 2      0   100%
google_sheets_source/__init__.py                        2      0   100%
google_sheets_source/helpers.py                       139     26    81%
google_sheets_source/client.py                         22      5    77%
google_sheets_source/google_sheets_source.py          103     84    18%
-----------------------------------------------------------------------
TOTAL                                                 314    115    63%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 22 passed, 1 skipped in 121.90s (0:02:01) ===================

pedroslopez · 2022-04-21T00:43:38Z

/test connector=connectors/source-exchange-rates

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199103908
✅ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199103908
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 19 passed, 2 skipped in 16.33s ========================

pedroslopez · 2022-04-21T00:48:15Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199117927
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199117927
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_hubspot/errors.py         6      0   100%
source_hubspot/__init__.py       2      0   100%
source_hubspot/streams.py      707     56    92%
source_hubspot/source.py        69     12    83%
------------------------------------------------
TOTAL                          784     68    91%

pedroslopez · 2022-04-21T00:50:44Z

airbyte-integrations/connectors/source-stripe/setup.py

@@ -19,7 +19,7 @@
    author_email="[email protected]",
    packages=find_packages(),
    install_requires=MAIN_REQUIREMENTS,
-    package_data={"": ["*.json", "schemas/*.json", "schemas/shared/*.json"]},
+    package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},


note: not adding *.yaml here weirdly didn't seem to cause any issues 🤔

pedroslopez · 2022-04-21T00:51:08Z

airbyte-integrations/connectors/source-stripe/source_stripe/spec.yaml

@@ -0,0 +1,47 @@
+documentationUrl: https://docs.airbyte.io/integrations/sources/stripe
+connectionSpecification:
+  $schema: http://json-schema.org/draft-07/schema#


feels a little odd to be doing this in a yaml file, but I think it's still right?

pedroslopez · 2022-04-21T00:53:05Z

airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.yaml

+              type: string
+              title: Credentials Title
+              description: Name of the credentials set
+              const: OAuth Credentials


This is slightly different than the spec.json counterpart: previously default and enum were defined, but it looks like that's not necessary anymore and was causing test failures.

Yup! Thanks for the change

pedroslopez · 2022-04-21T01:05:24Z

airbyte-integrations/connectors/source-hubspot/setup.py

@@ -6,7 +6,7 @@
 from setuptools import find_packages, setup

 MAIN_REQUIREMENTS = [
-    "airbyte-cdk~=0.1.49",
+    "airbyte-cdk~=0.1.55",


Just pointing out that I noticed only hubspot has a patch version in the requirements. Others either don't have a version at all, or are set to ~=0.1

pedroslopez · 2022-04-21T01:45:48Z

/publish connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2199283730
🚀 Successfully published connectors/source-stripe
❌ Couldn't auto-bump version for connectors/source-stripe

pedroslopez · 2022-04-21T01:46:20Z

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2199285286
🚀 Successfully published connectors/source-salesforce
🚀 Auto-bumped version for connectors/source-salesforce
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2199285286

codecov · 2022-04-21T02:08:45Z

Codecov Report

❗ No coverage uploaded for pull request base (master@53799cb). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #12230   +/-   ##
=========================================
  Coverage          ?   82.93%           
=========================================
  Files             ?       21           
  Lines             ?     1904           
  Branches          ?        0           
=========================================
  Hits              ?     1579           
  Misses            ?      325           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53799cb...16bc799. Read the comment docs.

pedroslopez · 2022-04-21T02:11:56Z

/publish connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199353483
🚀 Successfully published connectors/source-hubspot
🚀 Auto-bumped version for connectors/source-hubspot
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199353483

pedroslopez · 2022-04-21T02:13:07Z

/publish connector=connectors/source-google-sheets auto-bump-version=false

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199356387
🚀 Successfully published connectors/source-google-sheets
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199356387

pedroslopez · 2022-04-21T02:13:34Z

/publish connector=connectors/source-exchange-rates auto-bump-version=false

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199357564
🚀 Successfully published connectors/source-exchange-rates
✅ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199357564

* salesforce uses spec.yaml * stripe uses spec.yaml * revert unintended changes * bump stripe version * add yaml files to package_data * hubspot uses a spec.yaml * use double quotes in yaml files * google sheets uses spec.yaml * exhcange-rates uses spec.yaml * remove usage of default and enum in spec * bump hubspot cdk requirement * auto-bump connector version * auto-bump connector version * manually bump versions Co-authored-by: Octavia Squidington III <[email protected]>

pedroslopez added 2 commits April 20, 2022 19:42

salesforce uses spec.yaml

1ea0369

stripe uses spec.yaml

73e8efb

pedroslopez linked an issue Apr 20, 2022 that may be closed by this pull request

Update 5 connectors to use YAML spec #11891

Closed

github-actions bot added the area/connectors Connector related issues label Apr 20, 2022

pedroslopez added 2 commits April 20, 2022 19:48

revert unintended changes

a350121

bump stripe version

d7b3bd7

pedroslopez added 3 commits April 20, 2022 19:55

add yaml files to package_data

87e8609

hubspot uses a spec.yaml

f2cf0e6

use double quotes in yaml files

8f30a4e

google sheets uses spec.yaml

dd29c09

exhcange-rates uses spec.yaml

ff1d757

remove usage of default and enum in spec

741234b

pedroslopez commented Apr 21, 2022

View reviewed changes

bump hubspot cdk requirement

110e704

pedroslopez commented Apr 21, 2022

View reviewed changes

pedroslopez marked this pull request as ready for review April 21, 2022 01:05

pedroslopez requested review from sherifnada and Phlair April 21, 2022 01:07

sherifnada approved these changes Apr 21, 2022

View reviewed changes

auto-bump connector version

9e8a945

octavia-squidington-iii temporarily deployed to more-secrets April 21, 2022 02:06 Inactive

auto-bump connector version

e579adf

octavia-squidington-iii temporarily deployed to more-secrets April 21, 2022 02:28 Inactive

manually bump versions

16bc799

pedroslopez temporarily deployed to more-secrets April 21, 2022 02:37 Inactive

pedroslopez merged commit aa872d0 into master Apr 21, 2022

pedroslopez deleted the pedroslopez/yaml-spec-connectors branch April 21, 2022 02:59

octavia-squidington-iii mentioned this pull request Apr 21, 2022

Bump Airbyte version from 0.36.1-alpha to 0.36.2-alpha #12258

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update some connectors to use YAML specs #12230

update some connectors to use YAML specs #12230

pedroslopez commented Apr 20, 2022 •

edited

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez Apr 21, 2022

pedroslopez Apr 21, 2022

pedroslopez Apr 21, 2022

sherifnada Apr 21, 2022

pedroslopez Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

codecov bot commented Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

update some connectors to use YAML specs #12230

update some connectors to use YAML specs #12230

Conversation

pedroslopez commented Apr 20, 2022 • edited Loading

What

How

pedroslopez commented Apr 20, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 20, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez Apr 21, 2022

Choose a reason for hiding this comment

pedroslopez Apr 21, 2022

Choose a reason for hiding this comment

pedroslopez Apr 21, 2022

Choose a reason for hiding this comment

sherifnada Apr 21, 2022

Choose a reason for hiding this comment

pedroslopez Apr 21, 2022 • edited Loading

Choose a reason for hiding this comment

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

codecov bot commented Apr 21, 2022 • edited Loading

Codecov Report

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading

pedroslopez commented Apr 20, 2022 •

edited

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

codecov bot commented Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading