🎉 New Source: Chartmogul

[TSkrebe]

What

This PR adds a new source: chartmogul api

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed.

---

This change is 

[TSkrebe]

./acceptance-test-docker.sh

 test_core.py ✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                        93% █████████▍
 test_full_refresh.py ✓                                                                                                                                                                                           100% ██████████
==================================================================================================== short test summary info ====================================================================================================
SKIPPED [1] source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config

Results (73.78s):
      14 passed

[TSkrebe]

Not sure how to proceed with airbyte-integrations/builds.md

[marcosmarxm]

The connector looks well implemented, I created one issue to get credentials to apply CI tests; If possible to share a testing credentials this helps to merge sooner.

[marcosmarxm]

Not sure how to proceed with airbyte-integrations/builds.md

The link is built after the connector is merged.

[TSkrebe]

i'm in no rush, just curious on ETA when we could see this published?

[alafanechere]

Hi @TSkrebe, we created a sandbox account for Chartmogul. I'll now open a side branch to run the acceptance tests on this account. If they pass I'll go for a review, and if I've no additional change request we can publish it today the beginning of next week.

[alafanechere]

Hi @TSkrebe I made some minor changes to make the acceptance tests pass:

• Run ./gradlew format
• Fix some unused import to make Flake check pass
• Extensive use of mocker fixture in unit tests to replace mock.MagicMock with mocker.MagicMock
• Change customer id field to integer as it's the type the API returned
• Make primary key fields not nullable in streams schema. Your schemas have a lot of nullable fields, please double-check they are all really nullable according to Chartmogul API reference.
• Declare activities as an empty stream in basic_read acceptance test. Our sandbox account does not have any activity. Do you know how we could generate an activity? I'm not familiar with Chartmogul.

I highlighted the changes on which I'd like a reply from your side.

[alafanechere]

Not sure how to proceed with airbyte-integrations/builds.md

You have to add a line for your connector at the right alphabetical place and copy what's been made for other connectors.
I did it in bd82338

[alafanechere]

Thank you @TSkrebe, the overall code looks good to me. I still have some minor suggestions. Let me know what you think.

[TSkrebe]

@alafanechere

Change customer id field to integer as it's the type the API returned

makes sense, don't why it was a string. thanks

Declare activities as an empty stream in basic_read acceptance test. Our sandbox account does not have any activity. Do you know how we could generate an activity? I'm not familiar with Chartmogul.

in my case, there is integration with stripe https://app.chartmogul.com/#/data-platform/data-sources . Whenever there is a payment, a new activity is created. If there is any activities in your stripe sandbox it should theoretically work.

Make primary key fields not nullable in streams schema. Your schemas have a lot of nullable fields, please double-check they are all really nullable according to Chartmogul API reference.

Since they don't provide open api, it's difficult to know which ones can and can't be null.

thanks for the feedback on python style. Difficult to switch from golang midset... will address after the weekend.

[TSkrebe]

@alafanechere addressed your comments. let me know if i missed anything. you'll probably need to run gradlew command again as i wasn't able to get it up-and-running on my M1

[alafanechere]

Thanks for the changes you made. I confirm that acceptance test are passing locally for me after I made some unused import removal that blocked the flake check.
I'm going to make the test pass in the CI on my side branch + publish the connector, then we'll be ready to merge.