sendgrid as a data source provider? #24815

[tx0c]

Description

I'm seeing #2727 added Sendgrid EmailOperator, but another question is does Airflow support to synchronize data from Sendgrid as a data source?

here are some existing commercial solutions:

1. https://segment.com/docs/connections/sources/catalog/cloud-apps/sendgrid/
2. https://airbyte.com/connections/SendGrid-to-PostgreSQL
3. https://sendgrid.com/blog/pipe-your-sendgrid-data-in-a-data-warehouse/ (was written 2016)
4. https://hevodata.com/learn/sendgrid-to-bigquery/#method2

the need include whatever was written on the sendgrid blog post in 2016:

Pipe Your SendGrid Data in a Data Warehouse
Through Segment, SendGrid customers can:

• Find the best time of day and day of week to send emails
• Build a custom funnel measuring email events to in-app conversions
• Analyze email opens, email link clicks, and email unsubscribes across all your campaigns
• Get a complete history of how many emails customers receive over time
• Combine email interactions with in-app behavior to identify high intent prospects and customers likely to churn

but since 2016, during the past 6 years, did someone hit the same need to sync sendgrid data? is there an existing open source solution to synchronize data from Sendgrid?

I have researched this issues board and discussions board, it seemed all threads mentioned sendgrid were using Sendgrid as an EmailOperator,

Thanks,

Use case/motivation

No response

Related issues

link #24815
#24815 (reply in thread)

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I marked it as "good first issue" - this is an open source software that people contribute to, maybe someone will pick it up. But also as mentioned in #24815 the best way to make sure it gets implemented if you want it, is to implement it yourself @tx0c

[mik-laj]

Do you consider to use airbyte? https://airbyte.com/connectors/sendgrid

[ZhangCreations]

Hi can I claim this issue?

[potiuk]

Feel free - but airbyte track should be considered as alternative and maybe this one is not needed ?

[ZhangCreations]

Just to double check my understanding of the release cycle as outlined here. If I implement this provider, does that require me to continue to maintain the versioning of this provider indefinitely?

[ZhangCreations]

Hi just following up on this, hoping to get started on this toon and just wanted to see how contributions would work in this section of the code base.

[potiuk]

Just to double check my understanding of the release cycle as outlined here. If I implement this provider, does that require me to continue to maintain the versioning of this provider indefinitely?

What is in main and accepted by the community, is maintained by the community. It would be nice if you stick around and help to solve issues, but if it is sufficiently covered by tests, others will be able to fix the problems raised.

Our CI is pretty comprehensive and review is rather detailed, so once the PR is merged, all the requiremetns are checked. If you get it to the point it will be mergeable and merged (and approved), there is nothing more from your side "expected", but for sure staying around will be nice to take some care (especially about teething problems). But Airflow is done by volunteers, there is no way we can force or oblige anyone (that includes also commiters and PMC members) to continue contributing. As an individual you have completely free will.

There might a number of problems that might make it dfficult to merge it (dependency issues mainly) - because we need to make sure all our providers are working "together" and that we have a consistent set of dependencies. Community work is only focused on "main" releases of providers - basically we build and release only latest verison (and depending on changes applied by the community - it might be patchlevel/feature/breaking changes. But we also do not heve a way currently to run the tests against "real" services - so we rely on thoso who contribute changes to test them - but this also means that some versions of providers might get occasionally broken.

For those who wish to give more care - we have the AIP-47 effort - to allow external parties to use "example_dags" as system tests and run them regularly - and anyone can volunteer to do so and to build a pipeline of running actual "system tests" running against real services. This is not mandatory and requires some party (individual or organisation) to volunteer to do it. Currently Amazon and Google are preparing to run such regular tests on their provider. Others (like Databricks) might follow (I hope) - if Sendgrid would like to do it - they can do it as well - or you could do it as well. But we - in the community have no capacity to run them.

Finally if any party wants to release the provider for one of the past relases of it - they can do it by branching off that past release and cherry-picking the changes and fixing all the problems they find - once they do we then can release the past version (this is what the documentation you linked is mostly about). But this is only if someone wants to release past version of any provider for the reasons of backwards compatibility for example. Latest release is always prepared and relased by the provider release manager from the code that is currently in main and it only happens when there are some changes since last release (or when we bump minor airflow version). And there author of the change is usually asked to help with testing the released package (not original author of it). In the first release after it is merged, your changes will be there, so you will be asked for help in testing it.

Again - you do not HAVE TO do it - there is no way we can force anyone to do it - those expectations above however are kind of "good citizenship" expectations and it's great if you can comply .