Upgrade the Snowflake Python Connector to version 2.7.8 or later

[sfc-gh-rjoshi]

Description

As per the change made by Snowflake (affecting customers on GCP), kindly update the 'Snowflake' Python Connector version to version 2.7.8 or later.

Please note all recent versions of Snowflake SQL-alchemy connector have support for this change as they use the Python Connector more recent than above.

Here is the complete information on the change reasons and recommendations - https://community.snowflake.com/s/article/faq-2023-client-driver-deprecation-for-GCP-customers

Use case/motivation

If this change is not made Airflow customers on GCP will not be able to perform PUT operations to their Snowflake account.

Soft Cutover enforced by Snowflake is Oct 30, 2023.
Hard Cutover enforced by Google is Jan 15, 2024

https://community.snowflake.com/s/article/faq-2023-client-driver-deprecation-for-GCP-customers

Related issues

https://community.snowflake.com/s/article/faq-2023-client-driver-deprecation-for-GCP-customers

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[sfc-gh-rjoshi]

https://github.com/apache/airflow/blob/main/airflow/providers/snowflake/provider.yaml#L66
https://github.com/apache/airflow/blob/main/airflow/providers/snowflake/provider.yaml#L67

https://github.com/apache/airflow/blob/main/generated/provider_dependencies.json#L825
https://github.com/apache/airflow/blob/main/generated/provider_dependencies.json#L826

[Lee-W]

Maybe we can create a PR with the version updated and see how it works with the CI and how the community reacts to it 🙂

[eladkal]

We allow any version of snowflake-connector-python>=2.4.1
thus if you want to use 2.7.8 you can. Nothing prevents you from doing so.

If I use snowflake as is without gcp why should I be forced to use snowflake-connector-python>=2.7.8 ?

[sfc-gh-rjoshi]

Although version 2.7.8 is for GCP, but it is still very old.  The requirements in the repo mention unsupported versions (2.4.1 / 1.0.0) . The versions need an update either way to keep up with the versions supported by Snowflake.

Please find below from the requirements page https://docs.snowflake.com/en/release-notes/requirements

Snowflake Connector for Python

• Recommended 3.2.1 (or later)
• Nearing End of Support 2.7.0

Snowflake SQLAlchemy (for Python)

• Recommended - 1.5.0 (or later)
• Nearing End of Support 1.3.2

[eladkal]

But we are not limiting the users.. they can choose what they like.
Airflow also recommend users to update their Airflow versions but not everyone can. Some users still use versions who are 3 years old and in EOL.

We do our best to support all possible versions. We let the users to decide what to use.
Airflow providers and recommends to use the official constraints file:
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html
Which takes into account all dependencies and set the highest possible version for each library.

I am fine with bumping versions if there is good reason (old is not a reason) because old versions can still work.
I myself worked with some old versions of libraries due to some complex issue that it took long time to resolve.

If you'll take a look at the other dependencies we have you will see for example we bumped many Azure packages lately but that is because we had to do it to include new features that are part of the Azure integration to Airflow itself (hooks, operators).

Maybe @potiuk have different view on this?

[potiuk]

Although version 2.7.8 is for GCP, but it is still very old.  The requirements in the repo mention unsupported versions (2.4.1 / 1.0.0) . The versions need an update either way to keep up with the versions supported by Snowflake.

Maybe @potiuk have different view on this?

I think I agree with you @eladkal that we should be careful, but also I think bumping to 2.7.8 should be quite fine here. Those two are not conflicting IMHO in this case.

Currently the "golden" set of constraints we have - 3.2.1 / 1.5.0, so it will be generally "no-op" to bump the min versions to any higher version lower than that (from the maintenance point of view).

For many smaller providers, we can just use very "recent" versions as min versions. When those providers have very few dependencies on their own. But Snowflake provider is different - the connector has 21 dependencies - many of them limited and many of them shared with other libraries. So bumping min version might cause the potential conflicts that @eladkal described (it already has in the past).

One note worth mentioning though. We should be just a "little" careful, that it will only happen when someone actually installs snowflake provider. It does not limit "airlfow core" package, so it is highly likely that users who install snowflake provider will not want to run old, unsupported snowflake connector anyway. Also those who want to do it regardless, can still install previous provider.

I guess you assesed Airflow as important driver for your adoption - and we absolutely welcome involvement of the stakeholders in our providers in being more active in maintaining them, so I hope this might be a beggining of such cooperation.

Happy to review and help to get it merged - will you do it @sfc-gh-rjoshi ?

[potiuk]

@sfc-gh-rjoshi ? How about implementing it ?