Pattern parameter in S3ToSnowflakeOperator

[Taragolis]

closes: #24360

[uranusjr]

No expertise to judge whether pattern needs special escaping (e.g. what if the pattern contains a '?) but files is also currently not escaped or validated, so perhaps not? And if there’s a bug, we need to fix them all anyway, and that should happen in a separate PR, not this one.

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[uranusjr]

Doc build failure is unrelated (see #24575).

[Taragolis]

No expertise to judge whether pattern needs special escaping (e.g. what if the pattern contains a '?) but files is also currently not escaped or validated, so perhaps not? And if there’s a bug, we need to fix them all anyway, and that should happen in a separate PR, not this one.

This is real good question. About pattern it should always be escaped by single quote (doc) but It unclear what if pattern itself should contain single quote. I agree that at least it should be mention in documentation/docstring.

Never use option files for Snowflake, seems like it should also escaped, but documentation unclear about this.

Anyway I will try to check on actual Snowflake during the day different cases

[Taragolis]

files is also currently not escaped or validated,

Just check and found that files actually also quoted however not validated.

airflow/airflow/providers/snowflake/transfers/s3_to_snowflake.py

Line 122 in 904cc12

files = ", ".join(f"'{key}'" for key in self.s3_keys)

Single quote is a safe to use special character for S3 key/prefixes.

That mean if user has ' in S3 key in current implementation most probably operator execution would fail.

[josh-fell]

CC @grzegorzb1990 since the issue was assigned to you.

[Taragolis]

Test on real Snowflake: in COPY INTO instruction all ' in PATTERN should be replaced by two single quotes as well as in FILES