Now that Airflow 2.7.0 is released, we can remove exclusion that
we had for openlineage which prevented from using it as
dependency of Airflow in CI.

(cherry picked from commit 008f233)

The label is provider:openlineage instead of API-53

(cherry picked from commit de17b93)

We already fixed all deprecation warnings for Pydantic 2 and we can
thus remove Pydantic 2 limitation. Even if we are waiting for other
dependencies (aws-sam-translator) it should be save to remove the
limit - we will get Pydantic 2 when aws-sam-translate new version
is released in a week or two (Pydantic 2 support has been added
last week in
aws/serverless-application-model#3282)

(cherry picked from commit 754a4ab)

* Add MySQL 8.1 to supported versions.

Anticipating Lazy Consensus to be reached we add 8.1 version
of MySQL to supported versions.

* Apply suggestions from code review

(cherry picked from commit 825f65f)

We used to have problems with `pip` backtracking when we relaxed
too much open-telemetry dependencies. It turned out that the
backtracting was only happening on Python 3.8 and that it was
ultimately caused by conflict between importlib_metadata between
Airflow and newer versions of opentelemetry (we had <5 for Python
3.8, they had >6 for all versions. The reason for limiting it in
Airflow was Celery that was not working well with importlib 5.

Since Celery 5.3 solved the problems (released 6th of June) we can
now relax the importlib_metadata limit and set Celery to version >=
5.3.0) which nicely resolves the conflict and there is no more
backtracking when trying to install newer versions of opentelemetry
for Python 3.8.

Fixes: #33577
(cherry picked from commit ae25a52)

* Improve detection of when breeze CI image needs rebuilding

Previously we have been using provider.yaml file modification as
a sign that the docker image needs rebuilding when starting image.
However just modification of provider.yaml file is not a sign
that the image needs rebuilding. The image needs rebuilding when
provider dependencies changed, but there are many more reasons why
provider.yaml file changed - especially recently provider.yaml
file contains much more information and dependencies are only part
of it. Provider.yaml files can also be modified by release manager
wnen documentation is prepared, but none of the documentation
change is a reason for rebuilding the image.

This PR optimize the check for image building introducing two
step process:

* first we check if provider.yaml files changed
* if they did, we regenerate provider dependencies by manully
  running the pre-commit script
* then provider_dependencies.json is used instead of all providers
  to determine if the image needs rebuilding

This has several nice side effects:

* the list of files that have been modified displayed to the
  user is potentially much smaller (no provider.yaml files)
* provider_dependencies.json is regenereated automatically when
  you run any breeze command, which means that you do not have
  to have pre-commit installed to regenerate it
* the notification "image needs rebuilding" will be printed less
  frequently to the user - only when it is really needed
* preparing provider documentation in CI will not trigger
  image rebuilding (which might occasionally fail in such case
  especially when we bring back a provider from long suspension
  like it happened in #33574

* Update dev/breeze/src/airflow_breeze/commands/developer_commands.py

(cherry picked from commit ac0d5b3)

Botocore has a very peculiar process of releasing new version
every day, which means that it gives `pip` hard time to figure
what will be the non-conflicting set of packages when we have
too low of a minium version set as requirement.

Since we had > 1.24 that means that `pip` had to consider
more than 340 versions for botocore, but also for related
mypy packages and also a number of aiobotocore packages when
resolving eager-upgrade.

We limit all the relevant packages to 1.28 as minimum version
now, and we should continue doing that regularly in the future.

(cherry picked from commit 5f504e9)

In order to generate constraints, we need to temporarily limit
also hive provider. There is a gap between wnen we added it
in airflow setup and when we can generate constraints for
the released providers from PyPI - we need to release the provider
similarly like we have to do it for yandex.

Therefore - until the upcoming hive provider is released (in 3 days)
we need to limit hive from being considered in Python 3.11 consstraint
generation for providers from PyPI

(cherry picked from commit 984ba22)

Redis 5 relased last week breaks celery, celery is limiting it for
now and will resolve it later, we should similarly limit redis on
our side to limit redis for users who will not upgrade to celery
that will be released shortly.

Fixes: #33744
(cherry picked from commit 3ba994d)

There were still some left-overs of EAGER_UPGRADE in PROD image
building. Howwever "eager upgrade" only makes sense for CI images.
PROD images when being built should use eager upgrades as they
are produced in the CI image step.

This PR does the following:

* removes eager upgrade parameters from PROD image
* instead, prod image build has a new flag for installing
  the images: --use-constraints-for-context-packages which will
  automatically use constraints from "docker-context-files" if
  they are present there.
* modifies the CI workflows to upload constraints as artifacts
  and download them for PROD image build when "eager upgrade"
  has been used and directs it to use "source" constraints
* adds back support to "upgrade to newer dependencies" label
  that makes it easy to test "eager upgrade"

As the result, when PROD image is build in CI:

* when regular PR is run, it will use latest github "source" constraints
* whwn "eager upgrade" PR is run, it will use the eager-upgrade
  constraints that were generated during CI build

(cherry picked from commit 2b1a194)

When we are building PROD image in CI for non main branch, we are
installing providers from PyPI rather than building them locally
from sources. Therefore we should use `PyPI` constraints for
such builds not the "source" constraints (they might differ).

This PR adds two steps:

* In the CI build, when we do not build providers we generate
  PyPI constraints additionally to source constraints
* In the PROD build we use the PyPI constraints in case we
  do not build providers locally

(cherry picked from commit f9276f0)

…#32272)

* Fix rendering the mapped parameters in the mapped operator

Signed-off-by: Hussein Awala <[email protected]>

* add template_in_template arg to expand method to tell Airflow whether to resolve the xcom data or not

* fix dag serialization tests

* Revert "fix dag serialization tests"

This reverts commit 191351c.

* Revert "add template_in_template arg to expand method to tell Airflow whether to resolve the xcom data or not"

This reverts commit 14bd392.

* Fix ListOfDictsExpandInput resolve method

* remove _iter_parse_time_resolved_kwargs method

* remove unnecessary step

---------

Signed-off-by: Hussein Awala <[email protected]>
(cherry picked from commit d1e6a5c)

Update airflow/providers/apache/hive/CHANGELOG.rst

Co-authored-by: Tzu-ping Chung <[email protected]>
(cherry picked from commit 08188f8)

…ble schema before adding (#32731)

* Check if the index is in the table schema before adding

* add pre-condition assertion

* static checks

* Update test_models.py

* integrate upstream auth manager changes

(cherry picked from commit 2950fd7)

When a PR is referenced by other PRs, our dev tool for getting the correct
commit lists the latest commit when looking for the commmit sha but we should get the oldest.

(cherry picked from commit 5b104a9)

…3418)

This is a more complete fix to #33411. This is also a follow up on
earlier implementation of #33261 that addressed checking if PRs
are merged. This one applies the same pattern to finding commit
but also improves it by checking if the (#NNNNNN) ends the subject
- so even if the PR is in the same form in the message, it will be
filtered out.

The previous "--reverse" quick fix in #33411 had potential of problem in
case there were releated PRs merged before the original PR (which is
quite posssible when you have a series of PRs referring to each other.

(cherry picked from commit 3766ab0)

(cherry picked from commit 489ca14)

(cherry picked from commit 21656fc)

… mode (#33196)

(cherry picked from commit a1b5bdb)

* Fix pydantic warning about `orm_mode` rename

Pydantic 2 renamed orm_mode to from_attributes. This was missed during the upgrade to pydantic 2 and it gives excessive warning about the rename.
This PR fixes it

* Also rename from_orm to model_validate and use model_dump instead of dict

* Fix Pydantic 1.x compatibility

---------

Co-authored-by: Tzu-ping Chung <[email protected]>
(cherry picked from commit 75bb04b)

(cherry picked from commit 01a6c1e)

(cherry picked from commit 63c5df8)

(cherry picked from commit 95e9d83)

(cherry picked from commit cc8519d)

(cherry picked from commit 369b9bc)

(cherry picked from commit 50a6385)

(cherry picked from commit d747a79)

By going up to `parents[3]` we were going outside the repository root,
luckily(or unluckily the repo folder is also named `airflow` so the
pattern `airflow/**/example_dags/example_*.py` still worked,
but `tests/system/providers/**/example_*.py` wasn't being used.

This discovered 2 new errors:

- `example_local_to_wasb.py` was trivial to fix

- `example_redis_publish.py`is more interesting: this one fails because
`RedisPubSubSensor` constructor calls Redis.pubsub().subscribe(), which
just hangs and DagBag fails with timeout. For now I'm just deleting this
operator from the example.

(cherry picked from commit c048bd5)

(cherry picked from commit d2c0bbe)

* D205 Support - Root files

Updates setup.py, airflow/configuration.py, and airflow/exceptions.py

* missed one

(cherry picked from commit 4755fe4)

* D205 Support - WWW

* fix broken link

(cherry picked from commit 64c2a56)

(cherry picked from commit 44a752a)

(cherry picked from commit 9bf68ad)

(cherry picked from commit 23d5424)

(cherry picked from commit b657ae9)

(cherry picked from commit 3c48dc7)

(cherry picked from commit 396fd3c)

(cherry picked from commit d0c94d6)

* D401 Support - airflow/macros thru airflow/operators

* fix static checks

(cherry picked from commit 2efb3a6)

(cherry picked from commit 8279628)

(cherry picked from commit 5e1e5fa)

…bdir`` (#33357)

* Fix standalone DagProcessor interfering with DAG outsite of its subdir

* Add tests

* Update code review

(cherry picked from commit 35b1830)

(cherry picked from commit 08565dc)

(cherry picked from commit 7700fb1)

…cific logic for operators (#33424)

* Create a generic method used to resume the task in order to implement specific logic for each operator

* Provide resume_execution args

* add a unit test

* Fix soft fail after a trigger timeout

* wrap the exception with a str to get the message

* Apply suggestions from code review

Co-authored-by: Tzu-ping Chung <[email protected]>

---------

Co-authored-by: Tzu-ping Chung <[email protected]>
(cherry picked from commit 852f85b)

* feat(sensors/base): raise AirflowSkipException if soft_fail is set to True and exception occurs after running poke()

* test(sensor/base): add test case for respecting soft_fail option when other kinds of exception is raised

(cherry picked from commit d91c481)

(cherry picked from commit a6299d4)

(cherry picked from commit bfe08a7)

* make `conf.set` case insensitive

`conf.get` is insensitive (it converts section and key to lower case)
but set is not, which can lead to surprising behavior
(see the test, which is not passing without the fix).
I suggest that we override set as well to fix that.
Any value that was set before with upper case was unreacheable.

* fix remove_option as well

* away with the str()

* add significant change newsfragment

(cherry picked from commit abbd567)

(cherry picked from commit 2ae6d4b)

…ferrable mode (#33196)" (#33458)

This reverts commit a1b5bdb.

(cherry picked from commit 85a5677)

(cherry picked from commit 5ee1bcb)

(cherry picked from commit 996d8c5)

Observed a couple of links are broken on our pypi project listing
https://pypi.org/project/apache-airflow/. They work fine when
viewed on GitHub but those relative links don't work fine on the
pypi index. Also, applied a few quick-fixes that my PyCharm
suggested for the README.md file

(cherry picked from commit 5a37025)

(cherry picked from commit bcefe61)

(cherry picked from commit 8e88eb8)

(cherry picked from commit eed09da)

(cherry picked from commit 7c51c87)

fixing broken link in Modules Management

(cherry picked from commit 02d3378)

* Use `dialect.name` in custom SA types

* Fix removed import

(cherry picked from commit 46aa429)

The otel was 2.6.0 as can be seen here: #30160 and
the internal api and related were 2.6.0 even though they were under a flag. These need to be updated
except we want to modify the check tool to skip these ones

(cherry picked from commit ba4f541)

`db updgrade` to `db upgrade`

(cherry picked from commit bfba17b)

Co-authored-by: Tzu-ping Chung <[email protected]>

---------

Co-authored-by: Tzu-ping Chung <[email protected]>
(cherry picked from commit 4390524)

(cherry picked from commit f89af21)

…33516)

(cherry picked from commit d9814eb)

(cherry picked from commit e69aae1)

(cherry picked from commit 79b8cfc)

(cherry picked from commit 62b917a)

(cherry picked from commit 911cf46)

(cherry picked from commit 370348a)

* Set context inside templates

---------

Co-authored-by: Ivan Afonichkin <[email protected]>
Co-authored-by: Tzu-ping Chung <[email protected]>
(cherry picked from commit 9fa782f)

* Fix circular import from kubernetes_executor

* Do not validate SQL when initialising Sentry

---------

Co-authored-by: Lipu Fei <[email protected]>
(cherry picked from commit 4bdf908)

* Add back `get_url_for_login` in security manager

* Rename FAB_SECURITY_MANAGER_CLASS to SECURITY_MANAGER_CLASS

(cherry picked from commit b1cdab3)

…33690)

(cherry picked from commit f971ba2)

* Fix cleaning zombie RESTARTING tasks

* Fix test

* Improve naming

Renamed to adoptable_states because adoption is preferrable choice and reset is fallback.
Also "resettable" might be confused with RESTARTING itself.

---------

Co-authored-by: daniel.dylag <[email protected]>
(cherry picked from commit 5c35786)

…33735)

* Add documenation explaining template_ext (and how to override it)

* fix static checks

(cherry picked from commit a5aa161)

(cherry picked from commit 272b40a)

(cherry picked from commit 64948fa)

(cherry picked from commit 3b313e2)

(cherry picked from commit 83bd60f)

(cherry picked from commit a91ee7a)

(cherry picked from commit 1cdd823)

(cherry picked from commit 056f167)

(cherry picked from commit 95a930b)

…ublic hook methods (#33039)

* Validate SqoopHook connection string and disable extra options from hook methods

Check that the connection string constructed using the connection's
`host`, `port` and `schema` does not contain query params as it is not
intended. Additionally, also disable the `extra_import_options` and
`extra_export_options` arguments accepted directly by the hook
methods but accept it as a param via the hook constructor when
initialising the hook or by passing it in hook_params when initialising
the hook from operators.

* Propogate missing hook param changes to the operator

* Remove test for invalid port with query param as ports are integers and not accepted by databases

* Add 4.0.0 to provider.yaml

(cherry picked from commit 59f5f58)

(cherry picked from commit 63e6eab)

#33393)

Signed-off-by: Maciej Obuchowski <[email protected]>
(cherry picked from commit 8e738cd)

(cherry picked from commit abef61f)

(cherry picked from commit a1e6cd4)