* Remove gauge scheduler.tasks.running

* Add significant.rst file

* Update newsfragments/30374.significant.rst

---------

Co-authored-by: Niko Oliveira <[email protected]>

…atest `resource_version` (apache#30425)

* Recover from `too old resource version exception` by retrieving the latest `resource_version`

* Update airflow/executors/kubernetes_executor.py

Co-authored-by: Tzu-ping Chung <[email protected]>

---------

Co-authored-by: Tzu-ping Chung <[email protected]>

Co-authored-by: eladkal <[email protected]>

* readding after borked it

* pre-commit

* finally fixing after the github issue last week

* push fix

* feedback from hussein

The new moto (4.1.7) performs additional validation on the queues
created during tests and it failes the tests when content
deduplication is not specified.

Explicit setting the deduplication mode, fixes the problem and
allows the new moto to be installed.

* fix possible race condition when refreshing DAGs

* merge the two queries into one

* Remove provide_session from internal function

Since get_latest_version_hash_and_updated_datetime is internal and we
always pass in the session anyway, the provide_session decorator is
redundant and only introduce possibility for developer errors.

---------

Co-authored-by: Sébastien Brochet <[email protected]>
Co-authored-by: Tzu-ping Chung <[email protected]>

Hussein is now a commiter and Norm has completed building out the
initial AIP-52 tasks.

* Turn the package 'mysql-connector-python' as an optional feature

* Update airflow/providers/mysql/provider.yaml

* Update airflow/providers/mysql/CHANGELOG.rst

---------

Co-authored-by: eladkal <[email protected]>

…pache#30629)

When compatibility check is performed for old version of Airflow,
we attempt to install all providers for the old version. However if
one of the providers has >= limit on Airflow for newer version of
Airflow, this installation lead to attempting to upgrade airflow
rather than fail, which could lead to misleading errors.

This change adds "airflow==x.y.z" taken from --use-airflow-version
flag to the `pip install` command, which should in this case fail
with much more accurate message, that the provider conflicts with
airflo version.

…edirect error (apache#30631)

Updating the links to the Dataform product documentation to fix 404 redirect error

…er to validate certificate (apache#30309) (apache#30310)

* add `tls_verify` to choose whether to validate certificate (apache#30309)

---------

Co-authored-by: Hussein Awala <[email protected]>

* add max_active_tis_per_dagrun param to BaseOperator

* set has_task_concurrency_limits when max_active_tis_per_dagrun is not None

* check if max_active_tis_per_dagrun is reached in the task deps

* check if all the tasks have None max_active_tis_per_dagrun before auto schedule the dagrun

* check if the max_active_tis_per_dagrun is reached before queuing the ti

* check max_active_tis_per_dagrun in backfill job

* fix current tests and ensure everything is ok before adding new tests

* refacto TestTaskConcurrencyDep

* fix a bug in TaskConcurrencyDep

* test max_active_tis_per_dagrun in TaskConcurrencyDep

* tests max_active_tis_per_dagrun in TestTaskInstance

* test dag_file_processor with max_active_tis_per_dagrun

* test scheduling with max_active_tis_per_dagrun on different DAG runs

* test scheduling mapped task with max_active_tis_per_dagrun

* test max_active_tis_per_dagrun with backfill CLI

* add new starved_tasks filter to avoid affecting the scheduling perf

* unify the usage of TaskInstance filters and use TI

* refacto concurrecy map type and create a new dataclass

* move docstring to ConcurrencyMap class and create a method for default_factory

* move concurrency_map creation to ConcurrencyMap class

* replace default dicts by counters

* replace all default dicts by counters in the scheduler_job_runner module

* suggestions from review

…timeout (apache#30375)

* simplify and consolidate logic for tasks stuck in queued

* simplify and consolidate logic for tasks stuck in queued

* simplify and consolidate logic for tasks stuck in queued

* fixed tests; updated fail stuck tasks to use run_with_db_retries

* mypy; fixed tests

* fix task_adoption_timeout in celery integration test

* addressing comments

* remove useless print

* fix typo

* move failure logic to executor

* fix scheduler job test

* adjustments for new scheduler job

* appeasing static checks

* fix test for new scheduler job paradigm

* Updating docs for deprecations

* news & small changes

* news & small changes

* Update newsfragments/30375.significant.rst

Co-authored-by: Jed Cunningham <[email protected]>

* Update newsfragments/30375.significant.rst

Co-authored-by: Jed Cunningham <[email protected]>

* added cleanup stuck task functionality to base executor

* fix sloppy mistakes & mypy

* removing self.fail from base_executor

* Update airflow/jobs/scheduler_job_runner.py

Co-authored-by: Jed Cunningham <[email protected]>

* Update airflow/jobs/scheduler_job_runner.py

Co-authored-by: Jed Cunningham <[email protected]>

* Fix job_id filter

* Don't even run query if executor doesn't support timing out queued tasks

* Add support for LocalKubernetesExecutor and CeleryKubernetesExecutor

* Add config option to control how often it runs - we want it quicker than
the timeout

* Fixup newsfragment

* mark old KE pending pod check interval as deprecated by new check interval

* Fixup deprecation warnings

This more closely mirrors how deprecations are raised for "normal"
deprecations.

I've removed the depth, as moving up the stack doesn't really help the
user at all in this situation.

* Another deprecation cleanup

* Remove db retries

* Fix test

---------

Co-authored-by: Jed Cunningham <[email protected]>
Co-authored-by: Jed Cunningham <[email protected]>
Co-authored-by: Ephraim Anierobi <[email protected]>

* Display Video 360 cleanup v1 API usage

* Update docs

…apache#29913)

* Add a failing test to make it pass

* use partial_kwargs when they are provide and override only None values by dag default values

* update the test and check if the values are filled in the right order

* fix overriding retry_delay with default value when it is equal to 0

* add missing default value for inlets and outlets

* set partial_kwargs dict type to dict[str, Any] and remove type ignore comments

* create a dict for default values and use NotSet instead of None to support None as accepted value

* update partial typing by removing None type from some args and set NotSet for all args

* Tweak kwarg merging slightly

This should improve iteration a bit, I think.

* Fix unit tests

---------

Co-authored-by: Tzu-ping Chung <[email protected]>

This PR consistent mostly of code that was created in OpenLineage project. It 
consists of

- Provider wiring
- OpenLineageListener that uses Listener API to get notification about changes
  to TaskInstance and Dag states
- Extractor framework, which is used to extract lineage information from
  particular operators. It's ment to be replaced by direct implementation of
  lineage features in later phase and extracting them using DefaultExtractor.
  This PR does not include actual extractors, but code around using and registering them.
- OpenLineageAdapter that translates extracted information to OpenLineage events.
- Utils around specific Airflow OL facets and features

This is a base implementation that's not ment to be released yet, but to add
code modified to be consistent with Airflow standards, get early feedback and
provide canvas to add later features, docs, tests on.

Signed-off-by: Jakub Dardzinski <[email protected]>

…he#30640)

…pache#29809)

* Adding configuration to control retry parameters for k8s api client

* Handling review comments

* Fixing code bug

* Fixing failing tests

* Temporary commit with UT wip

* Fixing unit test

* Fixing the strict checks

* Handling review comments from Hussein

* Revert "Handling review comments from Hussein"

This reverts commit fa3bc26.

* Fixing failing ut

* Reverting bad hack

* Updating logic in kube_client.py

Co-authored-by: Hussein Awala <[email protected]>

* Fixing unit tests

* Fixing unit tests

* Handling review comments from Ash

* Fix loading mock call args for python3.7

* Apply suggestions from code review

* fix static check

* add in 2.6.0

---------

Co-authored-by: Amogh <[email protected]>
Co-authored-by: Hussein Awala <[email protected]>

* fix(chart): webserver probes timeout and period

* Update default values in JSON schema to reflect values.yaml

* remove defautl templated values

…ype` (apache#30595)

* Add  while initilizing

* Add  while initilizing

* Add logic to pick either client_type or resource_type

* Add test case

* Assert expected path

* update run clear+mark, update task clear

* add mark as tasks and include list of affected tasks

* Add support for mapped tasks, add shared modal component

* Clean up styling, restore warning for past/future tg clear

---------

Co-authored-by: Hussein Awala <[email protected]>
Co-authored-by: Hussein Awala <[email protected]>

…apache#29065)

* When clearing task instances try to get associated DAGs from database.

This fixes problems when recursively clearing task instances across multiple DAGs:
  * Task instances in downstream DAGs weren't having their `max_tries` property incremented, which could cause downstream external task sensors in reschedule mode to instantly time out (issue apache#29049).
  * Task instances in downstream DAGs could have some of their properties overridden by an unrelated task in the upstream DAG if they had the same task ID.

* Use session fixture for new `test_clear_task_instances_without_dag_param` test.

* Use session fixture for new `test_clear_task_instances_in_multiple_dags` test.

---------

Co-authored-by: eladkal <[email protected]>

---------

Co-authored-by: Tzu-ping Chung <[email protected]>
Co-authored-by: Ephraim Anierobi <[email protected]>

…30646)

---------

Co-authored-by: Josh Fell <[email protected]>

)

We've recently added google_venor package to vendor-in ads library,
and we had to do it outside of regular google provider package,
because internally the library assumed our google package is top
level package when discovering the right relative imports (apache#30544).

This confused the pre-commit that updates provider depedencies to
not recognise the package and print warnings about bad classification.

Special case handling will classify it to google provider.

The workday calendar expected pandas to be available and it is part
of our examples, however Airflow does not have pandas as a core
dependency, so in case someone does not have pandas installed, importing
of the workday example would fail.

This change makes pandas optional and fallbacks to regular working
days for the example in case it is not available (including warning
about it). It also fixes a slight inefficiency where the
USFederalHoliday calendar has been created every time next workday
was calculated.

Co-authored-by: eladkal <[email protected]>

…pache#29000)

* Skip KubernetesPodOperator task when it returns a provided exit code

* set default value to None, and get exit code only when skip_exit_code is not None

* get the exit code for the base container and check if everything is ok

* add unit test for the operator

* add a test for deffered mode

* apply change requests

---------

Co-authored-by: eladkal <[email protected]>

The apache#30660 was merged to quickly as it results in doc building
failure. This PR fixes it.

Pytest collection has been extracted recently to a separate job
and SUSPENDED_PROVIDERS_FOLDERS variable was not set in the new
job - which causes suspended provider tests being attempted by
pytest collection, leading to import errors when suspended providers
have some dependencies removed from our image.

…#30674)

The new attrs released today (11 hours ago) had added typing
information and they caused OpenLineageRedactor to fail mypy checks.

Temporary adding type: ignore should allow to upgrade to the new
attrs and stop PRs changing dependencies from failing.

Related: apache#30673

* Update the release note

During the beta release, I observed some minor things that need fixing.
Here's the PR

* Use local import

apache#30675)

In some cases when the machine has been reused across builds, pipx
installed twine might seem both installed and removed (this happens
when builds are cancelled while installing twine.

Installing twine with --force should fix the problem.

…he#30682)

When packages for "Other" test type are calculated, the list of
all test folders is generated and they are compared with the
packages previously selected by the "predefined" test types. This
is done via `find` method that returns the folders in arbitrary
order, mostly depending on the sequence the folders were created.

In case the tests from some packages have some side-effects that
impact tests in other packages (obviously not something that is
desired), this might end up that the tests succeed in one
environment, but fail in another. This happened for example
in case of apache#30362 that had cross-package side-effect later
fixed in apache#30588. There - results of "Other" test type depended
on where the tests were executed.

This PR sorts the find output so it is always in consistent order.
we are using ASCII for package names and the test types are
derived in the same Docker CI image with the same LOCALE, so it
should guarantee that the output of packages for "Other" test type
should be always consistent.

Upgrading to latest (released a week ago) MyPy in the hopes it
will fix some more problem with attrs after upgrading new packages,
but it seems that even the latest MyPy does not know about the
new typing changes introduced in attrs (traditionally mypy has
attrs plugin that injects appropriate typing but apparently it
needs to catch up with those changes.

Helm Unit tests are using template rendering and the rendering
uses a lot of CPU for `helm template command`. We have a lot of
those rendering tests (>800) so even running the tests in parallel
on a multi-cpu machine does not lead to a decreased elapsed time
to execute the tests.

However, each of the tests is run entirely independently and we
should be able to achieve much faster elapsed time if we run
a subset of tetsts on separate, multi-CPU machine. This will not
lower the job build time, however it might speed up elapsed time
and thus get a faster feedback.

This PR achieves that.

…de (apache#30690)

* Add a new argument to rais skip exception when the python callable exit with the same value

* add unit tests for skip_exit_code

…le codes (apache#30692)

* rename skip_exit_code to skip_on_exit_code and allow providing multiple codes

* replace list type by Container

Co-authored-by: Mark Richman <[email protected]>

This change separates out the policies we have for providers to
a separate PROVIERS.rst file. It also documents clearly the process
and policy we have for accepting new community-managed providers,
explaining the conditions that have to be fulfilled and stating
a very strong preference of keeping providers maintained by the
3rd-party providers when there are 3rd-party teams that manage
the providers.