Apache Airflow Helm Chart 1.8.0

Significant Changes

bitnami/postgresql subchart updated to 12.1.9 (#29071)

The version of postgresql installed is still version 11.

If you are upgrading an existing helm release with the built-in postgres database, you will either need to delete your release and reinstall fresh, or manually delete these 2 objects:

    kubectl delete secret {RELEASE_NAME}-postgresql
    kubectl delete statefulset {RELEASE_NAME}-postgresql

As a reminder, it is recommended to set up an external database <https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#database>_ in production.

This version of the chart uses different variable names for setting usernames and passwords in the postgres database.

• postgresql.auth.enablePostgresUser is used to determine if the "postgres" admin account will be created.
• postgresql.auth.postgresPassword sets the password for the "postgres" user.
• postgresql.auth.username and postrgesql.auth.password are used to set credentials for a non-admin account if desired.
• postgresql.postgresqlUsername and postgresql.postresqlPassword, which were used in the previous version of the chart, are no longer used.

Users will need to make those changes in their values files if they are changing the Postgres configuration.

Previously the subchart version was 10.5.3.

Default dags.gitSync.wait reduced to 5 seconds (#27625)

The default for dags.gitSync.wait has been reduced from 60 seconds to 5 seconds to reduce the likelihood of DAGs
becoming inconsistent between Airflow components. This will, however, increase traffic to the remote git repository.

Default Airflow image is updated to 2.5.1 (#29074)

The default Airflow image that is used with the Chart is now 2.5.1, previously it was 2.4.1.

Default git-sync image is updated to 3.6.3 (#27848)

The default git-sync image that is used with the Chart is now 3.6.3, previously it was 3.4.0.

Default redis image is updated to 7-bullseye (#27443)

The default redis image that is used with the Chart is now 7-bullseye, previously it was 6-bullseye.

New Features

• Add annotations on deployments (#28688)
• Add global volume & volumeMounts to the chart (#27781)

Improvements

• Add support for webserverConfigConfigMapName (#27419)
• Enhance chart to allow overriding command-line args to statsd exporter (#28041)
• Add support for NodePort in Services (#26945)
• Add worker log-groomer-sidecar enable option (#27178)
• Add HostAliases to Pod template file (#27544)
• Allow PgBouncer replicas to be configurable (#27439)

Bug Fixes

• Create scheduler service to serve task logs for LocalKubernetesExecutor (#28828)
• Fix NOTES.txt to show correct URL (#28264)
• Add worker service account for LocalKubernetesExecutor (#28813)
• Remove checks for 1.19 api checks (#28461)
• Add airflow_local_settings to all airflow containers (#27779)
• Make custom env vars optional for job templates (#27148)
• Decrease default gitSync wait (#27625)
• Add extraVolumeMounts to sidecars too (#27420)
• Fix PgBouncer after PostgreSQL subchart upgrade (#29207)

Doc only changes

• Enhance production guide with a few Argo specific guidelines (#29078)
• Add doc note about Pod template images (#29032)
• Update production guide db section (#28610)
• Fix to LoadBalancer snippet (#28014)
• Fix gitSync example code (#28083)
• Correct repo example for cloning via ssh (#27671)

Misc

• Update Airflow version to 2.5.1 (#29074)
• Update git-sync to 3.6.3 (#27848)
• Upgrade bitnami/postgresql subchart to 12.1.9 (#29071)
• Update redis to 7 (#27443)
• Replace helm chart icon (#27704)

Apache Airflow 2.5.1

Bug Fixes

• Fix masking of non-sensitive environment variables (#28802)
• Remove swagger-ui extra from connexion and install swagger-ui-dist via npm package (#28788)
• Fix UIAlert should_show when AUTH_ROLE_PUBLIC set (#28781)
• Only patch single label when adopting pod (#28776)
• Update CSRF token to expire with session (#28730)
• Fix "airflow tasks render" cli command for mapped task instances (#28698)
• Allow XComArgs for external_task_ids of ExternalTaskSensor (#28692)
• Row-lock TIs to be removed during mapped task expansion (#28689)
• Handle ConnectionReset exception in Executor cleanup (#28685)
• Fix description of output redirection for access_log for gunicorn (#28672)
• Add back join to zombie query that was dropped in #28198 (#28544)
• Fix calendar view for CronTriggerTimeTable dags (#28411)
• After running the DAG the employees table is empty. (#28353)
• Fix DetachedInstanceError when finding zombies in Dag Parsing process (#28198)
• Nest header blocks in divs to fix dagid copy nit on dag.html (#28643)
• Fix UI caret direction (#28624)
• Guard not-yet-expanded ti in trigger rule dep (#28592)
• Move TI setNote endpoints under TaskInstance in OpenAPI (#28566)
• Consider previous run in CronTriggerTimetable (#28532)
• Ensure correct log dir in file task handler (#28477)
• Fix bad pods pickled in executor_config (#28454)
• Add ensure_ascii=False in trigger dag run API (#28451)
• Add setters to MappedOperator on_*_callbacks (#28313)
• Fix ti._try_number for deferred and up_for_reschedule tasks (#26993)
• separate callModal from dag.js (#28410)
• A manual run can't look like a scheduled one (#28397)
• Dont show task/run durations when there is no start_date (#28395)
• Maintain manual scroll position in task logs (#28386)
• Correctly select a mapped task's "previous" task (#28379)
• Trigger gevent monkeypatching via environment variable (#28283)
• Fix db clean warnings (#28243)
• Make arguments 'offset' and 'length' not required (#28234)
• Make live logs reading work for "other" k8s executors (#28213)
• Add custom pickling hooks to LazyXComAccess (#28191)
• fix next run datasets error (#28165)
• Ensure that warnings from @dag decorator are reported in dag file (#28153)
• Do not warn when airflow dags tests command is used (#28138)
• Ensure the dagbag_size metric decreases when files are deleted (#28135)
• Improve run/task grid view actions (#28130)
• Make BaseJob.most_recent_job favor "running" jobs (#28119)
• Don't emit FutureWarning when code not calling old key (#28109)
• Add airflow.api.auth.backend.session to backend sessions in compose (#28094)
• Resolve false warning about calling conf.get on moved item (#28075)
• Return list of tasks that will be changed (#28066)
• Handle bad zip files nicely when parsing DAGs. (#28011)
• Prevent double loading of providers from local paths (#27988)
• Fix deadlock when chaining multiple empty mapped tasks (#27964)
• fix: current_state method on TaskInstance doesn't filter by map_index (#27898)
• Don't log CLI actions if db not initialized (#27851)
• Make sure we can get out of a faulty scheduler state (#27834)
• dagrun, next_dagruns_to_examine, add MySQL index hint (#27821)
• Handle DAG disappearing mid-flight when dag verification happens (#27720)
• fix: continue checking sla (#26968)
• Allow generation of connection URI to work when no conn type (#26765)

Misc/Internal

• Remove limit for dnspython after eventlet got fixed (#29004)
• Limit dnspython to < 2.3.0 until eventlet incompatibility is solved (#28962)
• Add automated version replacement in example dag indexes (#28090)
• Cleanup and do housekeeping with plugin examples (#28537)
• Limit SQLAlchemy to below 2.0 (#28725)
• Bump json5 from 1.0.1 to 1.0.2 in /airflow/www (#28715)
• Fix some docs on using sensors with taskflow (#28708)
• Change Architecture and OperatingSystem classes into Enums (#28627)
• Add doc-strings and small improvement to email util (#28634)
• Fix Connection.get_extra type (#28594)
• navbar, cap dropdown size, and add scroll bar (#28561)
• Emit warnings for conf.get* from the right source location (#28543)
• Move MyPY plugins of ours to dev folder (#28498)
• Add retry to purge_inactive_dag_warnings (#28481)
• Re-enable Plyvel on ARM as it now builds cleanly (#28443)
• Add SIGUSR2 handler for LocalTaskJob and workers to aid debugging (#28309)
• Convert test_task_command to Pytest and unquarantine tests in it (#28247)
• Make invalid characters exception more readable (#28181)
• Bump decode-uri-component from 0.2.0 to 0.2.2 in /airflow/www (#28080)
• Use asserts instead of exceptions for executor not started (#28019)
• Simplify dataset subgraph logic (#27987)
• Order TIs by map_index (#27904)
• Additional info about Segmentation Fault in LocalTaskJob (#27381)

Doc Only Changes

• Mention mapped operator in cluster policy doc (#28885)
• Slightly improve description of Dynamic DAG generation preamble (#28650)
• Restructure Docs (#27235)
• Update scheduler docs about low priority tasks (#28831)
• Clarify that versioned constraints are fixed at release time (#28762)
• Clarify about docker compose (#28729)
• Adding an example dag for dynamic task mapping (#28325)
• Use docker compose v2 command (#28605)
• Add AIRFLOW_PROJ_DIR to docker-compose example (#28517)
• Remove outdated Optional Provider Feature outdated documentation (#28506)
• Add documentation for [core] mp_start_method config (#27993)
• Documentation for the LocalTaskJob return code counter (#27972)
• Note which versions of Python are supported (#27798)

Apache Airflow 2.5.0

New Features

• TaskRunner: notify of component start and finish (#27855)
• Add DagRun state change to the Listener plugin system(#27113)
• Metric for raw task return codes (#27155)
• Add logic for XComArg to pull specific map indexes (#27771)
• Clear TaskGroup (#26658, #28003)
• Add critical section query duration metric (#27700)
• Add: #23880 :: Audit log for AirflowModelViews(Variables/Connection) (#24079, #27994, #27923)
• Add postgres 15 support (#27444)
• Expand tasks in mapped group at run time (#27491)
• reset commits, clean submodules (#27560)
• scheduler_job, add metric for scheduler loop timer (#27605)
• Allow datasets to be used in taskflow (#27540)
• Add expanded_ti_count to ti context (#27680)
• Add user comment to task instance and dag run (#26457, #27849, #27867)
• Enable copying DagRun JSON to clipboard (#27639)
• Implement extra controls for SLAs (#27557)
• add dag parsed time in DAG view (#27573)
• Add max_wait for exponential_backoff in BaseSensor (#27597)
• Expand tasks in mapped group at parse time (#27158)
• Add disable retry flag on backfill (#23829)
• Adding sensor decorator (#22562)
• Api endpoint update ti (#26165)
• Filtering datasets by recent update events (#26942)
• Support Is /not Null filter for value is None on webui (#26584)
• Add search to datasets list (#26893)
• Split out and handle 'params' in mapped operator (#26100)
• Add authoring API for TaskGroup mapping (#26844)
• Add one_done trigger rule (#26146)
• Create a more efficient airflow dag test command that also has better local logging (#26400)
• Support add/remove permissions to roles commands (#26338)
• Auto tail file logs in Web UI (#26169)
• Add triggerer info to task instance in API (#26249)
• Flag to deserialize value on custom XCom backend (#26343)

Improvements

• Allow depth-first execution (#27827)
• UI: Update offset height if data changes (#27865)
• Improve TriggerRuleDep typing and readability (#27810)
• Make views requiring session, keyword only args (#27790)
• Optimize TI.xcom_pull() with explicit task_ids and map_indexes (#27699)
• Allow hyphens in pod id used by k8s executor (#27737)
• optimise task instances filtering (#27102)
• Use context managers to simplify log serve management (#27756)
• Fix formatting leftovers (#27750)
• Improve task deadlock messaging (#27734)
• Improve "sensor timeout" messaging (#27733)
• Replace urlparse with urlsplit (#27389)
• Align TaskGroup semantics to AbstractOperator (#27723)
• Add new files to parsing queue on every loop of dag processing (#27060)
• Make Kubernetes Executor & Scheduler resilient to error during PMH execution (#27611)
• Separate dataset deps into individual graphs (#27356)
• Use log.exception where more economical than log.error (#27517)
• Move validation branch_task_ids into SkipMixin (#27434)
• Coerce LazyXComAccess to list when pushed to XCom (#27251)
• Update cluster-policies.rst docs (#27362)
• Add warning if connection type already registered within the provider (#27520)
• Activate debug logging in commands with --verbose option (#27447)
• Add classic examples for Python Operators (#27403)
• change .first() to .scalar() (#27323)
• Improve reset_dag_run description (#26755)
• Add examples and howtos about sensors (#27333)
• Make grid view widths adjustable (#27273)
• Sorting plugins custom menu links by category before name (#27152)
• Simplify DagRun.verify_integrity (#26894)
• Add mapped task group info to serialization (#27027)
• Correct the JSON style used for Run config in Grid View (#27119)
• No extra__conn_type__ prefix required for UI behaviors (#26995)
• Improve dataset update blurb (#26878)
• Rename kubernetes config section to kubernetes_executor (#26873)
• decode params for dataset searches (#26941)
• Get rid of the DAGRun details page & rely completely on Grid (#26837)
• Fix scheduler crashloopbackoff when using hostname_callable (#24999)
• Reduce log verbosity in KubernetesExecutor. (#26582)
• Don't iterate tis list twice for no reason (#26740)
• Clearer code for PodGenerator.deserialize_model_file (#26641)
• Don't import kubernetes unless you have a V1Pod (#26496)
• Add updated_at column to DagRun and Ti tables (#26252)
• Move the deserialization of custom XCom Backend to 2.4.0 (#26392)
• Avoid calculating all elements when one item is needed (#26377)
• Add __future__.annotations automatically by isort (#26383)
• Handle list when serializing expand_kwargs (#26369)
• Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290)
• Add more weekday operator and sensor examples #26071 (#26098)
• Align TaskGroup semantics to AbstractOperator (#27723)

Bug Fixes

• Gracefully handle whole config sections being renamed (#28008)
• Add allow list for imports during deserialization (#27887)
• Soft delete datasets that are no longer referenced in DAG schedules or task outlets (#27828)
• Redirect to home view when there are no valid tags in the URL (#25715)
• Refresh next run datasets info in dags view (#27839)
• Make MappedTaskGroup depend on its expand inputs (#27876)
• Make DagRun state updates for paused DAGs faster (#27725)
• Don't explicitly set include_examples to False on task run command (#27813)
• Fix menu border color (#27789)
• Fix backfill queued task getting reset to scheduled state. (#23720)
• Fix clearing child dag mapped tasks from parent dag (#27501)
• Handle json encoding of V1Pod in task callback (#27609)
• Fix ExternalTaskSensor can't check zipped dag (#27056)
• Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)
• Continue on exception when retrieving metadata (#27665)
• External task sensor fail fix (#27190)
• Add the default None when pop actions (#27537)
• Display parameter values from serialized dag in trigger dag view. (#27482, #27944)
• Move TriggerDagRun conf check to execute (#27035)
• Resolve trigger assignment race condition (#27072)
• Update google_analytics.html (#27226)
• Fix some bug in web ui dags list page (auto-refresh & jump search null state) (#27141)
• Fixed broken URL for docker-compose.yaml (#26721)
• Fix xcom arg.py .zip bug (#26636)
• Fix 404 taskInstance errors and split into two tables (#26575)
• Fix browser warning of improper thread usage (#26551)
• template rendering issue fix (#26390)
• Clear autoregistered DAGs if there are any import errors (#26398)
• Fix from airflow import version lazy import (#26239)
• allow scroll in triggered dag runs modal (#27965)

Misc/Internal

• Remove is_mapped attribute (#27881)
• Simplify FAB table resetting (#27869)
• Fix old-style typing in Base Sensor (#27871)
• Switch (back) to late imports (#27730)
• Completed D400 for multiple folders (#27748)
• simplify notes accordion test (#27757)
• completed D400 for airflow/callbacks/* airflow/cli/* (#27721)
• Completed D400 for airflow/api_connexion/* directory (#27718)
• Completed D400 for airflow/listener/* directory (#27731)
• Completed D400 for airflow/lineage/* directory (#27732)
• Update API & Python Client versions (#27642)
• Completed D400 & D401 for airflow/api/* directory (#27716)
• Completed D400 for multiple folders (#27722)
• Bump minimatch from 3.0.4 to 3.0.8 in /airflow/www (#27688)
• Bump loader-utils from 1.4.1 to 1.4.2 in /airflow/www (#27697)
• Disable nested task mapping for now (#27681)
• bump alembic minimum version (#27629)
• remove unused code.html (#27585)
• Enable python string normalization everywhere (#27588)
• Upgrade dependencies in order to avoid backtracking (#27531)
• Strengthen a bit and clarify importance of triaging issues (#27262)
• Deduplicate type hints (#27508)
• Add stub 'yield' to BaseTrigger.run (#27416)
• Remove upper-bound limit to dask (#27415)
• Limit Dask to under 2022.10.1 (#27383)
• Update old style typing (#26872)
• Enable string normalization for docs (#27269)
• Slightly faster up/downgrade tests (#26939)
• Deprecate use of core get_kube_client in PodManager (#26848)
• Add memray files to gitignore / dockerignore (#27001)
• Bump sphinx and sphinx-autoapi (#26743)
• Simplify RTIF.delete_old_records() (#26667)
• migrate last react files to typescript (#26112)
• Work around pyupgrade edge cases (#26384)

Doc only changes

• Document dag_file_processor_timeouts metric as deprecated (#27067)
• Drop support for PostgreSQL 10 (#27594)
• Update index.rst (#27529)
• Add note about pushing the lazy XCom proxy to XCom (#27250)
• Fix BaseOperator link (#27441)
• [docs] best-practices add use variable with template example. (#27316)
• docs for custom view using plugin (#27244)
• Update graph view and grid view on overview page (#26909)
• Documentation fixes (#26819)
• make consistency on markup title string level (#26696)
• Add documentation to dag test function (#26713)
• Fix broken URL for docker-compose.yaml (#26726)
• Add a note against use of top level code in timetable (#26649)
• Fix example_datasets dag names (#26495)
• Update docs: zip-like effect is now possible in task mapping (#26435)
• changing to task decorator in docs from classic operator use (#25711)

Apache Airflow 2.4.3

Bug Fixes

• Fix double logging with some task logging handler (#27591)
• Replace FAB url filtering function with Airflow's (#27576)
• Fix mini scheduler expansion of mapped task (#27506)
• SLAMiss is nullable and not always given back when pulling task instances (#27423)
• Fix behavior of _ when searching for DAGs (#27448)
• Fix getting the dag/task ids from BaseExecutor (#27550)
• Fix SQLAlchemy primary key black-out error on DDRQ (#27538)
• Fix IntegrityError during webserver startup (#27297)
• Add case insensitive constraint to username (#27266)
• Fix python external template keys (#27256)
• Reduce extraneous task log requests (#27233)
• Make RotatingFilehandler used in DagProcessor non-caching (#27223)
• Listener: Set task on SQLAlchemy TaskInstance object (#27167)
• Fix dags list page auto-refresh & jump search null state (#27141)
• Set executor.job_id to BackfillJob.id for backfills (#27020)

Misc/Internal

• Bump loader-utils from 1.4.0 to 1.4.1 in /airflow/www (#27552)
• Reduce log level for k8s TCP_KEEPALIVE etc warnings (#26981)

Doc only changes

• Use correct executable in docker compose docs (#27529)
• Fix wording in DAG Runs description (#27470)
• Document that KubernetesExecutor overwrites container args (#27450)
• Fix BaseOperator links (#27441)
• Correct timer units to seconds from milliseconds. (#27360)
• Add missed import in the Trigger Rules example (#27309)
• Update SLA wording to reflect it is relative to Dag Run start. (#27111)
• Add kerberos environment variables to the docs (#27028)

Apache Airflow 2.4.2

Bug Fixes

• Make tracebacks opt-in (#27059)
• Add missing AUTOINC/SERIAL for FAB tables (#26885)
• Add separate error handler for 405(Method not allowed) errors (#26880)
• Don't re-patch pods that are already controlled by current worker (#26778)
• Handle mapped tasks in task duration chart (#26722)
• Fix task duration cumulative chart (#26717)
• Avoid 500 on dag redirect (#27064)
• Filter dataset dependency data on webserver (#27046)
• Remove double collection of dags in airflow dags reserialize (#27030)
• Fix auto refresh for graph view (#26926)
• Don't overwrite connection extra with invalid json (#27142)
• Fix next run dataset modal links (#26897)
• Change dag audit log sort by date from asc to desc (#26895)
• Bump min version of jinja2 (#26866)
• Add missing colors to state_color_mapping jinja global (#26822)
• Fix running debuggers inside airflow tasks test (#26806)
• Fix warning when using xcomarg dependencies (#26801)
• demote Removed state in priority for displaying task summaries (#26789)
• Ensure the log messages from operators during parsing go somewhere (#26779)
• Add restarting state to TaskState Enum in REST API (#26776)
• Allow retrieving error message from data.detail (#26762)
• Simplify origin string cleaning (#27143)
• Remove DAG parsing from StandardTaskRunner (#26750)
• Fix non-hidden cumulative chart on duration view (#26716)
• Remove TaskFail duplicates check (#26714)
• Fix airflow tasks run --local when dags_folder differs from that of processor (#26509)
• Fix yarn warning from d3-color (#27139)
• Fix version for a couple configurations (#26491)
• Revert "No grid auto-refresh for backfill dag runs (#25042)" (#26463)
• Retry on Airflow Schedule DAG Run DB Deadlock (#26347)

Misc/Internal

• Clean-ups around task-mapping code (#26879)
• Move user-facing string to template (#26815)
• add icon legend to datasets graph (#26781)
• Bump sphinx and sphinx-autoapi (#26743)
• Simplify RTIF.delete_old_records() (#26667)
• Bump FAB to 4.1.4 (#26393)

Doc only changes

• Fixed triple quotes in task group example (#26829)
• Documentation fixes (#26819)
• make consistency on markup title string level (#26696)
• Add a note against use of top level code in timetable (#26649)
• Fix broken URL for docker-compose.yaml (#26726)

Apache Airflow Helm Chart 1.7.0

New Features

• Make cleanup job history configurable (#26838)
• Added labels to specific Airflow components (#25031)
• Add StatsD overrideMappings in Helm chart values (#26598)
• Adding podAnnotations to StatsD deployment template (#25732)
• Container specific extra environment variables (#24784)
• Custom labels for extra Secrets and ConfigMaps (#25283)
• Add revisionHistoryLimit to all deployments (#25059)
• Adding podAnnotations to Redis StatefulSet (#23708)
• Provision Standalone Dag Processor (#23711)
• Add configurable scheme for webserver probes (#22815)
• Add support for KEDA HPA config to Helm chart (#24220)

Improvements

• Add 'executor' label to Airflow scheduler deployment (#25684)
• Add default flower_url_prefix in Helm chart values (#26415)
• Add liveness probe to Celery workers (#25561)
• Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

• Fix pod template imagePullPolicy (#26423)
• Do not declare a volume for sshKeySecret if dag persistence is enabled (#22913)
• Pass worker annotations to generated pod template (#24647)
• Fix semver compare number for jobs check command (#24480)
• Use --local flag for liveness probes in Airflow 2.5+ (#24999)

Doc only changes

• Improve documentation on helm hooks disabling (#26747)
• Remove ssh:// prefix from git repo value (#26632)
• Fix defaultAirflowRepository comment (#26428)
• Baking DAGs into Docker image (#26401)
• Reload pods when using the same DAG tag (#24576)
• Minor clarifications about result_backend, dag processor, and helm uninstall (#24929)
• Add hyperlinks to GitHub PRs for Release Notes (#24532)
• Terraform should not use Helm hooks for starting jobs (#26604)
• Flux should not use Helm hooks for starting jobs (#24288)
• Provide details on how to pull Airflow image from a private repository (#24394)
• Helm logo no longer a link (#23977)
• Document LocalKubernetesExecutor support in chart (#23876)
• Update Production Guide (#23836)

Misc

• Default Airflow version to 2.4.1 (#26485)
• Vendor in the Bitnami chart (#24395)
• Remove kubernetes 1.20 support (#25871)

Apache Airflow 2.4.1

Bug Fixes

• When rendering template, unmap task in context (#26702)
• Fix scroll overflow for ConfirmDialog (#26681)
• Resolve deprecation warning re Table.exists() (#26616)
• Fix XComArg zip bug (#26636)
• Use COALESCE when ordering runs to handle NULL (#26626)
• Check user is active (#26635)
• No missing user warning for public admin (#26611)
• Allow MapXComArg to resolve after serialization (#26591)
• Resolve warning about DISTINCT ON query on dags view (#26608)
• Log warning when secret backend kwargs is invalid (#26580)
• Fix grid view log try numbers (#26556)
• Template rendering issue in passing templates_dict to task decorator (#26390)
• Fix Deferrable stuck as scheduled during backfill (#26205)
• Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)
• Correctly set json_provider_class on Flask app so it uses our encoder (#26554)
• Fix WSGI root app (#26549)
• Fix deadlock when mapped task with removed upstream is rerun (#26518)
• ExecutorConfigType should be cacheable (#26498)
• Fix proper joining of the path for logs retrieved from celery workers (#26493)
• DAG Deps extends base_template (#26439)
• Don't update backfill run from the scheduler (#26342)

Doc only changes

• Clarify owner links document (#26515)
• Fix invalid RST in dataset concepts doc (#26434)
• Document the non-sensitive-only option for expose_config (#26507)
• Fix example_datasets dag names (#26495)
• Zip-like effect is now possible in task mapping (#26435)
• Use task decorator in docs instead of classic operators (#25711)

Apache Airflow 2.4.0

New Features

• Add Data-aware Scheduling (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22)
• Add @task.short_circuit TaskFlow decorator (#25752)
• Make execution_date_or_run_id optional in tasks test command (#26114)
• Automatically register DAGs that are used in a context manager (#23592, #26398)
• Add option of sending DAG parser logs to stdout. (#25754)
• Support multiple DagProcessors parsing files from different locations. (#25935)
• Implement ExternalPythonOperator (#25780)
• Make execution_date optional for command dags test (#26111)
• Implement expand_kwargs() against a literal list (#25925)
• Add trigger rule tooltip (#26043)
• Add conf parameter to CLI for airflow dags test (#25900)
• Include scheduled slots in pools view (#26006)
• Add output property to MappedOperator (#25604)
• Add roles delete command to cli (#25854)
• Add Airflow specific warning classes (#25799)
• Add support for TaskGroup in ExternalTaskSensor (#24902)
• Add @task.kubernetes taskflow decorator (#25663)
• Add a way to import Airflow without side-effects (#25832)
• Let timetables control generated run_ids. (#25795)
• Allow per-timetable ordering override in grid view (#25633)
• Grid logs for mapped instances (#25610, #25621, #25611)
• Consolidate to one schedule param (#25410)
• DAG regex flag in backfill command (#23870)
• Adding support for owner links in the Dags view UI (#25280)
• Ability to clear a specific DAG Run's task instances via REST API (#23516)
• Possibility to document DAG with a separate markdown file (#25509)
• Add parsing context to DAG Parsing (#25161)
• Implement CronTriggerTimetable (#23662)
• Add option to mask sensitive data in UI configuration page (#25346)
• Create new databases from the ORM (#24156)
• Implement XComArg.zip(*xcom_args) (#25176)
• Introduce sla_miss metric (#23402)
• Implement map() semantic (#25085)
• Add override method to TaskGroupDecorator (#25160)
• Implement expand_kwargs() (#24989)
• Add parameter to turn off SQL query logging (#24570)
• Add DagWarning model, and a check for missing pools (#23317)
• Add Task Logs to Grid details panel (#24249)
• Added small health check server and endpoint in scheduler(#23905)
• Add built-in External Link for ExternalTaskMarker operator (#23964)
• Add default task retry delay config (#23861)
• Add clear DagRun endpoint. (#23451)
• Add support for timezone as string in cron interval timetable (#23279)
• Add auto-refresh to dags home page (#22900, #24770)

Improvements

• Add more weekday operator and sensor examples #26071 (#26098)
• Add subdir parameter to dags reserialize command (#26170)
• Update zombie message to be more descriptive (#26141)
• Only send an SlaCallbackRequest if the DAG is scheduled (#26089)
• Promote Operator.output more (#25617)
• Upgrade API files to typescript (#25098)
• Less hacky double-rendering prevention in mapped task (#25924)
• Improve Audit log (#25856)
• Remove mapped operator validation code (#25870)
• More DAG(schedule=...) improvements (#25648)
• Reduce operator_name dupe in serialized JSON (#25819)
• Make grid view group/mapped summary UI more consistent (#25723)
• Remove useless statement in task_group_to_grid (#25654)
• Add optional data interval to CronTriggerTimetable (#25503)
• Remove unused code in /grid endpoint (#25481)
• Add and document description fields (#25370)
• Improve Airflow logging for operator Jinja template processing (#25452)
• Update core example DAGs to use @task.branch decorator (#25242)
• Update DAG audit_log route (#25415)
• Change stdout and stderr access mode to append in commands (#25253)
• Remove getTasks from Grid view (#25359)
• Improve taskflow type hints with ParamSpec (#25173)
• Use tables in grid details panes (#25258)
• Explicitly list @dag arguments (#25044)
• More typing in SchedulerJob and TaskInstance (#24912)
• Patch getfqdn with more resilient version (#24981)
• Replace all NBSP characters by whitespaces (#24797)
• Re-serialize all DAGs on airflow db upgrade (#24518)
• Rework contract of try_adopt_task_instances method (#23188)
• Make expand() error vague so it's not misleading (#24018)
• Add enum validation for [webserver]analytics_tool (#24032)
• Add dttm searchable field in audit log (#23794)
• Allow more parameters to be piped through via execute_in_subprocess (#23286)
• Use func.count to count rows (#23657)
• Remove stale serialized dags (#22917)
• AIP45 Remove dag parsing in airflow run local (#21877)
• Add support for queued state in DagRun update endpoint. (#23481)
• Add fields to dagrun endpoint (#23440)
• Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

• Have consistent types between the ORM and the migration files (#24044, #25869)
• Disallow any dag tags longer than 100 char (#25196)
• Add the dag_id to AirflowDagCycleException message (#26204)
• Properly build URL to retrieve logs independently from system (#26337)
• For worker log servers only bind to IPV6 when dual stack is available (#26222)
• Fix TaskInstance.task not defined before handle_failure (#26040)
• Undo secrets backend config caching (#26223)
• Fix faulty executor config serialization logic (#26191)
• Show DAGs and Datasets menu links based on role permission (#26183)
• Allow setting TaskGroup tooltip via function docstring (#26028)
• Fix RecursionError on graph view of a DAG with many tasks (#26175)
• Fix backfill occasional deadlocking (#26161)
• Fix DagRun.start_date not set during backfill with --reset-dagruns True (#26135)
• Use label instead of id for dynamic task labels in graph (#26108)
• Don't fail DagRun when leaf mapped_task is SKIPPED (#25995)
• Add group prefix to decorated mapped task (#26081)
• Fix UI flash when triggering with dup logical date (#26094)
• Fix Make items nullable for TaskInstance related endpoints to avoid API errors (#26076)
• Fix BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)
• Fix legacy timetable schedule interval params (#25999)
• Fix response schema for list-mapped-task-instance (#25965)
• Properly check the existence of missing mapped TIs (#25788)
• Fix broken auto-refresh on grid view (#25950)
• Use per-timetable ordering in grid UI (#25880)
• Rewrite recursion when parsing DAG into iteration (#25898)
• Find cross-group tasks in iter_mapped_dependants (#25793)
• Fail task if mapping upstream fails (#25757)
• Support / in variable get endpoint (#25774)
• Use cfg default_wrap value for grid logs (#25731)
• Add origin request args when triggering a run (#25729)
• Operator name separate from class (#22834)
• Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
• Return None if an XComArg fails to resolve (#25661)
• Correct json arg help in airflow variables set command (#25726)
• Added MySQL index hint to use ti_state on find_zombies query (#25725)
• Only excluded actually expanded fields from render (#25599)
• Grid, fix toast for axios errors (#25703)
• Fix UI redirect (#26409)
• Require dag_id arg for dags list-runs (#26357)
• Check for queued states for dags auto-refresh (#25695)
• Fix upgrade code for the dag_owner_attributes table (#25579)
• Add map index to task logs api (#25568)
• Ensure that zombie tasks for dags with errors get cleaned up (#25550)
• Make extra link work in UI (#25500)
• Sync up plugin API schema and definition (#25524)
• First/last names can be empty (#25476)
• Refactor DAG pages to be consistent (#25402)
• Check expand_kwargs() input type before unmapping (#25355)
• Filter XCOM by key when calculating map lengths (#24530)
• Fix ExternalTaskSensor not working with dynamic task (#25215)
• Added exception catching to send default email if template file raises any exception (#24943)
• Bring MappedOperator members in sync with BaseOperator (#24034)

Misc/Internal

• Add automatically generated ERD schema for the MetaData DB (#26217)
• Mark serialization functions as internal (#26193)
• Remove remaining deprecated classes and replace them with PEP562 (#26167)
• Move dag_edges and task_group_to_dict to corresponding util modules (#26212)
• Lazily import many modules to improve import speed (#24486, #26239)
• FIX Incorrect typing information (#26077)
• Add missing contrib classes to deprecated dictionaries (#26179)
• Re-configure/connect the ORM after forking to run a DAG processor (#26216)
• Remove cattrs from lineage processing. (#26134)
• Removed deprecated contrib files and replace them with PEP-562 getattr (#26153)
• Make BaseSerialization.serialize "public" to other classes. (#26142)
• Change the template to use human readable task_instance description (#25960)
• Bump moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)
• Fix Flask deprecation warning (#25753)
• Add CamelCase to generated operations types (#25887)
• Fix migration issues and tighten the CI upgrade/downgrade test (#25869)
• Fix type annotations in SkipMixin (#25864)
• Workaround setuptools editable packages path issue (#25848)
• Bump undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)
• Add custom_operator_name attr to _BranchPythonDecoratedOperator (#25783)
• Clarify filename_template deprecation message (#25749)
• Use ParamSpec to replace ... in Callable (#25658)
• Remove deprecated modules (#25543)
• Documentation on task mapping additions (#24489)
• Remove Smart Sensors (#25507)
• Fix elasticsearch test config to avoid warning on deprecated templa...

Apache Airflow 2.3.4

Bug Fixes

• Disable attrs state management on MappedOperator (#24772)
• Serialize pod_override to JSON before pickling executor_config (#24356)
• Fix pid check (#24636)
• Rotate session id during login (#25771)
• Fix mapped sensor with reschedule mode (#25594)
• Cache the custom secrets backend so the same instance gets re-used (#25556)
• Add right padding (#25554)
• Fix reducing mapped length of a mapped task at runtime after a clear (#25531)
• Fix airflow db reset when dangling tables exist (#25441)
• Change disable_verify_ssl behaviour (#25023)
• Set default task group in dag.add_task method (#25000)
• Removed interfering force of index. (#25404)
• Remove useless logging line (#25347)
• Adding mysql index hint to use index on task_instance.state in critical section query (#25673)
• Configurable umask to all daemonized processes. (#25664)
• Fix the errors raised when None is passed to template filters (#25593)
• Allow wildcarded CORS origins (#25553)
• Fix "This Session's transaction has been rolled back" (#25532)
• Fix Serialization error in TaskCallbackRequest (#25471)
• fix - resolve bash by absolute path (#25331)
• Add __repr__ to ParamsDict class (#25305)
• Only load distribution of a name once (#25296)
• convert TimeSensorAsync target_time to utc on call time (#25221)
• call updateNodeLabels after expandGroup (#25217)
• Stop SLA callbacks gazumping other callbacks and DOS'ing the DagProcessorManager queue (#25147)
• Fix invalidateQueries call (#25097)
• airflow/www/package.json: Add name, version fields. (#25065)
• No grid auto-refresh for backfill dag runs (#25042)
• Fix tag link on dag detail page (#24918)
• Fix zombie task handling with multiple schedulers (#24906)
• Bind log server on worker to IPv6 address (#24755) (#24846)
• Add %z for %(asctime)s to fix timezone for logs on UI (#24811)
• TriggerDagRunOperator.operator_extra_links is attr (#24676)
• Send DAG timeout callbacks to processor outside of prohibit_commit (#24366)
• Don't rely on current ORM structure for db clean command (#23574)
• Clear next method when clearing TIs (#23929)
• Two typing fixes (#25690)

Doc only changes

• Update set-up-database.rst (#24983)
• Fix syntax in mysql setup documentation (#24893 (#24939)
• Note how DAG policy works with default_args (#24804)
• Update PythonVirtualenvOperator Howto (#24782)
• Doc: Add hyperlinks to Github PRs for Release Notes (#24532)

Misc/Internal

• Remove depreciation warning when use default remote tasks logging handlers (#25764)
• clearer method name in scheduler_job.py (#23702)
• Bump cattrs version (#25689)
• Include missing mention of external_executor_id in sql_engine_collation_for_ids docs (#25197)
• Refactor DR.task_instance_scheduling_decisions (#24774)
• Sort operator extra links (#24992)
• Extends resolve_xcom_backend function level documentation (#24965)
• Upgrade FAB to 4.1.3 (#24884)
• Limit Flask to <2.3 in the wake of 2.2 breaking our tests (#25511)
• Limit astroid version to < 2.12 (#24982)
• Move javascript compilation to host (#25169)
• Bump typing-extensions and mypy for ParamSpec (#25088)

Apache Airflow 2.3.3

Bug Fixes

• Fix exception in mini task scheduler (#24865)
• Fix cycle bug with attaching label to task group (#24847)
• Fix timestamp defaults for sensorinstance (#24638)
• Move fallible ti.task.dag assignment back inside try/except block (#24533) (#24592)
• Add missing types to FSHook (#24470)
• Mask secrets in stdout for airflow tasks test (#24362)
• DebugExecutor use ti.run() instead of ti._run_raw_task (#24357)
• Fix bugs in URI constructor for MySQL connection (#24320)
• Missing scheduleinterval nullable true added in openapi (#24253)
• Unify return_code interface for task runner (#24093)
• Handle occasional deadlocks in trigger with retries (#24071)
• Remove special serde logic for mapped op_kwargs (#23860)
• ExternalTaskSensor respects soft_fail if the external task enters a failed_state (#23647)
• Fix StatD timing metric units (#21106)
• Add cache_ok flag to sqlalchemy TypeDecorators. (#24499)
• Allow for LOGGING_LEVEL=DEBUG (#23360)
• Fix grid date ticks (#24738, #24849)
• Debounce status highlighting in Grid view (#24710)
• Fix Grid vertical scrolling (#24684)
• don't try to render child rows for closed groups (#24637)
• Do not calculate grid root instances (#24528)
• Maintain grid view selection on filtering upstream (#23779)
• Speed up grid_data endpoint by 10x (#24284)
• Apply per-run log templates to log handlers (#24153)
• Don't crash scheduler if exec config has old k8s objects (#24117)
• TI.log_url fix for map_index (#24335)
• Fix migration 0080_2_0_2 - Replace null values before setting column not null (#24585)
• Patch sql_alchemy_conn if old Postgres schemes used (#24569)
• Seed log_template table (#24511)
• Fix deprecated log_id_template value (#24506)
• Fix toast messages (#24505)
• Add indexes for CASCADE deletes for task_instance (#24488)
• Return empty dict if Pod JSON encoding fails (#24478)
• Improve grid rendering performance with a custom tooltip (#24417, #24449)
• Check for run_id for grid group summaries (#24327)
• Optimize calendar view for cron scheduled DAGs (#24262)
• Use get_hostname instead of socket.getfqdn (#24260)
• Check that edge nodes actually exist (#24166)
• Fix useTasks crash on error (#24152)
• Do not fail re-queued TIs (#23846)
• Reduce grid view API calls (#24083)
• Rename Permissions to Permission Pairs. (#24065)
• Replace use_task_execution_date with use_task_logical_date (#23983)
• Grid fix details button truncated and small UI tweaks (#23934)
• Add TaskInstance State REMOVED to finished states and success states (#23797)
• Fix mapped task immutability after clear (#23667)
• Fix permission issue for dag that has dot in name (#23510)
• Fix closing connection dbapi.get_pandas_df (#23452)
• Check bag DAG schedule_interval match timetable (#23113)
• Parse error for task added to multiple groups (#23071)
• Fix flaky order of returned dag runs (#24405)
• Migrate jsx files that affect run/task selection to tsx (#24509)
• Fix links to sources for examples (#24386)
• Set proper Content-Type and chartset on grid_data endpoint (#24375)

Doc only changes

• Fix doc description of [core] parallelism config setting (#23768)
• Update templates doc to mention extras and format Airflow Vars / Conns (#24735)
• Document built in Timetables (#23099)
• Alphabetizes two tables (#23923)
• Clarify that users should not use Maria DB (#24556)
• Add imports to deferring code samples (#24544)
• Add note about image regeneration in June 2022 (#24524)
• Small cleanup of get_current_context() chapter (#24482)
• Fix default 2.2.5 log_id_template (#24455)
• Update description of installing providers separately from core (#24454)
• Mention context variables and logging (#24304)

Misc/Internal

• Do not support MSSQL less than v2017 in code (#24095)
• Drop Python 3.6 compatibility objects/modules (#24048)
• Remove upper-binding for SQLAlchemy (#24819)
• Remove internet explorer support (#24495)
• Removing magic status code numbers from api_connexion (#24050)
• Upgrade FAB to 4.1.2 (#24619)
• Switch Markdown engine to markdown-it-py (#19702)
• Update rich to latest version across the board. (#24186)
• Get rid of TimedJSONWebSignatureSerializer (#24519)
• Update flask-appbuilder authlib/ oauth dependency (#24516)
• Upgrade to webpack 5 (#24485)
• Add typescript (#24337)
• The JWT claims in the request to retrieve logs have been standardized: we use nbf and aud claims for
  maturity and audience of the requests. Also "filename" payload field is used to keep log name. (#24519)
• Address all yarn test warnings (#24722)
• Upgrade to react 18 and chakra 2 (#24430)
• Refactor DagRun.verify_integrity (#24114)
• Upgrade FAB to 4.1.1 (#24399)
• We now need at least Flask-WTF 0.15 (#24621)