Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Allow set Dag Run resource into Dag Level permission #40703

Merged
merged 269 commits into from
Aug 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
269 commits
Select commit Hold shift + click to select a range
dcac0e1
Brings back mypy-checks (#38597)
potiuk Mar 28, 2024
4cba4ef
Rename to XComObjectStorageBackend (#38607)
uranusjr Mar 29, 2024
a3ff1ce
bugfix: properly serialize nested attrs classes (#38591)
tomrutter Mar 29, 2024
886cb55
Add padding to bottom of log block. (#38610)
tirkarthi Mar 29, 2024
373abd0
Fix `total_entries` count on the event logs endpoint (#38625)
jedcunningham Mar 30, 2024
5e77935
Rename `allowed_filter_attrs` to `allowed_sort_attrs` (#38626)
jedcunningham Mar 30, 2024
3c214a5
Mark more fields on API as dump-only (#38616)
jscheffl Mar 31, 2024
8ee1240
Add 2.9.0b2 to issue template (#38586)
ephraimbuddy Mar 28, 2024
17fa8d3
Upgrade gitpython as build dependency (#38644)
potiuk Mar 31, 2024
4918174
Prohibit to use unsupported DB backends in tests (#38530)
Taragolis Mar 27, 2024
5cc3dc8
Disable support of a legacy `LocalPath` in favor of stdlib `pathlib.P…
Taragolis Apr 1, 2024
4467cd6
Add a migration script for encrypted trigger kwargs (#38358)
hussein-awala Apr 1, 2024
9ed3535
bump ruff to 0.3.5 (#38656)
hussein-awala Apr 1, 2024
6746fea
More human friendly "show tables" output for db cleanup (#38654)
jedcunningham Apr 1, 2024
0d32e45
Move cleanup-docker to a shell scripts from composite action (#38659)
potiuk Apr 1, 2024
40379ed
bump uv to 0.1.27 (#38657)
hussein-awala Apr 1, 2024
90dc2cc
Remove display of param from the UI (#38660)
hussein-awala Apr 2, 2024
4260f56
Clean up unused triggers in a single query for all dialects except My…
hussein-awala Apr 2, 2024
9ec5c41
Improve stability of remove_task_decorator function (#38649)
romsharon98 Apr 1, 2024
67c7392
Add exclude/include events filters to audit log (#38506)
bbovenzi Apr 2, 2024
af8c481
Improve trigger assign_unassigned by merging alive_triggerer_ids and …
hussein-awala Apr 2, 2024
a2e513c
Add a task instance dependency for mapped dependencies (#37498)
stevenschaerer Apr 2, 2024
3f51039
Check-out the target commit when preparing PROD build artifacts (#38678)
potiuk Apr 2, 2024
356897a
Remove image tag from Helm test name (#38621)
potiuk Mar 29, 2024
367e8fe
Use explicit and easier to use runs-on approach for CI workflows (#38…
potiuk Apr 1, 2024
471164b
Rename build arg for upgrade-to-newer-dependencies (#38637)
potiuk Apr 2, 2024
cf6357f
Update known issue with Pendulum 3 in 2.8.1 release notes (#38675)
potiuk Apr 2, 2024
8d12bb1
Update UI doc screenshots (#38680)
bbovenzi Apr 2, 2024
d46d7ec
Remove google-cloud-aiplatform when tests without pydantic are run (#…
potiuk Mar 29, 2024
c8c1863
Update RELEASE_NOTES.rst
ephraimbuddy Mar 31, 2024
e7fa4fb
Extract checkout target commit to a composite action (#38682)
potiuk Apr 2, 2024
037effd
Drop pre-release specification from pre-installed provider versions (…
potiuk Apr 3, 2024
be98e45
Fix few nits in tag-providers command (#38705)
potiuk Apr 3, 2024
99a64c4
Remove obtain postgres APT repo signing key from mysql install script…
Taragolis Apr 3, 2024
048518f
Remove FAB provider from "chicken-egg" providers. (#38698)
potiuk Apr 3, 2024
0dee402
The dev0 suffix should only be applied in main (#38727)
potiuk Apr 3, 2024
dccfe1a
bump uv to 0.1.28 (#38690)
hussein-awala Apr 3, 2024
2190a0f
Rename UV_REQUEST_TIMEOUT to UV_HTTP_TIMEOUT and make it more usable …
potiuk Apr 4, 2024
4871178
Make UV_HTTP_TIMEOUT default value bigger in WSL2 environment (#38742)
potiuk Apr 4, 2024
4f0c15c
Unhide CI instructions when dependencies need regeneration (#38748)
potiuk Apr 4, 2024
37b54bf
Switch to `pip` for all PROD images built in the release branches (#3…
potiuk Apr 4, 2024
83e6cf2
Also disable UV for PROD cache build in release branches (#38752)
potiuk Apr 4, 2024
0961e42
Reorder OpenAPI Spec tags alphabetically (#38717)
pankajkoti Apr 3, 2024
0518eba
Fix grid header rendering (#38720)
bbovenzi Apr 4, 2024
1fdfaf3
Edited timetable docs (#38505)
lzdanski Apr 4, 2024
cbe61bc
Fix decryption of trigger kwargs when downgrading. (#38743)
ephraimbuddy Apr 4, 2024
920921e
Update RELEASE_NOTES.rst
ephraimbuddy Apr 4, 2024
a3eb5f7
Load providers configuration when gunicorn workers start (#38795)
potiuk Apr 6, 2024
b38575a
fixup! Update RELEASE_NOTES.rst
ephraimbuddy Apr 6, 2024
35684a8
Rename pre-commit scripts, drop pre_commit_ prefix (#38667)
uranusjr Apr 2, 2024
3a9391c
Update hatchling to latest version (1.22.5) (#38780)
potiuk Apr 5, 2024
50f22ff
fixup! fixup! Update RELEASE_NOTES.rst
potiuk Apr 6, 2024
225fe51
Add FAB provider to Airflow extras in the image (#38824)
ephraimbuddy Apr 8, 2024
f8365f2
Update Airflow version in Dockerfile
ephraimbuddy Apr 8, 2024
a1a565e
Add changelog for airflow python client 2.9.0 (#39060)
ephraimbuddy Apr 16, 2024
5155e17
Mark `extending/add-build-essential-extend/Dockerfile` docker example…
Taragolis Apr 14, 2024
5a6e7ee
Fix PROD image package installation in CI (#39035)
potiuk Apr 15, 2024
4727b4b
Do not add local airflow sources in CI when building k8s image (#39043)
potiuk Apr 15, 2024
40c1cf8
Hide internals of some groupped workflows (#38604)
potiuk Mar 29, 2024
a54fb16
Fix kubeconform pre-commit to not leave around containers (#38688)
jedcunningham Apr 2, 2024
3b7aecd
Workaround occasional failures of docker system prune in public runne…
potiuk Apr 8, 2024
84046b2
Update build dependencies to their latest versions (#38930)
potiuk Apr 11, 2024
9220e1c
Upgrade to latest hatchling as build dependency (#39044)
potiuk Apr 16, 2024
dbebce4
Update hatchling to version 1.24.0 (#39072)
ephraimbuddy Apr 17, 2024
a4027b7
update simple-salesforce type hints to support 1.12.6 (#39047)
hussein-awala Apr 16, 2024
789051c
Fix main failing because of the new xmlsec 1.3.14 compatibility (#39104)
potiuk Apr 18, 2024
eee0478
Upgrade to latest hatchling 1.24.1 (again).
potiuk Apr 18, 2024
e61cb8f
Apply PROVIDE_PROJECT_ID mypy workaround across Google provider (#39129)
potiuk Apr 19, 2024
1d40343
Fix typo with DatasetOrTimeSchedule as it was renamed (#38576)
sunank200 Mar 28, 2024
70d31c0
Raise deprecated warning in `airflow dags backfill` only if `-I`/`--i…
Taragolis Apr 2, 2024
c501d3a
Use `methodtools.lru_cache` instead of `functools.lru_cache` in class…
Taragolis Apr 3, 2024
83e7dec
Improve audit log events filter (#38719)
bbovenzi Apr 4, 2024
16256a2
Bump undici from 5.28.3 to 5.28.4 in /airflow/www (#38751)
dependabot[bot] Apr 4, 2024
c6c589f
Use async db calls in WorkflowTrigger (#38689)
stevenschaerer Apr 4, 2024
6d36c80
Fix 'use case' spelling (#38817)
uranusjr Apr 8, 2024
a0249d4
Remove decorator from rendering fields example (#38827)
eladkal Apr 8, 2024
3b84410
Update version added field in config.yml (#38840)
ephraimbuddy Apr 8, 2024
672f8ad
Remove a couple items from the 2.9.0 changelog (#38866)
jedcunningham Apr 9, 2024
84b76e0
Fix dag run link params (#38873)
bbovenzi Apr 9, 2024
72b2ec0
Fix typo in templates-ref.rst (#38865)
alxkolm Apr 9, 2024
1725139
Fix problem that links in legacy log view can not be clicked (#38882)
jscheffl Apr 10, 2024
2659704
Fix implicit cartessian product in AirflowSecurityManagerV2 (#38913)
Taragolis Apr 11, 2024
e998604
Limit redis image to 7.2 in sample docker-compose.yaml (#38938)
Taragolis Apr 11, 2024
df37d60
Fix `SAWarning` 'Coercing Subquery object into a select() for use in …
Taragolis Apr 11, 2024
39dd593
Fix dag run selection (#38941)
bbovenzi Apr 11, 2024
2e25153
(external_task.py): changed the background color of external task in …
idantepper Apr 12, 2024
b35f67d
Fix calendar view with no DAG Run. (#38964)
tirkarthi Apr 12, 2024
5c6736d
Don't wait for DagRun lock in mini scheduler (#38914)
dstandish Apr 11, 2024
1f70db4
Use debug level for minischeduler skip (#38976)
dstandish Apr 12, 2024
adfabe8
Add in Trove classifiers Python 3.12 support (#39004)
Lee-W Apr 14, 2024
82a1b6a
Add `inherit_cache` attribute for `CreateTableAs` custom SA Clause (#…
Taragolis Apr 14, 2024
d6e397f
Correcting an example taskflow example (#39015)
amoghrajesh Apr 15, 2024
083a781
Undeprecate `BaseXCom.get_one` method for now (#38991)
Taragolis Apr 15, 2024
3487fcc
Bugfix: Move rendering of `map_index_template` so it renders for fail…
TJaniF Apr 15, 2024
983ab58
Check whether `AUTH_ROLE_PUBLIC` is set in `check_authentication` (#…
Lee-W Apr 15, 2024
394821f
Hostname was "redacted", not "redact"; remove it when there is no con…
jedcunningham Apr 15, 2024
2970e35
Fix doc redirects (#39039)
vincbeck Apr 16, 2024
672d47e
[docs] update `DagBag` class docstring to include all params (#38814)
rawwar Apr 16, 2024
0602de2
Data aware scheduling docs edits (#38687)
lzdanski Apr 16, 2024
ed4cd93
update document (#39068)
humit0 Apr 16, 2024
5fcf36c
Check that the dataset<>task exists before trying to render graph (#3…
bbovenzi Apr 17, 2024
7cd476e
Graph view improvements (#38940)
bbovenzi Apr 17, 2024
6cee9b0
Improve rendered_template ux in react dag page (#39122)
bbovenzi Apr 19, 2024
2091072
Improve task filtering UX (#39119)
bbovenzi Apr 22, 2024
fd0226e
Add k8s 1.29 to supported version in docs (#39168)
jedcunningham Apr 22, 2024
092ceae
Update `is_authorized_custom_view` from auth manager to handle custom…
vincbeck Apr 22, 2024
8e458d9
Add warning about run/import Airflow under the Windows (#39196)
Taragolis Apr 23, 2024
c583674
Add link to `user` and `role` commands (#39224)
potiuk Apr 24, 2024
7cd311d
Use grid view for Task Instance's `log_url` (#39183)
AetherUnbound Apr 24, 2024
899d4e8
Fixed side effect of menu filtering causing disappearing menus (#39229)
potiuk Apr 24, 2024
f299149
doc: dynamictaskmapping pythonoperator op_kwargs (#39242)
raphaelauv Apr 25, 2024
505ecd9
Get served logs when remote or executor logs not available for non-ru…
kahlstrm Apr 25, 2024
d2ae8c9
Force to use Airflow Deprecation warnings categories on `@deprecated`…
Taragolis Apr 25, 2024
df9db0c
Add Grid button to Task Instance view (#39223)
csp33 Apr 25, 2024
d5d8b58
Add workaround for datetime-local input in firefox (#39261)
bbovenzi Apr 25, 2024
c9cc726
Fix trigger kwarg encryption migration (#39246)
jedcunningham Apr 25, 2024
0a40b34
Fix supported k8s version in docs (#39172)
jedcunningham Apr 22, 2024
eb11360
(img/edge_label_example.png): changed imaged to new label view (#38802)
idantepper Apr 7, 2024
368ff13
Move significant note for past release to release notes (#39283)
jedcunningham Apr 27, 2024
cddaf23
Copy menu_item href for nav bar (#39282)
bbovenzi Apr 27, 2024
06296f7
fix(google,log): Avoid log name overriding (#38071)
AlexisBRENON Apr 3, 2024
24f7a89
Make task log messages include run_id (#39280)
RyuSA Apr 27, 2024
8b3c24f
`TriggerDagRunOperator` depreacte `exection_date` in favor of `logica…
flolas Apr 27, 2024
e33dd61
bump uv to 0.1.29 (#38758)
hussein-awala Apr 5, 2024
f7b9749
bump uv to 0.1.32 (#38874)
hussein-awala Apr 16, 2024
afefa76
Upgrade to latest build/install dependencies (#39160)
potiuk Apr 22, 2024
74abd01
Fix ERD sha256 & svg
ephraimbuddy Apr 29, 2024
34f23ce
Avoid to use `functools.lru_cache` in class methods in `google` provi…
Taragolis Apr 1, 2024
042f302
Update version to 2.9.1
ephraimbuddy Apr 29, 2024
e203935
Update RELEASE_NOTES.rst
ephraimbuddy Apr 29, 2024
fe7dd31
Fix reproducible build hash
ephraimbuddy Apr 29, 2024
44a1fe7
Avoid to use `functools.lru_cache` in class methods in dev (#38627)
Taragolis Mar 29, 2024
5356cb9
Update RELEASE_NOTES.rst
jedcunningham Apr 29, 2024
fd87589
Update trigger kwargs migration to specify existing_nullable (#39361)
ephraimbuddy May 2, 2024
5ca5f95
Fix sqlite constraints update and ignore session_session_id_uq index
ephraimbuddy May 2, 2024
2d53c10
Update RELEASE_NOTES.rst
ephraimbuddy May 2, 2024
3646de3
Add multi-team diagrams (#38861)
potiuk Apr 10, 2024
0b1502a
Fix `sql_alchemy_engine_args` config example (#38971)
jedcunningham Apr 12, 2024
4518856
Update testing_packages.rst (#38996)
shahar1 Apr 14, 2024
e4a4e2d
Fix stacklevel for TaskContextLogger (#39142)
dstandish Apr 23, 2024
0c2d53e
fix(io.path): add missing conn_id to string representation of ObjectS…
Lee-W Apr 30, 2024
de98856
Add support for OrbStack in breeze (#38725)
jedcunningham Apr 3, 2024
6d183e7
Do not provide deprecated `execution_date` in `@apply_lineage` (#39327)
Taragolis Apr 30, 2024
2e612ba
Remove unused index `idx_last_scheduling_decision` on `dag_run` table…
pankajkoti Apr 30, 2024
c4624fd
docs: fix environment variable names when section name has dot in it …
kahlstrm May 1, 2024
3562374
Fix TaskHandlerWithCustomFormatter now adds prefix only once (#38502)
TiDeane May 3, 2024
dff4a12
Add whitespace after inline literal in docs (#39401)
Taragolis May 4, 2024
ea3e24a
Fix static file caching is disabled in Airflow Webserver. (#39345)
VertexToEdge May 4, 2024
9b259f6
Fix HTTP 500 Internal Server Error if DAG is triggered with bad param…
jscheffl May 4, 2024
c7606af
Fix typo sensitive masking words in docs (#39415)
lifnaja May 5, 2024
1bd3ad1
Updating S3LogLink with an invalid bucket link (#39424)
amoghrajesh May 6, 2024
a74c52d
Fix custom actions in security manager `has_access` (#39421)
jedcunningham May 6, 2024
2d88cca
Refactor `SafeDogStatsdLogger` to use `get_validator` to enable patte…
rawwar May 6, 2024
c2f93bd
Update note about restarting triggerer proccess (#39436)
eladkal May 6, 2024
b8b845a
Resolve SA warnings in migrations scripts (#39418)
Taragolis May 6, 2024
81169e7
Use non-dev version for FAB Provider (#39452)
kaxil May 6, 2024
9a4196c
Update docker-compose command (#39504)
Zorgosto May 9, 2024
faddcc2
Adding note for provider dependencies (#39512)
amoghrajesh May 9, 2024
0920ae9
Add Cache-Control "no-store" to all dynamically generated content (#3…
potiuk May 10, 2024
fe57f83
Update best-practices.rst: fix Connection.get -> Connection. get_conn…
iamtodor May 12, 2024
5baa884
Change sorting order of parallel test types (#39579)
potiuk May 13, 2024
39abb6b
Fix typo in Auth Manager docs (#39598)
tomakita May 14, 2024
03b9d1e
Docs: Update tracking-user-activity.rst (#39611)
AndrewRocky May 14, 2024
4310789
fix nit in pg set-up doc (#39628)
EricGao888 May 15, 2024
a85df25
Remove mysql/postgres from extras before determining the installation…
macchiang May 15, 2024
d59ae48
Doc fix: Remove misplaced backquotes in ``faq.rst`` (#39661)
le-chartreux May 16, 2024
9db34af
Fix #35946: Visible DAG RUN doesn't point to the same dag run id (#38…
lotrias17 May 16, 2024
3232a18
Change dataset URI validation to raise warning instead of error in Ai…
tatiana May 17, 2024
c49e220
Update plugins.rst examples to use pyproject.toml over setup.py (#39665)
hterik May 17, 2024
6dd44dd
Reraise exception from strict dataset URI checks (#39719)
uranusjr May 20, 2024
aad65ed
More typing in TimeSensor and TimeSensorAsync (#39696)
jedcunningham May 20, 2024
f0785b8
Better typing for BaseOperator `defer` (#39742)
jedcunningham May 21, 2024
bf89bed
Fix example_branch_operator failing in python 3.12 (#39783)
ephraimbuddy May 23, 2024
f85cc3b
Limit concurrent downloads by UV (#39810)
potiuk May 24, 2024
c9881e5
Remove 'legacy' definition for `CronDataIntervalTimetable` (#39780)
Taragolis May 25, 2024
de3874d
Add max_consecutive_failed_dag_runs in API spec (#39830)
pankajastro May 25, 2024
46e4fba
Also limit concurrent uv downloads when building (#39852)
potiuk May 26, 2024
4fe4781
Change type definition for `provider_info_cache` decorator (#39750)
Taragolis May 26, 2024
a4a6d7e
Remove DAG Run Add option from FAB view (#39881)
jscheffl May 27, 2024
d638c6b
Use SKIP LOCKED instead of NOWAIT in mini scheduler (#39745)
VladimirYushkevich May 28, 2024
68556a7
Improve visibility of links / variables / other configs in Configurat…
Taragolis May 30, 2024
9f06c30
Fix Mark Instance state buttons stay disabled if user lacks permissio…
RodrigoGanancia May 31, 2024
7cb2c0f
Add the existing_nullable to the downgrade side of the migration (#39…
ephraimbuddy May 2, 2024
d14f269
Fix alembic autogeneration and rename mismatching constraints (#39032)
ephraimbuddy May 4, 2024
c1dd584
Activate RUF006 rule to detect dangling asyncio tasks (#38947)
hussein-awala Apr 12, 2024
090ff90
Bump `ruff` to `0.4.3` (#39438)
Taragolis May 6, 2024
2826718
Update uv to 0.1.41 (#39476)
potiuk May 8, 2024
8790af4
Update uv to 0.42 (#39561)
potiuk May 11, 2024
3437e44
Provide extra tip on labeling DynamicTaskMapping (#39977)
fritz-astronomer May 31, 2024
dac25fc
Remove colon from notes header to make headlines in UI consistent (#4…
jscheffl Jun 2, 2024
481fb8e
Warn on mini scheduler failures instead of debug (#39760)
RNHTTR Jun 2, 2024
6ea2c43
Fix bug that makes `AirflowSecurityManagerV2` leave transactions in t…
alejandro-rivera Jun 3, 2024
079dfb3
Update uv and build dependencies. (#39808)
potiuk May 24, 2024
c4c5c7e
add lost space to breeze help (#39458)
uzhastik May 7, 2024
2ea4836
Increase volume size for ARM instance (#39472)
potiuk May 7, 2024
8f7d891
Add hatch_build.py to k8s test venv cache calculation (#39473)
potiuk May 7, 2024
894202c
Fix inconsistency of cache-directive vs. docker-cache (#38580)
potiuk Mar 28, 2024
551b550
Replace to broad exceptions into the dev (#38489)
Taragolis Apr 4, 2024
bfa6811
Add "use self-hosted runners" to CI setup (#38779)
potiuk Apr 5, 2024
b2efa5e
Clean up Breeze output for Docker context (#38857)
uranusjr Apr 9, 2024
f76b3cc
Fix mocking in kerberos integration test (#40042)
potiuk Jun 4, 2024
68671eb
Upgrade to FAB 4.5.0 (#39851)
potiuk May 27, 2024
911740a
Limit yandex provider to avoid mypy errors (#39990)
potiuk Jun 1, 2024
e7e54a2
Update version to 2.9.2
utkarsharma2 Jun 4, 2024
688a7ac
Update RELEASE_NOTES.rst
utkarsharma2 Jun 5, 2024
f56f134
Fix Apprise Mypy checks added in 1.8.0 (#39580)
potiuk May 13, 2024
412792c
Add dag run resource access control in dag level
joaopamaral Jul 9, 2024
1b91501
updates
joaopamaral Jul 10, 2024
3134eb6
Fix test role
joaopamaral Jul 10, 2024
45853f5
updates
joaopamaral Jul 10, 2024
44d0314
merge master
joaopamaral Jul 10, 2024
14e3d3e
merge master
joaopamaral Jul 10, 2024
680addf
merge main
joaopamaral Jul 10, 2024
12f07eb
format fix and simplify _sync_dag_view_permissions
joaopamaral Jul 11, 2024
4fdb95e
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Jul 11, 2024
b215fa8
Fix serialization tests after change access control type
joaopamaral Jul 11, 2024
abda160
Simplify some code
joaopamaral Jul 11, 2024
931ea95
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Jul 11, 2024
f9eebc0
fix /endpoints/test_dag_endpoint.py access_control sync_perm_for_dag arg
joaopamaral Jul 11, 2024
ca227d6
Merge remote-tracking branch 'joaopamaral/feature/dag-run-access-cont…
joaopamaral Jul 11, 2024
f1d4c82
capture invalid resource name
joaopamaral Jul 11, 2024
7597bcd
Fix mistyping
joaopamaral Jul 12, 2024
9e33a75
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Jul 12, 2024
1ced2f3
Update airflow/providers/fab/auth_manager/fab_auth_manager.py
joaopamaral Jul 12, 2024
6691c68
Keeping DAG_ACTIONS to keep the compatibility with outdated versions …
joaopamaral Jul 12, 2024
b5ce864
Merge remote-tracking branch 'joaopamaral/feature/dag-run-access-cont…
joaopamaral Jul 12, 2024
2e027f6
Add correct typing for access_control params
joaopamaral Jul 12, 2024
7686725
Add more asserts to test_replace_outdated_access_control_actions to v…
joaopamaral Jul 12, 2024
c7ebe69
Trying to clarify the description of access_control in sync_dag_view_…
joaopamaral Jul 12, 2024
e591267
revert changes in tests/system/providers/apache/iceberg/__init__.py
joaopamaral Jul 12, 2024
b2be841
revert changes in tests/system/providers/apache/iceberg/__init__.py
joaopamaral Jul 12, 2024
1e80c84
Keep compatibility running a new FAB in old airflow version
joaopamaral Jul 12, 2024
d105d07
Fix _resource_name param
joaopamaral Jul 12, 2024
1850856
fix apache/iceberg/__init__.py and apply static checks for override.py
shahar1 Jul 12, 2024
0b79459
Merge pull request #1 from shahar1/feature/dag-run-access-control
joaopamaral Jul 12, 2024
6b119f3
Keep compatibility in tests for old airflow versions
joaopamaral Jul 12, 2024
105c335
typing fix
joaopamaral Jul 12, 2024
bf027eb
fixes for static tests and compatibility tests
joaopamaral Jul 12, 2024
b8fa890
fix prefix map and include a dot in docstring end
joaopamaral Jul 12, 2024
c142dfb
fix static check
joaopamaral Jul 12, 2024
914448c
add licence to new permissions utils file
joaopamaral Jul 12, 2024
97daee4
add check new resource name method to keep compatibility
joaopamaral Jul 13, 2024
348dc27
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Jul 25, 2024
f182af9
Skip permissions with prefix used in dag level permission in admin role
joaopamaral Jul 25, 2024
bd3d4bb
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Jul 25, 2024
93290df
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Aug 5, 2024
14fb9e1
Add docs
joaopamaral Aug 5, 2024
27ba034
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Aug 5, 2024
68f1e1d
Increasing the count queries because this feature add new permissions…
joaopamaral Aug 6, 2024
0bd88aa
Merge branch 'main' into feature/dag-run-access-control
joaopamaral Aug 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 26 additions & 12 deletions airflow/models/dag.py
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,8 @@ class DAG(LoggingMixin):
that it is executed when the dag succeeds.
:param access_control: Specify optional DAG-level actions, e.g.,
"{'role1': {'can_read'}, 'role2': {'can_read', 'can_edit', 'can_delete'}}"
or it can specify the resource name if there is a DAGs Run resource, e.g.,
"{'role1': {'DAG Runs': {'can_create'}}, 'role2': {'DAGs': {'can_read', 'can_edit', 'can_delete'}}"
:param is_paused_upon_creation: Specifies if the dag is paused when created for the first time.
If the dag exists already, this flag will be ignored. If this optional parameter
is not specified, the global config setting will be used.
Expand Down Expand Up @@ -540,7 +542,7 @@ def __init__(
on_failure_callback: None | DagStateChangeCallback | list[DagStateChangeCallback] = None,
doc_md: str | None = None,
params: abc.MutableMapping | None = None,
access_control: dict | None = None,
access_control: dict[str, dict[str, Collection[str]]] | dict[str, Collection[str]] | None = None,
is_paused_upon_creation: bool | None = None,
jinja_environment_kwargs: dict | None = None,
render_template_as_native_obj: bool = False,
Expand Down Expand Up @@ -907,21 +909,33 @@ def _upgrade_outdated_dag_access_control(access_control=None):
"""
if access_control is None:
return None
new_perm_mapping = {
new_dag_perm_mapping = {
permissions.DEPRECATED_ACTION_CAN_DAG_READ: permissions.ACTION_CAN_READ,
permissions.DEPRECATED_ACTION_CAN_DAG_EDIT: permissions.ACTION_CAN_EDIT,
}

def update_old_perm(permission: str):
new_perm = new_dag_perm_mapping.get(permission, permission)
if new_perm != permission:
warnings.warn(
f"The '{permission}' permission is deprecated. Please use '{new_perm}'.",
RemovedInAirflow3Warning,
stacklevel=3,
)
joaopamaral marked this conversation as resolved.
Show resolved Hide resolved
return new_perm

updated_access_control = {}
for role, perms in access_control.items():
updated_access_control[role] = {new_perm_mapping.get(perm, perm) for perm in perms}

if access_control != updated_access_control:
warnings.warn(
"The 'can_dag_read' and 'can_dag_edit' permissions are deprecated. "
"Please use 'can_read' and 'can_edit', respectively.",
RemovedInAirflow3Warning,
stacklevel=3,
)
updated_access_control[role] = updated_access_control.get(role, {})
if isinstance(perms, (set, list)):
# Support for old-style access_control where only the actions are specified
updated_access_control[role][permissions.RESOURCE_DAG] = set(perms)
else:
updated_access_control[role] = perms
if permissions.RESOURCE_DAG in updated_access_control[role]:
updated_access_control[role][permissions.RESOURCE_DAG] = {
update_old_perm(perm) for perm in updated_access_control[role][permissions.RESOURCE_DAG]
}

return updated_access_control

Expand Down Expand Up @@ -4125,7 +4139,7 @@ def dag(
on_failure_callback: None | DagStateChangeCallback | list[DagStateChangeCallback] = None,
doc_md: str | None = None,
params: abc.MutableMapping | None = None,
access_control: dict | None = None,
access_control: dict[str, dict[str, Collection[str]]] | dict[str, Collection[str]] | None = None,
is_paused_upon_creation: bool | None = None,
jinja_environment_kwargs: dict | None = None,
render_template_as_native_obj: bool = False,
Expand Down
41 changes: 33 additions & 8 deletions airflow/providers/fab/auth_manager/fab_auth_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@
RESOURCE_DAG,
RESOURCE_DAG_CODE,
RESOURCE_DAG_DEPENDENCIES,
RESOURCE_DAG_PREFIX,
RESOURCE_DAG_RUN,
RESOURCE_DAG_WARNING,
RESOURCE_DATASET,
Expand Down Expand Up @@ -242,6 +241,8 @@ def is_authorized_dag(

return all(
self._is_authorized(method=method, resource_type=resource_type, user=user)
if resource_type != RESOURCE_DAG_RUN or not hasattr(permissions, "resource_name")
else self._is_authorized_dag_run(method=method, details=details, user=user)
for resource_type in resource_types
)

Expand Down Expand Up @@ -412,7 +413,33 @@ def _is_authorized_dag(

if details and details.id:
# Check whether the user has permissions to access a specific DAG
resource_dag_name = self._resource_name_for_dag(details.id)
resource_dag_name = self._resource_name(details.id, RESOURCE_DAG)
return self._is_authorized(method=method, resource_type=resource_dag_name, user=user)

return False

def _is_authorized_dag_run(
self,
method: ResourceMethod,
details: DagDetails | None = None,
user: BaseUser | None = None,
) -> bool:
"""
Return whether the user is authorized to perform a given action on a DAG Run.

:param method: the method to perform
:param details: optional, details about the DAG
:param user: optional, the user to perform the action on. If not provided, it uses the current user

:meta private:
"""
is_global_authorized = self._is_authorized(method=method, resource_type=RESOURCE_DAG_RUN, user=user)
if is_global_authorized:
return True

if details and details.id:
# Check whether the user has permissions to access a specific DAG Run permission on a DAG Level
resource_dag_name = self._resource_name(details.id, RESOURCE_DAG_RUN)
return self._is_authorized(method=method, resource_type=resource_dag_name, user=user)

return False
Expand Down Expand Up @@ -444,7 +471,7 @@ def _get_fab_resource_types(dag_access_entity: DagAccessEntity) -> tuple[str, ..
raise AirflowException(f"Unknown DAG access entity: {dag_access_entity}")
return _MAP_DAG_ACCESS_ENTITY_TO_FAB_RESOURCE_TYPE[dag_access_entity]

def _resource_name_for_dag(self, dag_id: str) -> str:
def _resource_name(self, dag_id: str, resource_type: str) -> str:
"""
Return the FAB resource name for a DAG id.

Expand All @@ -453,11 +480,9 @@ def _resource_name_for_dag(self, dag_id: str) -> str:
:meta private:
"""
root_dag_id = self._get_root_dag_id(dag_id)
if root_dag_id == RESOURCE_DAG:
return root_dag_id
if root_dag_id.startswith(RESOURCE_DAG_PREFIX):
return root_dag_id
return f"{RESOURCE_DAG_PREFIX}{root_dag_id}"
if hasattr(permissions, "resource_name"):
return getattr(permissions, "resource_name")(root_dag_id, resource_type)
return getattr(permissions, "resource_name_for_dag")(root_dag_id)

@staticmethod
def _get_user_permissions(user: BaseUser):
Expand Down
158 changes: 100 additions & 58 deletions airflow/providers/fab/auth_manager/security_manager/override.py
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,16 @@ class FabAirflowSecurityManagerOverride(AirflowSecurityManagerV2):
]

# global resource for dag-level access
DAG_ACTIONS = permissions.DAG_ACTIONS
RESOURCE_DETAILS_MAP = getattr(
permissions,
"RESOURCE_DETAILS_MAP",
{
permissions.RESOURCE_DAG: {
"actions": permissions.DAG_ACTIONS,
}
},
)
DAG_ACTIONS = RESOURCE_DETAILS_MAP[permissions.RESOURCE_DAG]["actions"]

def __init__(self, appbuilder):
# done in super, but we need it before we can call super.
Expand Down Expand Up @@ -1031,7 +1040,7 @@ def can_access_some_dags(self, action: str, dag_id: str | None = None) -> bool:
"""Check if user has read or write access to some dags."""
if dag_id and dag_id != "~":
root_dag_id = self._get_root_dag_id(dag_id)
return self.has_access(action, permissions.resource_name_for_dag(root_dag_id))
return self.has_access(action, self._resource_name(root_dag_id, permissions.RESOURCE_DAG))

user = g.user
if action == permissions.ACTION_CAN_READ:
Expand Down Expand Up @@ -1065,24 +1074,25 @@ def create_dag_specific_permissions(self) -> None:

for dag in dags:
root_dag_id = dag.parent_dag.dag_id if dag.parent_dag else dag.dag_id
dag_resource_name = permissions.resource_name_for_dag(root_dag_id)
for action_name in self.DAG_ACTIONS:
if (action_name, dag_resource_name) not in perms:
self._merge_perm(action_name, dag_resource_name)
for resource_name, resource_values in self.RESOURCE_DETAILS_MAP.items():
dag_resource_name = self._resource_name(root_dag_id, resource_name)
for action_name in resource_values["actions"]:
if (action_name, dag_resource_name) not in perms:
self._merge_perm(action_name, dag_resource_name)

if dag.access_control is not None:
self.sync_perm_for_dag(dag_resource_name, dag.access_control)
self.sync_perm_for_dag(root_dag_id, dag.access_control)

def prefixed_dag_id(self, dag_id: str) -> str:
"""Return the permission name for a DAG id."""
warnings.warn(
"`prefixed_dag_id` has been deprecated. "
"Please use `airflow.security.permissions.resource_name_for_dag` instead.",
"Please use `airflow.security.permissions.resource_name` instead.",
RemovedInAirflow3Warning,
stacklevel=2,
)
root_dag_id = self._get_root_dag_id(dag_id)
return permissions.resource_name_for_dag(root_dag_id)
return self._resource_name(root_dag_id, permissions.RESOURCE_DAG)

def is_dag_resource(self, resource_name: str) -> bool:
"""Determine if a resource belongs to a DAG or all DAGs."""
Expand All @@ -1093,90 +1103,121 @@ def is_dag_resource(self, resource_name: str) -> bool:
def sync_perm_for_dag(
self,
dag_id: str,
access_control: dict[str, Collection[str]] | None = None,
joaopamaral marked this conversation as resolved.
Show resolved Hide resolved
access_control: dict[str, dict[str, Collection[str]]] | None = None,
) -> None:
"""
Sync permissions for given dag id.

The dag id surely exists in our dag bag as only / refresh button or DagBag will call this function.

:param dag_id: the ID of the DAG whose permissions should be updated
:param access_control: a dict where each key is a role name and
each value is a set() of action names (e.g.,
{'can_read'}
:param access_control: a dict where each key is a role name and each value can be:
- a set() of DAGs resource action names (e.g. `{'can_read'}`)
- or a dict where each key is a resource name ('DAGs' or 'DAG Runs') and each value
is a set() of action names (e.g., `{'DAG Runs': {'can_create'}, 'DAGs': {'can_read'}}`)
:return:
"""
dag_resource_name = permissions.resource_name_for_dag(dag_id)
for dag_action_name in self.DAG_ACTIONS:
self.create_permission(dag_action_name, dag_resource_name)
for resource_name, resource_values in self.RESOURCE_DETAILS_MAP.items():
dag_resource_name = self._resource_name(dag_id, resource_name)
for dag_action_name in resource_values["actions"]:
self.create_permission(dag_action_name, dag_resource_name)

if access_control is not None:
self.log.debug("Syncing DAG-level permissions for DAG '%s'", dag_resource_name)
self._sync_dag_view_permissions(dag_resource_name, access_control)
self.log.debug("Syncing DAG-level permissions for DAG '%s'", dag_id)
self._sync_dag_view_permissions(dag_id, access_control.copy())
else:
self.log.debug(
"Not syncing DAG-level permissions for DAG '%s' as access control is unset.",
dag_resource_name,
dag_id,
)

def _sync_dag_view_permissions(self, dag_id: str, access_control: dict[str, Collection[str]]) -> None:
def _resource_name(self, dag_id: str, resource_name: str) -> str:
"""
Get the resource name from permissions.

This method is to keep compatibility with new FAB versions
running with old airflow versions.
"""
if hasattr(permissions, "resource_name"):
return getattr(permissions, "resource_name")(dag_id, resource_name)
return getattr(permissions, "resource_name_for_dag")(dag_id)

def _sync_dag_view_permissions(
self,
dag_id: str,
access_control: dict[str, dict[str, Collection[str]]],
) -> None:
"""
Set the access policy on the given DAG's ViewModel.

:param dag_id: the ID of the DAG whose permissions should be updated
:param access_control: a dict where each key is a role name and
each value is a set() of action names (e.g. {'can_read'})
:param access_control: a dict where each key is a role name and each value is:
- a dict where each key is a resource name ('DAGs' or 'DAG Runs') and each value
is a set() of action names (e.g., `{'DAG Runs': {'can_create'}, 'DAGs': {'can_read'}}`)
"""
dag_resource_name = permissions.resource_name_for_dag(dag_id)

def _get_or_create_dag_permission(action_name: str) -> Permission | None:
def _get_or_create_dag_permission(action_name: str, dag_resource_name: str) -> Permission | None:
perm = self.get_permission(action_name, dag_resource_name)
if not perm:
self.log.info("Creating new action '%s' on resource '%s'", action_name, dag_resource_name)
perm = self.create_permission(action_name, dag_resource_name)

return perm

def _revoke_stale_permissions(resource: Resource):
existing_dag_perms = self.get_resource_permissions(resource)
for perm in existing_dag_perms:
non_admin_roles = [role for role in perm.role if role.name != "Admin"]
for role in non_admin_roles:
target_perms_for_role = access_control.get(role.name, ())
if perm.action.name not in target_perms_for_role:
self.log.info(
"Revoking '%s' on DAG '%s' for role '%s'",
perm.action,
dag_resource_name,
role.name,
)
self.remove_permission_from_role(role, perm)

resource = self.get_resource(dag_resource_name)
if resource:
_revoke_stale_permissions(resource)

for rolename, action_names in access_control.items():
# Revoking stale permissions for all possible DAG level resources
for resource_name in self.RESOURCE_DETAILS_MAP.keys():
dag_resource_name = self._resource_name(dag_id, resource_name)
if resource := self.get_resource(dag_resource_name):
existing_dag_perms = self.get_resource_permissions(resource)
for perm in existing_dag_perms:
non_admin_roles = [role for role in perm.role if role.name != "Admin"]
for role in non_admin_roles:
target_perms_for_role = access_control.get(role.name, {}).get(resource_name, set())
if perm.action.name not in target_perms_for_role:
self.log.info(
"Revoking '%s' on DAG '%s' for role '%s'",
perm.action,
dag_resource_name,
role.name,
)
self.remove_permission_from_role(role, perm)

# Adding the access control permissions
for rolename, resource_actions in access_control.items():
role = self.find_role(rolename)
if not role:
raise AirflowException(
f"The access_control mapping for DAG '{dag_id}' includes a role named "
f"'{rolename}', but that role does not exist"
)

action_names = set(action_names)
invalid_action_names = action_names - self.DAG_ACTIONS
if invalid_action_names:
raise AirflowException(
f"The access_control map for DAG '{dag_resource_name}' includes "
f"the following invalid permissions: {invalid_action_names}; "
f"The set of valid permissions is: {self.DAG_ACTIONS}"
)
if isinstance(resource_actions, (set, list)):
# Support for old-style access_control where only the actions are specified
resource_actions = {permissions.RESOURCE_DAG: set(resource_actions)}

for resource_name, actions in resource_actions.items():
if resource_name not in self.RESOURCE_DETAILS_MAP:
raise AirflowException(
f"The access_control map for DAG '{dag_id}' includes the following invalid "
f"resource name: '{resource_name}'; "
f"The set of valid resource names is: {self.RESOURCE_DETAILS_MAP.keys()}"
)

dag_resource_name = self._resource_name(dag_id, resource_name)
self.log.debug("Syncing DAG-level permissions for DAG '%s'", dag_resource_name)

invalid_actions = set(actions) - self.RESOURCE_DETAILS_MAP[resource_name]["actions"]

if invalid_actions:
raise AirflowException(
f"The access_control map for DAG '{dag_resource_name}' includes "
f"the following invalid permissions: {invalid_actions}; "
f"The set of valid permissions is: {self.RESOURCE_DETAILS_MAP[resource_name]['actions']}"
)

for action_name in action_names:
dag_perm = _get_or_create_dag_permission(action_name)
if dag_perm:
self.add_permission_to_role(role, dag_perm)
for action_name in actions:
dag_perm = _get_or_create_dag_permission(action_name, dag_resource_name)
if dag_perm:
self.add_permission_to_role(role, dag_perm)

def add_permissions_view(self, base_action_names, resource_name): # Keep name for compatibility with FAB.
"""
Expand Down Expand Up @@ -1307,8 +1348,9 @@ def update_admin_permission(self) -> None:
Add the missing ones to the table for admin.
"""
session = self.appbuilder.get_session
prefixes = getattr(permissions, "PREFIX_LIST", [permissions.RESOURCE_DAG_PREFIX])
dag_resources = session.scalars(
select(Resource).where(Resource.name.like(f"{permissions.RESOURCE_DAG_PREFIX}%"))
select(Resource).where(or_(*[Resource.name.like(f"{prefix}%") for prefix in prefixes]))
)
resource_ids = [resource.id for resource in dag_resources]

Expand Down
Loading