-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support restricted index patterns in Elasticsearch log handler #23888
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
@jedcunningham hi! I am have run airflow with elasticsearch logging and it is solving the issue for us. We configured a role with restricted access to a specific index pattern, and where before Airflow could not read the logs, with the patch it could. Appreciate it if you could take a look. |
8d648fd
to
44caca6
Compare
tests/providers/elasticsearch/log/elasticmock/fake_elasticsearch.py
Outdated
Show resolved
Hide resolved
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
8aef878
to
3d42e66
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs some changes to support backwards compatibility. We can't set index_patterns
in airflow_local_settings
, because we might have an older provider. We also should try and get the config value when on older core versions. Look at how es_kwargs
is handled - that is a good example.
Otherwise, this looks good.
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
@kouk are you still working on this PR? |
Yeah, I was on vacation..will look at this today.
…On Mon, Aug 15, 2022, 12:38 eladkal ***@***.***> wrote:
@kouk <https://github.com/kouk> are you still working on this PR?
—
Reply to this email directly, view it on GitHub
<#23888 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADPKRZ56Z75UETF35ZX273VZIF2JANCNFSM5WZLICWQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
91a33bd
to
4eb1ce2
Compare
4eb1ce2
to
683bf27
Compare
@jedcunningham - I think that one waits for you :) |
You will need to rebase @kouk to account for recent changes in main. |
I think it will have to be 2.5.0 - > we are following semver and we only add bugfixes to "patchlevel" versions. This one will have to wait for 2.5 to be released. |
Marking this for 2.5.0, but after the bulk of these changes land in a provider release, it'll work if you set the config. |
1c832ed
to
eedfa36
Compare
@jedcunningham thanks for pointing that out. I wasn't familiar enough with the config code to realize that it's ConfigParser under the hood. I've updated both the version and the config handling code like you suggested. |
@eladkal @jedcunningham anything else we need here? |
011d935
to
310ca97
Compare
@jedcunningham appreciate if you could check if anything else is needed here. |
Sometimes Airflow doesn't have the ability to search across all indices in an Elasticsearch server. This might be due to security settings in the server. In these cases fetching the remote logs fails. To fix this we create a index_patterns configuration setting that can be set to a more restrictive pattern.
310ca97
to
81d7f81
Compare
Rebased to be sure. @jedcunningham - want to still take a look? |
Awesome work, congrats on your first merged pull request! |
Thanks @kouk! Congrats on your first commit 🎉. (I apologize for the delay) As I mentioned earlier, since almost all of the changes are in the provider, this feature will be available once it's released in the provider (even before 2.6 with the config option is released). |
thank you @jedcunningham and everyone else. |
Sometimes Airflow doesn't have the ability to search across all indices
in an Elasticsearch server. This might be due to security settings in
the server. In these cases fetching the remote logs fails. To fix this
we create a index_patterns configuration setting that can be set to a
more restrictive pattern.
closes: #16828