Update python.rst with multi-Scheduler LocalExecutor info #32310
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR updates the PythonOperator section of the documentation to include information about how the LocalExecutor works when multiple Airflow Schedulers are in use. In the updated documentation, I have added a note that explains: Each Scheduler, when configured with executor = LocalExecutor, runs its own LocalExecutor. Tasks are processed in a distributed fashion across the machines running the Schedulers. It takes into consideration the scenario when a scheduler is restarted. It also explains the limitations on sharing of instances across tasks with multiple Schedulers. This information was not clear in the existing documentation, and this update should help users understand how LocalExecutor works in a multi-Scheduler setup.
boring-cyborg
bot
added
area:core-operators
potiuk
reviewed
Jul 2, 2023
| @@ -28,6 +28,16 @@ Use the ``@task`` decorator to execute Python callables. | |||
| The ``@task`` decorator is recommended over the classic :class:`~airflow.operators.python.PythonOperator` | |||
| to execute Python callables. | |||
|
|
|||
| .. note:: | |||
There was a problem hiding this comment.
Why here? It is importand and valid potint to mention but it works the same way for every other operator, so I guess it should be explained in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html#local-executor
There was a problem hiding this comment.
@potiuk I missed that. I have made the changes in local.rst now. Let me know if that looks good
There was a problem hiding this comment.
@potiuk Updated now
pankajkoti
reviewed
Jul 2, 2023
|
|
||
| .. note:: | ||
|
|
||
| When using the ``@task`` decorator with multiple Schedulers configured with ``executor = LocalExecutor`` in the [core] section of airflow.cfg, each scheduler will run a LocalExecutor. This means tasks would be processed in a distributed fashion across the machines running the Schedulers. |
There was a problem hiding this comment.
My understanding is that it is general
There was a problem hiding this comment.
@pankajkoti Updated and done
uranusjr
reviewed
Jul 3, 2023
uranusjr
reviewed
Jul 3, 2023
raphaelauv
reviewed
Jul 4, 2023
potiuk
approved these changes
Jul 5, 2023
uranusjr
approved these changes
Jul 5, 2023
ephraimbuddy
pushed a commit
that referenced
this pull request
Jul 6, 2023
Co-authored-by: Pankaj Koti <[email protected]> (cherry picked from commit e53320d)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the PythonOperator section of the documentation to include information about how the LocalExecutor works when multiple Airflow Schedulers are in use.
In the updated documentation, I have added a note that explains:
Each Scheduler, when configured with executor = LocalExecutor, runs its own LocalExecutor. Tasks are processed in a distributed fashion across the machines running the Schedulers. It takes into consideration the scenario when a scheduler is restarted. It also explains the limitations on sharing of instances across tasks with multiple Schedulers. This information was not clear in the existing documentation, and this update should help users understand how LocalExecutor works in a multi-Scheduler setup.
closes: #26565
related: #26565
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.