fix(helm): safer worker pod annotations

[hakuno]

Hey, you all. How are you doing?

It's an initiative of #35130

This PR brings two things:

1. Values workers.podAnnotations and airflowPodAnnotations won't fight each other anymore. Value workers.podAnnotations will have precedence over airflowPodAnnotations value. Think of local scope and global scope (fallback). So specific settings can override generic settings on component side.
2. Value workers.safeToEvict will be embedded into pod annotations. And workers.podAnnotations will have precedence over, so it keeps retrocompatibility on any workaround done in past by users.

Tests

test_safetoevict_annotations (refactored)

About the safeToEvict use cases. I did a "full coverage" of following scenarios:

$ pytest helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations

=================================================================== test session starts ====================================================================
platform linux -- Python 3.8.18, pytest-7.4.3, pluggy-1.3.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow
configfile: pyproject.toml
plugins: requests-mock-1.11.0, xdist-3.3.1, anyio-4.0.0, instafail-0.5.0, capture-warnings-0.0.4, rerunfailures-12.0, timeouts-1.2.1, asyncio-0.21.1, mock-3.12.0, httpx-0.21.3, time-machine-2.13.0, cov-4.1.0
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 19 items

helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope0-localScope0-true] PASSED                               [  5%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope1-localScope1-true] PASSED                               [ 10%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope2-localScope2-false] PASSED                              [ 15%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope3-localScope3-true] PASSED                               [ 21%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope4-localScope4-true] PASSED                               [ 26%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope5-localScope5-false] PASSED                              [ 31%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope6-localScope6-false] PASSED                              [ 36%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope7-localScope7-true] PASSED                               [ 42%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope8-localScope8-false] PASSED                              [ 47%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope9-localScope9-true] PASSED                               [ 52%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope10-localScope10-false] PASSED                            [ 57%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope11-localScope11-true] PASSED                             [ 63%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope12-localScope12-false] PASSED                            [ 68%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope13-localScope13-true] PASSED                             [ 73%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope14-localScope14-false] PASSED                            [ 78%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope15-localScope15-true] PASSED                             [ 84%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope16-localScope16-false] PASSED                            [ 89%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope17-localScope17-true] PASSED                             [ 94%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_safetoevict_annotations[globalScope18-localScope18-false] PASSED                            [100%]

============================================================== 19 passed in 121.07s (0:02:01) ==============================================================

test_podannotations_precedence

About the pod annotations precedence.

$ pytest helm_tests/airflow_core/test_worker.py::TestWorker::test_podannotations_precedence

=================================================================== test session starts ====================================================================
platform linux -- Python 3.8.18, pytest-7.4.3, pluggy-1.3.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow
configfile: pyproject.toml
plugins: requests-mock-1.11.0, xdist-3.3.1, anyio-4.0.0, instafail-0.5.0, capture-warnings-0.0.4, rerunfailures-12.0, timeouts-1.2.1, asyncio-0.21.1, mock-3.12.0, httpx-0.21.3, time-machine-2.13.0, cov-4.1.0
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 4 items

helm_tests/airflow_core/test_worker.py::TestWorker::test_podannotations_precedence[globalScope0-localScope0-global] PASSED                           [ 25%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_podannotations_precedence[globalScope1-localScope1-local] PASSED                            [ 50%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_podannotations_precedence[globalScope2-localScope2-local] PASSED                            [ 75%]
helm_tests/airflow_core/test_worker.py::TestWorker::test_podannotations_precedence[globalScope3-localScope3-None] PASSED                             [100%]

==================================================================== 4 passed in 25.72s ====================================================================

Thanks in advance!

[hussein-awala]

It looks good! The failed test will be fixed by #35313; you need to merge/rebase main once the other PR is merged.

[hakuno]

It looks good! The failed test will be fixed by #35313; you need to merge/rebase main once the other PR is merged.

All right. Thank you so!