Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync v2 1 2 stable v2 #16862

Merged
merged 7 commits into from
Jul 9, 2021
Merged

Sync v2 1 2 stable v2 #16862

merged 7 commits into from
Jul 9, 2021

Conversation

jhtimmins
Copy link
Contributor

Sync remaining commits into v2-1-2

@boring-cyborg boring-cyborg bot added area:logging area:Scheduler including HA (high availability) scheduler area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues labels Jul 7, 2021
@github-actions
Copy link

github-actions bot commented Jul 7, 2021

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jul 7, 2021
@potiuk
Copy link
Member

potiuk commented Jul 7, 2021

Might also make sense to run constraints upgrade. I will run it

@potiuk
Copy link
Member

potiuk commented Jul 7, 2021

BTW. Something wrong with airflow.compat missing (both pylint and tests)

@kaxil kaxil marked this pull request as draft July 7, 2021 20:08
@potiuk
Copy link
Member

potiuk commented Jul 7, 2021

Seems like this one #16700 brings airflow.compat ?

@potiuk
Copy link
Member

potiuk commented Jul 7, 2021

i think cherry picking 3db347e and 866a601 should bring the compat change and remove pylint (some more cherry-picks might be needed)

@potiuk
Copy link
Member

potiuk commented Jul 7, 2021

BTW. Constraints upgraded for constraints-2-1 branch - following https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#manually-generating-constraint-files

@jhtimmins jhtimmins force-pushed the sync-v2-1-2-stable-v2 branch from 258ab59 to c258646 Compare July 9, 2021 18:10
kaxil and others added 7 commits July 9, 2021 11:11
@kaxil @jhtimmins
Fix CeleryKubernetesExecutor (apache#16700)
b350481 
closes apache#16326

Currently when running celery tasks when running with ``CeleryKubernetesExecutor``,
we see the following error. This error occurs as the ``BaseJob`` (via ``LocalTaskJob``) tries to needlessly
instantiate a `KubernetesExecutor` which in turn tries to create a multiprocessing process/Manager
which fails.

```
[2021-06-29 00:23:45,301: ERROR/ForkPoolWorker-16] Failed to execute task daemonic processes are not allowed to have children.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 116, in _execute_in_fork
    args.func(args)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 91, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 237, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
    _run_task_by_local_task_job(args, ti)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 117, in _run_task_by_local_task_job
    pool=args.pool,
  File "<string>", line 4, in __init__
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/local_task_job.py", line 76, in __init__
    super().__init__(*args, **kwargs)
  File "<string>", line 6, in __init__
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 97, in __init__
    self.executor = executor or ExecutorLoader.get_default_executor()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/executor_loader.py", line 62, in get_default_executor
    cls._default_executor = cls.load_executor(executor_name)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/executor_loader.py", line 79, in load_executor
    return cls.__load_celery_kubernetes_executor()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/executor_loader.py", line 116, in __load_celery_kubernetes_executor
    kubernetes_executor = import_string(cls.executors[KUBERNETES_EXECUTOR])()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py", line 421, in __init__
    self._manager = multiprocessing.Manager()
  File "/usr/local/lib/python3.6/multiprocessing/context.py", line 56, in Manager
    m.start()
  File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 513, in start
    self._process.start()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
```

We don't need to instantiate an executor when running ``LocalTaskJob`` as executor isn't used in it.

(cherry picked from commit 7857a9b)
@bbovenzi @jhtimmins
yarn audit (apache#16440)
79399c0 
Bump a lot of npm modules in packages.json to resolve all severe and moderate vulnerabilities found when using `yarn audit`

(cherry picked from commit f18e4ba)
@robertsaxby @jhtimmins
Mask value if the key is token (apache#16474)
0a52997 
Some connections (including the databricks connection) use the key 'token' in the 'extra' field (this has always been the case). Including it here so that these sensitive tokens are also masked by default.

The prior implementation just masked all of the 'extra' json: "XXXXXXXX" if conn.extra_dejson else None https://github.com/apache/airflow/blob/88199eefccb4c805f8d6527bab5bf600b397c35e/airflow/hooks/base.py#L78

(cherry picked from commit d1d04fe)
@ephraimbuddy @jhtimmins
Fix impersonation issue with LocalTaskJob (apache#16852)
3d67a75 
Running a task with run_as_user fails because PIDs are not matched
correctly.

This change fixes it by matching the parent process ID (the `sudo`
process) of the task instance to the current process ID of the task_runner
process when we use impersonation

Co-authored-by: Ash Berlin-Taylor <[email protected]>
(cherry picked from commit feea380)
@jhtimmins
Update release date and changelog for 2.1.2.
93aab40
@uranusjr @jhtimmins
Introduce compat shim airflow.compat.functools (apache#15969)
bcbe65e 
This module shims 'cached_property' and 'cache' so modules don't need to
all do their own ad-hoc try-except ImportError.

(cherry picked from commit 3db347e)
@potiuk @jhtimmins
Removes pylint from our toolchain (apache#16682)
fe060b5 
We've agreed during the voting process that Pylint support
should be disabled: https://lists.apache.org/thread.html/r9e2cc385db8737ec0874ad09872081bd083593ee29e8303e58d21efb%40%3Cdev.airflow.apache.org%3E

This PR:

* removes all # pylint comments
* removes pylint pre-commits and related scripts/files
* removes CI jobs running pylint checks
* removes documentation about pylint
* removes unnecessary #noga (adds pre-commit for that)
* fixes some remaining pydocstyle errors after removing #noqa's

(cherry picked from commit 866a601)
@jhtimmins jhtimmins force-pushed the sync-v2-1-2-stable-v2 branch from c258646 to fe060b5 Compare July 9, 2021 18:12
@jhtimmins jhtimmins closed this Jul 9, 2021
@jhtimmins jhtimmins reopened this Jul 9, 2021
@jhtimmins jhtimmins marked this pull request as ready for review July 9, 2021 19:56
@jhtimmins jhtimmins merged commit 06df29e into apache:v2-1-stable Jul 9, 2021
@jhtimmins jhtimmins deleted the sync-v2-1-2-stable-v2 branch July 9, 2021 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaxil kaxil kaxil approved these changes

@potiuk potiuk potiuk approved these changes

@ashb ashb ashb approved these changes

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@dimberman dimberman Awaiting requested review from dimberman dimberman is a code owner

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@mik-laj mik-laj Awaiting requested review from mik-laj mik-laj is a code owner

@turbaszek turbaszek Awaiting requested review from turbaszek turbaszek is a code owner

@vikramkoka vikramkoka Awaiting requested review from vikramkoka vikramkoka is a code owner

Assignees
No one assigned
Labels
area:logging area:Scheduler including HA (high availability) scheduler area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@jhtimmins @potiuk @ashb @kaxil @bbovenzi @robertsaxby @ephraimbuddy @uranusjr