WASB remote logging too verbose in task logger

[flolas]

Apache Airflow version: 2.1.0

Environment:
apache-airflow-providers-microsoft-azure == 2.0.0

What happened:

When wasb remote logger wants to create container (always), the log is part of the task logger.

[2021-06-02 13:55:51,619] {wasb.py:385} INFO - Attempting to create container:  xxxxx
[2021-06-02 13:55:51,479] {_universal.py:419} INFO - Request URL: 'XXXXXXX'
[2021-06-02 13:55:51,483] {_universal.py:420} INFO - Request method: 'HEAD'
[2021-06-02 13:55:51,490] {_universal.py:421} INFO - Request headers:
[2021-06-02 13:55:51,495] {_universal.py:424} INFO -     'x-ms-version': 'REDACTED'
[2021-06-02 13:55:51,499] {_universal.py:424} INFO -     'Accept': 'application/xml'
[2021-06-02 13:55:51,507] {_universal.py:424} INFO -     'User-Agent': 'azsdk-python-storage-blob/12.8.1 Python/3.7.10 (Linux-5.4.0-1046-azure-x86_64-with-debian-10.9)'
[2021-06-02 13:55:51,511] {_universal.py:424} INFO -     'x-ms-date': 'REDACTED'
[2021-06-02 13:55:51,517] {_universal.py:424} INFO -     'x-ms-client-request-id': ''
[2021-06-02 13:55:51,523] {_universal.py:424} INFO -     'Authorization': 'REDACTED'
[2021-06-02 13:55:51,529] {_universal.py:437} INFO - No body was attached to the request
[2021-06-02 13:55:51,541] {_universal.py:452} INFO - Response status: 404
[2021-06-02 13:55:51,550] {_universal.py:453} INFO - Response headers:
[2021-06-02 13:55:51,556] {_universal.py:456} INFO -     'Transfer-Encoding': 'chunked'
[2021-06-02 13:55:51,561] {_universal.py:456} INFO -     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
[2021-06-02 13:55:51,567] {_universal.py:456} INFO -     'x-ms-request-id': 'xxxx'
[2021-06-02 13:55:51,573] {_universal.py:456} INFO -     'x-ms-client-request-id': 'xxxx'
[2021-06-02 13:55:51,578] {_universal.py:456} INFO -     'x-ms-version': 'REDACTED'
[2021-06-02 13:55:51,607] {_universal.py:456} INFO -     'x-ms-error-code': 'REDACTED'
[2021-06-02 13:55:51,613] {_universal.py:456} INFO -     'Date': 'Wed, 02 Jun 2021 13:55:50 GMT'

What you expected to happen:
HTTP verbose logging not showing and "Attempting to create container" maybe it's not useful in the task log

ref: Azure/azure-sdk-for-python#9422

[flolas]

based on Azure/azure-sdk-for-python#9422

maybe a solution could be adding

logger = logging.getLogger("azure.core.pipeline.policies.http_logging_policy")
logger.setLevel(logging.WARNING)

to wasb.py file

[uranusjr]

I’m thinking mayne it’d make sense to set the entire azure namespace to WARNING. This can be better done by modifying airflow.config_templates_airflow_local_settings.DEFAULT_LOGGING_CONFIG (and we will likely need a way to enable these logs back on, see https://issues.apache.org/jira/browse/AIRFLOW-1455 (it’s closed as WONTFIX but there is one person wanting to see more verbose logs).