Add logging signal to main (#2251)

* Add initial overall structure and classes for logs sdk (#1894)

* Add global LogEmitterProvider and convenience function get_log_emitter (#1901)

* Add OTLPHandler for standard library logging module (#1903)

* Add LogProcessors implementation (#1916)

* Fix typos in test_handler.py (#1953)

* Add support for OTLP Log exporter (#1943)

* Add support for user defined attributes in OTLPHandler (#1952)

* use timeout in force_flush (#2118)

* use timeout in force_flush

* fix lint

* Update opentelemetry-sdk/src/opentelemetry/sdk/logs/export/__init__.py

Co-authored-by: Srikanth Chekuri <[email protected]>

* fix lint

Co-authored-by: Srikanth Chekuri <[email protected]>

* add a ConsoleExporter for logging (#2099)

Co-authored-by: Srikanth Chekuri <[email protected]>

* Update SDK docs and Add example with OTEL collector logging (debug) exporter (#2050)

* Fix exception in severity number transformation (#2208)

* Fix exception with warning message transformation

* Fix lint

* Fix lint

* fstring

* Demonstrate how to set the Resource for LogEmitterProvider (#2209)

* Demonstrate how to set the Resource for LogEmitterProvider

Added a Resource to the logs example to make it more complete.
Previously it was using the built-in Resource. Now it adds the
service.name and service.instance.id attributes.

The resulting emitted log records look like this:
```
Resource labels:
     -> telemetry.sdk.language: STRING(python)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(1.5.0)
     -> service.name: STRING(shoppingcart)
     -> service.instance.id: STRING(instance-12)
InstrumentationLibraryLogs #0
InstrumentationLibrary __main__ 0.1
LogRecord #0
Timestamp: 2021-10-14 18:33:43.425820928 +0000 UTC
Severity: ERROR
ShortName:
Body: Hyderabad, we have a major problem.
Trace ID: ce1577e4a703f42d569e72593ad71888
Span ID: f8908ac4258ceff6
Flags: 1
```

* Fix linting

* Use batch processor in example (#2225)

* move logs to _logs (#2240)

* move logs to _logs

* fix lint

* move log_exporter to _log_exporter as it's still experimental (#2252)

Co-authored-by: Srikanth Chekuri <[email protected]>
Co-authored-by: Adrian Garcia Badaracco <[email protected]>
Co-authored-by: Leighton Chen <[email protected]>
Co-authored-by: Tigran Najaryan <[email protected]>
Co-authored-by: Owais Lone <[email protected]>

CHANGELOG.md

@@ -115,6 +115,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added dropped count to otlp, jaeger and zipkin exporters.
   ([#1893](https://github.com/open-telemetry/opentelemetry-python/pull/1893))
 
+### Added
+- Give OTLPHandler the ability to process attributes
+  ([#1952](https://github.com/open-telemetry/opentelemetry-python/pull/1952))
+- Add global LogEmitterProvider and convenience function get_log_emitter
+  ([#1901](https://github.com/open-telemetry/opentelemetry-python/pull/1901))
+- Add OTLPHandler for standard library logging module
+  ([#1903](https://github.com/open-telemetry/opentelemetry-python/pull/1903))
+
 ### Changed
 - Updated `opentelemetry-opencensus-exporter` to use `service_name` of spans instead of resource
   ([#1897](https://github.com/open-telemetry/opentelemetry-python/pull/1897))

docs/examples/logs/README.rst

@@ -0,0 +1,75 @@
+OpenTelemetry Logs SDK
+======================
+
+Start the Collector locally to see data being exported. Write the following file:
+
+.. code-block:: yaml
+
+    # otel-collector-config.yaml
+    receivers:
+        otlp:
+            protocols:
+                grpc:
+  
+    exporters:
+        logging:
+  
+    processors:
+        batch:
+  
+Then start the Docker container:
+
+.. code-block:: sh
+
+    docker run \
+        -p 4317:4317 \
+        -v $(pwd)/otel-collector-config.yaml:/etc/otel/config.yaml \
+        otel/opentelemetry-collector-contrib:latest
+
+.. code-block:: sh
+
+    $ python example.py
+
+The resulting logs will appear in the output from the collector and look similar to this:
+
+.. code-block:: sh
+
+        ResourceLog #0
+        Resource labels:
+             -> telemetry.sdk.language: STRING(python)
+             -> telemetry.sdk.name: STRING(opentelemetry)
+             -> telemetry.sdk.version: STRING(1.5.0.dev0)
+             -> service.name: STRING(unknown_service)
+        InstrumentationLibraryLogs #0
+        InstrumentationLibrary __main__ 0.1
+        LogRecord #0
+        Timestamp: 2021-08-18 08:26:53.837349888 +0000 UTC
+        Severity: ERROR
+        ShortName:
+        Body: Exception while exporting logs.
+        ResourceLog #1
+        Resource labels:
+             -> telemetry.sdk.language: STRING(python)
+             -> telemetry.sdk.name: STRING(opentelemetry)
+             -> telemetry.sdk.version: STRING(1.5.0.dev0)
+             -> service.name: STRING(unknown_service)
+        InstrumentationLibraryLogs #0
+        InstrumentationLibrary __main__ 0.1
+        LogRecord #0
+        Timestamp: 2021-08-18 08:26:53.842546944 +0000 UTC
+        Severity: ERROR
+        ShortName:
+        Body: The five boxing wizards jump quickly.
+        ResourceLog #2
+        Resource labels:
+             -> telemetry.sdk.language: STRING(python)
+             -> telemetry.sdk.name: STRING(opentelemetry)
+             -> telemetry.sdk.version: STRING(1.5.0.dev0)
+             -> service.name: STRING(unknown_service)
+        InstrumentationLibraryLogs #0
+        InstrumentationLibrary __main__ 0.1
+        LogRecord #0
+        Timestamp: 2021-08-18 08:26:53.843979008 +0000 UTC
+        Severity: ERROR
+        ShortName:
+        Body: Hyderabad, we have a major problem.

docs/examples/logs/example.py

@@ -0,0 +1,62 @@
+import logging
+
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
+    OTLPLogExporter,
+)
+from opentelemetry.sdk._logs import (
+    LogEmitterProvider,
+    OTLPHandler,
+    set_log_emitter_provider,
+)
+from opentelemetry.sdk._logs.export import BatchLogProcessor
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import (
+    BatchSpanProcessor,
+    ConsoleSpanExporter,
+)
+
+trace.set_tracer_provider(TracerProvider())
+trace.get_tracer_provider().add_span_processor(
+    BatchSpanProcessor(ConsoleSpanExporter())
+)
+
+log_emitter_provider = LogEmitterProvider(
+    resource=Resource.create(
+        {
+            "service.name": "shoppingcart",
+            "service.instance.id": "instance-12",
+        }
+    ),
+)
+set_log_emitter_provider(log_emitter_provider)
+
+exporter = OTLPLogExporter(insecure=True)
+log_emitter_provider.add_log_processor(BatchLogProcessor(exporter))
+log_emitter = log_emitter_provider.get_log_emitter(__name__, "0.1")
+handler = OTLPHandler(level=logging.NOTSET, log_emitter=log_emitter)
+
+# Attach OTLP handler to root logger
+logging.getLogger("root").addHandler(handler)
+
+# Log directly
+logging.info("Jackdaws love my big sphinx of quartz.")
+
+# Create different namespaced loggers
+logger1 = logging.getLogger("myapp.area1")
+logger2 = logging.getLogger("myapp.area2")
+
+logger1.debug("Quick zephyrs blow, vexing daft Jim.")
+logger1.info("How quickly daft jumping zebras vex.")
+logger2.warning("Jail zesty vixen who grabbed pay from quack.")
+logger2.error("The five boxing wizards jump quickly.")
+
+
+# Trace context correlation
+tracer = trace.get_tracer(__name__)
+with tracer.start_as_current_span("foo"):
+    # Do something
+    logger2.error("Hyderabad, we have a major problem.")
+
+log_emitter_provider.shutdown()

docs/examples/logs/otel-collector-config.yaml

@@ -0,0 +1,10 @@
+receivers:
+  otlp:
+    protocols:
+      grpc:
+
+exporters:
+  logging:
+
+processors:
+  batch:

docs/sdk/logs.export.rst

@@ -0,0 +1,7 @@
+opentelemetry.sdk._logs.export
+==============================
+
+.. automodule:: opentelemetry.sdk._logs.export
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/sdk/logs.rst

@@ -0,0 +1,22 @@
+opentelemetry.sdk._logs package
+===============================
+
+.. warning::
+   OpenTelemetry Python logs are in an experimental state. The APIs within
+   :mod:`opentelemetry.sdk._logs` are subject to change in minor/patch releases and make no
+   backward compatability guarantees at this time.
+
+   Once logs become stable, this package will be be renamed to ``opentelemetry.sdk.logs``.
+
+Submodules
+----------
+
+.. toctree::
+
+   logs.export
+   logs.severity
+
+.. automodule:: opentelemetry.sdk._logs
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/sdk/logs.severity.rst

@@ -0,0 +1,7 @@
+opentelemetry.sdk._logs.severity
+================================
+
+.. automodule:: opentelemetry.sdk._logs.severity
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/sdk/sdk.rst

@@ -8,5 +8,6 @@ OpenTelemetry Python SDK
 
     resources
     trace
+    logs
     error_handler
     environment_variables

exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/_log_exporter/__init__.py

@@ -0,0 +1,186 @@
+# Copyright The OpenTelemetry Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Optional, Sequence
+from grpc import ChannelCredentials, Compression
+from opentelemetry.exporter.otlp.proto.grpc.exporter import (
+    OTLPExporterMixin,
+    _translate_key_values,
+    get_resource_data,
+    _translate_value,
+)
+from opentelemetry.proto.collector.logs.v1.logs_service_pb2 import (
+    ExportLogsServiceRequest,
+)
+from opentelemetry.proto.collector.logs.v1.logs_service_pb2_grpc import (
+    LogsServiceStub,
+)
+from opentelemetry.proto.common.v1.common_pb2 import InstrumentationLibrary
+from opentelemetry.proto.logs.v1.logs_pb2 import (
+    InstrumentationLibraryLogs,
+    ResourceLogs,
+)
+from opentelemetry.proto.logs.v1.logs_pb2 import LogRecord as PB2LogRecord
+from opentelemetry.sdk._logs import LogRecord as SDKLogRecord
+from opentelemetry.sdk._logs import LogData
+from opentelemetry.sdk._logs.export import LogExporter, LogExportResult
+
+
+class OTLPLogExporter(
+    LogExporter,
+    OTLPExporterMixin[SDKLogRecord, ExportLogsServiceRequest, LogExportResult],
+):
+
+    _result = LogExportResult
+    _stub = LogsServiceStub
+
+    def __init__(
+        self,
+        endpoint: Optional[str] = None,
+        insecure: Optional[bool] = None,
+        credentials: Optional[ChannelCredentials] = None,
+        headers: Optional[Sequence] = None,
+        timeout: Optional[int] = None,
+        compression: Optional[Compression] = None,
+    ):
+        super().__init__(
+            **{
+                "endpoint": endpoint,
+                "insecure": insecure,
+                "credentials": credentials,
+                "headers": headers,
+                "timeout": timeout,
+                "compression": compression,
+            }
+        )
+
+    def _translate_name(self, log_data: LogData) -> None:
+        self._collector_log_kwargs["name"] = log_data.log_record.name
+
+    def _translate_time(self, log_data: LogData) -> None:
+        self._collector_log_kwargs[
+            "time_unix_nano"
+        ] = log_data.log_record.timestamp
+
+    def _translate_span_id(self, log_data: LogData) -> None:
+        self._collector_log_kwargs[
+            "span_id"
+        ] = log_data.log_record.span_id.to_bytes(8, "big")
+
+    def _translate_trace_id(self, log_data: LogData) -> None:
+        self._collector_log_kwargs[
+            "trace_id"
+        ] = log_data.log_record.trace_id.to_bytes(16, "big")
+
+    def _translate_trace_flags(self, log_data: LogData) -> None:
+        self._collector_log_kwargs["flags"] = int(
+            log_data.log_record.trace_flags
+        )
+
+    def _translate_body(self, log_data: LogData):
+        self._collector_log_kwargs["body"] = _translate_value(
+            log_data.log_record.body
+        )
+
+    def _translate_severity_text(self, log_data: LogData):
+        self._collector_log_kwargs[
+            "severity_text"
+        ] = log_data.log_record.severity_text
+
+    def _translate_attributes(self, log_data: LogData) -> None:
+        if log_data.log_record.attributes:
+            self._collector_log_kwargs["attributes"] = []
+            for key, value in log_data.log_record.attributes.items():
+                try:
+                    self._collector_log_kwargs["attributes"].append(
+                        _translate_key_values(key, value)
+                    )
+                except Exception:  # pylint: disable=broad-except
+                    pass
+
+    def _translate_data(
+        self, data: Sequence[LogData]
+    ) -> ExportLogsServiceRequest:
+        # pylint: disable=attribute-defined-outside-init
+
+        sdk_resource_instrumentation_library_logs = {}
+
+        for log_data in data:
+            resource = log_data.log_record.resource
+
+            instrumentation_library_logs_map = (
+                sdk_resource_instrumentation_library_logs.get(resource, {})
+            )
+            if not instrumentation_library_logs_map:
+                sdk_resource_instrumentation_library_logs[
+                    resource
+                ] = instrumentation_library_logs_map
+
+            instrumentation_library_logs = (
+                instrumentation_library_logs_map.get(
+                    log_data.instrumentation_info
+                )
+            )
+            if not instrumentation_library_logs:
+                if log_data.instrumentation_info is not None:
+                    instrumentation_library_logs_map[
+                        log_data.instrumentation_info
+                    ] = InstrumentationLibraryLogs(
+                        instrumentation_library=InstrumentationLibrary(
+                            name=log_data.instrumentation_info.name,
+                            version=log_data.instrumentation_info.version,
+                        )
+                    )
+                else:
+                    instrumentation_library_logs_map[
+                        log_data.instrumentation_info
+                    ] = InstrumentationLibraryLogs()
+
+            instrumentation_library_logs = (
+                instrumentation_library_logs_map.get(
+                    log_data.instrumentation_info
+                )
+            )
+
+            self._collector_log_kwargs = {}
+
+            self._translate_name(log_data)
+            self._translate_time(log_data)
+            self._translate_span_id(log_data)
+            self._translate_trace_id(log_data)
+            self._translate_trace_flags(log_data)
+            self._translate_body(log_data)
+            self._translate_severity_text(log_data)
+            self._translate_attributes(log_data)
+
+            self._collector_log_kwargs[
+                "severity_number"
+            ] = log_data.log_record.severity_number.value
+
+            instrumentation_library_logs.logs.append(
+                PB2LogRecord(**self._collector_log_kwargs)
+            )
+
+        return ExportLogsServiceRequest(
+            resource_logs=get_resource_data(
+                sdk_resource_instrumentation_library_logs,
+                ResourceLogs,
+                "logs",
+            )
+        )
+
+    def export(self, batch: Sequence[LogData]) -> LogExportResult:
+        return self._export(batch)
+
+    def shutdown(self) -> None:
+        pass