You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since the sort task will be run on different TaskManager nodes, in order to more conveniently view the log output, we can consider using OpenTelemetry to achieve centralized log reporting.
Use case
To achieve the above function, we needs to configure an OpenTelemetryAppender for log4j:
When submitting tasks in the form of jobGraph, TaskManager uses log4j2-console.properties for configuration by default. However, this method will cause the OpenTelemetryAppender in the connector to be inconsistent with the OpenTelemetryAppender instance loaded when Flink is initialized, resulting in the appender not being able to be used normally.
For this reason, consider dynamically adding openTelemetryAppender in connect based on log4j-api, developing OpenTelemetryLogger class, and providing initial, install and ubinstall functions for OpenTelemetryAppender.
The way to integrate OpenTelemetryAppender for connectors that comply with the FLIP-27 standard is as follows:
Create a new OpenTelemetryLogger object in the corresponding SourceReader construction method and configure relevant parameters(log-collector endpoint,logLevel,layout and so on)
Call the install method of the openTelemetryLogger object in the start method of SourceReader
Call the uninstall method of the openTelemetryLogger object in the close method of SourceReader
qy-liuhuo
changed the title
[Feature][Sort] Provides a method to add openTelemetryAppender to the sort connector
[Feature][Sort] Provides a method to add openTelemetryAppender for the sort connector
Sep 9, 2024
Description
Since the sort task will be run on different TaskManager nodes, in order to more conveniently view the log output, we can consider using
OpenTelemetry
to achieve centralized log reporting.Use case
To achieve the above function, we needs to configure an OpenTelemetryAppender for log4j:
When submitting tasks in the form of jobGraph, TaskManager uses
log4j2-console.properties
for configuration by default. However, this method will cause the OpenTelemetryAppender in the connector to be inconsistent with the OpenTelemetryAppender instance loaded when Flink is initialized, resulting in the appender not being able to be used normally.For this reason, consider dynamically adding openTelemetryAppender in connect based on log4j-api, developing OpenTelemetryLogger class, and providing
initial
,install
andubinstall
functions for OpenTelemetryAppender.The way to integrate OpenTelemetryAppender for connectors that comply with the FLIP-27 standard is as follows:
OpenTelemetryLogger
object in the corresponding SourceReader construction method and configure relevant parameters(log-collector endpoint,logLevel,layout and so on)install
method of the openTelemetryLogger object in the start method of SourceReaderuninstall
method of the openTelemetryLogger object in the close method of SourceReaderAre you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: