Support enhanced configuration of the Kafka source and buffer loggers

[dlvenable]

Is your feature request related to a problem? Please describe.

The kafka buffer and kafka source share a significant number of classes. Many of these classes have their own loggers. And they also use org.apache.kafka classes and use those loggers.

I'd like to have more logging control over the Kafka components depending on whether the logger is used by the source or the buffer. This solution should also extend to the upcoming kafka sink.

Describe the solution you'd like

I'd like to have the Kafka plugins set an MDC context. This way, we can know the context for the different loggers.

try {
    MDC.put("kafkaPluginType", "buffer");
    // All existing work
} finally {
     MDC.remove("kafkaPluginType");
}

Describe alternatives you've considered (Optional)

One idea I had is to attempt to set MDC in Data Prepper core and then plugins can get MDC automatically. This could be a nice feature, but may take a significant amount of effort. I think we can start with this simple approach and then see how we'd like to make it evolve.

Another approach would be to have different loggers. This would likely require a base class and then sub-classes.

public abstract class KafkaCustomProducer<T> {
  protected abstract Logger getLogger();
}

public class KafkaBufferCustomProducer<T> {
  private static final Logger LOG = LoggerFactory.getLogger(KafkaBufferCustomProducer.class);
  protected Logger getLogger() {
    return LOG;
  }
}

This approach adds quite a bit of code complexity. And it will not allow control logging in the org.apache.kafka package.

Additional context

N/A

Tasks

• Add MDC context for buffer plugin
• Add MDC context for source plugin