Logging Documentation / Best Practices

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/170

Originally created by @vischous on 2021-07-16 22:27:20

---

Could we get logging documentation and best practices?

I spent a few hours diving into this today. It took me that long because I refused to read the advanced logging section in python for some silly reason. But to the point

I believe https://github.com/transferwise/pipelinewise-singer-python/blob/master/singer/logging.conf is where the majority of the logging configuration is coming from.

Maybe we could tell everyone they could specify a LOGGING_CONF_FILE in their environment variables if they need debug logging. With an example or two?

Context:

1. I want debug logging, and I need to turn on debug logging sometimes in production but there's PII data in the logs so I can't output them all the time.
2. I have a custom target with some custom classes that are using logger.debug("yadayada") and I was manually adding the logging configuration to see this data. In production I need a better way than this (LOGGING_CONF_FILE is one way)

Subtopics (curated by AJ)

1. logging.conf: We can document how works, how to set and give examples for users/devs to utilize this.
2. Changing log level for particularly noisy operations in the SDK.
   • For high-cardinality parent-child relationships, the "starting" and "stopping" section of each partition context can be 10+ times per second.
   • For high-cardinality parent-child relationships, Singer INFO metrics also can be noisy if emitted after each stream.
3. Within a Meltano logging.conf, we might want separate control of the SDK vs Meltano itself.
4. Pretty-printed exceptions, ideally togglable.

[MeltyBot]

View 12 previous comments from the original issue on GitLab

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

We do have https://sdk.meltano.com/en/v0.30.0/implementation/logging.html.

Comment if you think it's lacking.