Skip to content

Commit

Permalink
Add note about log stream naming
Browse files Browse the repository at this point in the history
  • Loading branch information
kislyuk committed Jan 8, 2022
1 parent 2eae3e5 commit a2a7237
Showing 1 changed file with 10 additions and 1 deletion.
11 changes: 10 additions & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Install `awscli <https://pypi.python.org/pypi/awscli>`_ and set your AWS credent
logger.info(dict(foo="bar", details={}))
After running the example, you can see the log output in your `AWS console
<https://console.aws.amazon.com/cloudwatch/home>`_.
<https://console.aws.amazon.com/cloudwatch/home>`_ under the **watchtower** log group.

IAM permissions
~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -242,6 +242,15 @@ Finally, the following shows how to load the configuration into the working appl
logging.config.dictConfig(config_dict)
app.run()
Log stream naming
~~~~~~~~~~~~~~~~~
For high volume logging applications that utilize process pools, it is recommended that you keep the default log stream
name (``{machine_name}/{program_name}/{logger_name}/{process_id}``) or make it unique per process using the
``{process_id}`` template string. Because logs must be submitted sequentially to each log stream, independent processes
sending logs to the same log stream will encounter sequence token synchronization errors and spend extra resources
automatically recovering from them. As the number of processes increases, this overhead will grow until logs fail to
deliver and get dropped (causing a warning on stderr). Partitioning logs into streams by source avoids this contention.

Boto3/botocore/urllib3 logs
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Because watchtower uses boto3 to send logs, the act of sending them generates a number of DEBUG level log messages
Expand Down

0 comments on commit a2a7237

Please sign in to comment.