Skip to content

Commit

Permalink
debugging: small improvement to log loss and overlimit discussion
Browse files Browse the repository at this point in the history
Signed-off-by: Wesley Pettit <[email protected]>
  • Loading branch information
PettitWesley committed Apr 22, 2023
1 parent a099780 commit cee11a0
Showing 1 changed file with 7 additions and 1 deletion.
8 changes: 7 additions & 1 deletion troubleshooting/debugging.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,9 @@ Even if you see this message, you still have not lost logs yet. Since it will re
[2022/02/16 20:11:36] [ warn] [engine] chunk '1-1645042288.260516436.flb' cannot be retried: task_id=0, input=tcp.3 > output=cloudwatch_logs.1
```

When you see this message, you have lost logs. The other case that indicates log loss is when an input is paused, which is covered in the [overlimit error section](#overlimit-warnings).
When you see this message, you have lost logs.

The main other case that indicates log loss is when an input is paused, which is covered in the [overlimit error section](#overlimit-warnings). Please also review our [Log Loss Summary: Common Causes](#log-loss-summary-common-causes).

#### Common Network Errors

Expand Down Expand Up @@ -201,6 +203,10 @@ The `storage buf overlimit` occurs when the number of in memory ("up") chunks ex

The `mem buf overlimit` occurs when the input has exceeded the configured `Mem_Buf_Limit` and `storage.type memory` is configured.

When the input is able to receive logs again, you will see one of the `resume` messages above.

With some inputs, an overlimit warning indicates that you are losing logs- new logs will not be ingested. This is the case with most inputs that stream in data ([forward](https://docs.fluentbit.io/manual/pipeline/inputs/forward) and [TCP](https://docs.fluentbit.io/manual/pipeline/inputs/tcp), for example). If you use [ECS FireLens](https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/) with Fluent Bit, then the stdout/stderr log input is a forward input and an overlimit warning means that new logs will not be ingested and will be lost. The exception to this is the [tail](https://docs.fluentbit.io/manual/pipeline/inputs/tail) input, which can safely pause and resume without losing logs because it tracks its file offset. When it resumes, it can pick back up reading the file at the last offset (assuming the file was not deleted).

#### invalid JSON message

Users who followed [this tutorial](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/ecs-log-collection) or similar to send logs to the TCP input often see message like:
Expand Down

0 comments on commit cee11a0

Please sign in to comment.