Non-error logs are printed to stderr

[ku1ik]

Describe the bug

As in title. This makes alerting configuration based on log severity problematic because everything printed by loki itself seems to be an error. Most logging aggregation tools and platforms treat stderr output as level=error.

For example:

2022-02-06 11:42:17 | level=info ts=2022-02-06T10:42:17.760239159Z caller=table.go:362 msg="finished uploading table index_19029

^^^ not an error but it's logged to stderr.

To Reproduce

Steps to reproduce the behavior:

1. Start Loki (2.4.2)
2. Wait until it prints its standard log messages (e.g. db, table related info)

Expected behavior
Informational logs are printed to stdout, and only errors are printed to stderr.

Environment:

• bare-metal

[muffl0n]

This seems to be defined here:

loki/pkg/util/log/log.go

Lines 46 to 49 in 46956d0

	logger := log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
	if format.String() == "json" {
	logger = log.NewJSONLogger(log.NewSyncWriter(os.Stderr))
	}

I'm not sure how one would fix this. Just changing the code to log.NewLogfmtLogger(log.NewSyncWriter(os.Stdout)) is wrong either, cause everything above "WARN" should be printed to stderr furthermore.

It's similar for promtail, btw.

[munsayac13]

Im having similar issue. We are using loki with fluentbit. Loki version 2.5.0 and Fluentbit 2.1.0 under kubernetes cluster 1.21