Improve logql query statistics collection.

[cyriltovena]

This also add information about ingester queries using grpc trailers.

Next step:

• Adding this to the all API result.
• Merging stats in the frontend.
• Instrument line and byte throughput per query.

Signed-off-by: Cyril Tovena [email protected]

[cyriltovena]

benchmark before:

goos: darwin
goarch: amd64
pkg: github.com/grafana/loki/pkg/chunkenc
BenchmarkHeadBlockIterator/Size_100000-16         	     528	   2142509 ns/op	 2400298 B/op	       2 allocs/op
BenchmarkHeadBlockIterator/Size_50000-16          	    1009	   1086106 ns/op	 1204261 B/op	       2 allocs/op
BenchmarkHeadBlockIterator/Size_15000-16          	    4485	    253231 ns/op	  360480 B/op	       2 allocs/op
BenchmarkHeadBlockIterator/Size_10000-16          	    7348	    164947 ns/op	  245792 B/op	       2 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/chunkenc	29.927s

after

goos: darwin
goarch: amd64
pkg: github.com/grafana/loki/pkg/chunkenc
BenchmarkHeadBlockIterator/Size_100000-16         	     513	   2251863 ns/op	 2400296 B/op	       2 allocs/op
BenchmarkHeadBlockIterator/Size_50000-16          	    1060	   1123540 ns/op	 1204259 B/op	       2 allocs/op
BenchmarkHeadBlockIterator/Size_15000-16          	    4336	    260386 ns/op	  360480 B/op	       2 allocs/op
BenchmarkHeadBlockIterator/Size_10000-16          	    7159	    172213 ns/op	  245792 B/op	       2 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/chunkenc	28.987s

This is pretty much similar now ! I had to change a bit the implementation.

[owen-d]

can you add some comments to these describing what they refer to? I'm having a hard time understandingUncompressed vs Decompressed vs Compressed. I'm guessing one is before decompression, one is after decompression, and one is after recompression?

[cyriltovena]

• Uncompressed is data that was processed but not compressed from headchunk.
• Decompressed is data that was processed from decompression store or ingester.
• Compressed is data that that was decompressed. but the value before decompression.

[cyriltovena]

I'm happy to change the names, or document it better where would it be better to document this ?

@slim-bean ??

[owen-d]

I think if you put your comment in the code as a comment it'd be great.

[slim-bean]

2 thoughts:

Do we really care that they came out of the head chunk? Should we just add the line and byte count to the decompressed counts?

If we do care, i would suggest naming them:

HeadChunkBytes
HeadChunkLines

[owen-d]

This looks great.