tool/logs: flushable ingests not parsed #2899
Comments
bananabrick
added a commit
that referenced
this issue
Sep 20, 2023
There are two kinds of log lines which are relevant to flushable ingestions. The first one is of the form: ``` ingested as flushable 024323 (1.5KB), 024328 (1.0KB), 024324 (2.0KB), 024325 (1.1KB), 024326 (1.1KB), 024327 (54KB) ``` The second one is of the form: ``` flushed 6 ingested flushables L0:024323 (1.5KB) + L0:024328 (1.0KB) + L0:024324 (2.0KB) + L2:024325 (1.1KB) + L2:024326 (1.1KB) + L0:024327 (54KB) in 0.0s (0.0s total), output rate 152MB/s ``` The first type gets printed when we perform a flushable ingestion, and the second type gets printed when we actually ingest the sstables during the flush. For now, we ignore the first log type, and parse the second log type as ingestions. On the following logs: ``` I230831 04:13:28.689575 3717 3@pebble/event.go:685 ⋮ [n10,s10,pebble] 345 [JOB 219] flushed 6 ingested flushables L0:024323 (1.5KB) + L0:024328 (1.0KB) + L0:024324 (2.0KB) + L2:024325 (1.1KB) + L2:024326 (1.1KB) + L0:024327 (54KB) in 0.0s (0.0s total), output rate 152MB/s I230831 04:13:28.824280 3780 3@pebble/event.go:685 ⋮ [n10,s10,pebble] 365 [JOB 226] flushed 6 ingested flushables L0:024334 (1.5KB) + L0:024339 (1.0KB) + L0:024335 (1.9KB) + L0:024336 (1.1KB) + L0:024337 (1.1KB) + L0:024338 (12KB) in 0.0s (0.0s total), output rate 67MB/s ``` We get the following ingestion summary: ``` _kind______from______to_____________________________________count___bytes______time flush L0 2 227KB 0s ingest L0 10 77KB ingest L2 2 2.2KB total 14 306KB 0s _kind______from______to___default____move___elide__delete___count___in(B)__out(B)__mov(B)__del(B)______time compact L2 L3 2 0 0 0 2 31MB 31MB 0B 0B 0s compact L3 L4 0 4 0 0 4 0B 0B 60MB 0B 0s compact L4 L5 2 3 0 0 5 64MB 64MB 32MB 0B 0s compact L5 L6 4 0 0 0 4 320MB 320MB 0B 0B 0s compact L6 L6 0 0 32 1 33 1.3GB 1.3GB 0B 4.4MB 3s total 8 7 32 1 48 1.7GB 1.7GB 92MB 4.4MB 4s ``` Fixes: #2899
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
So these don't appear in the stats.
We are able to parse normal ingests that look like:
But flushable ingests appear as:
The text was updated successfully, but these errors were encountered: