Optimize decimal state serializers for small value case #13573

lukasz-stec · 2022-08-09T14:39:02Z

Description

Given that many decimal aggregations (sum, avg) stay in the long range, aggregation state serializer can be optimized for this case, limiting the number of bytes per position significantly (3-4X).

tpch/tpcds benchmarks

Benchmarks_decimal_aggr_serde_simple_case.pdf

Is this change a fix, improvement, new feature, refactoring, or other?

improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

core query engine (sum, avg aggregation state serialization)

How would you describe this change to a non-technical end user or system administrator?

improve performance of queries with sum or avg aggregations

Related issues, pull requests, and links

Documentation

( X) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(X ) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

sopel39 · 2022-08-10T21:46:24Z

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

@@ -42,7 +42,13 @@ public void serialize(LongDecimalWithOverflowAndLongState state, BlockBuilder ou
            long overflow = state.getOverflow();
            long[] decimal = state.getDecimalArray();
            int offset = state.getDecimalArrayOffset();
-            VARBINARY.writeSlice(out, Slices.wrappedLongArray(count, overflow, decimal[offset], decimal[offset + 1]));
+            if (count == 1 && overflow == 0 && decimal[offset] == 0) {


count == 1 && overflow == 0 truncate should be separate from decimal[offset] == 0

decimal[offset] == 0 will often be true when count == 1 && overflow == 0 isn't.

So you have cases:

1 long -> count == 1 && overflow == 0 && decimal[offset] == 0

2 longs -> count == 1 && overflow == 0

3 longs -> decimal[offset] == 0

4 longs -> full case

I think you can even make it branchless:

append decimal[offset + 1] unconditionally, len += 1

append decimal[offset] unconditionally, len = len + (decimal[offset] == 0 ? 0 : 1)

append overflow and count, len = len + ((count == 1 & overflow == 0) ? 0 : 2)

this may help in some cases, I will try that. This probably won't help in tpch q17/ q18 as it's already hitting the condition.

the implementation you suggested was not correct, I did something similar but I couldn't figure out an efficient way to do branchless in deserialization. Let me know if you see one.

sopel39 · 2022-08-10T22:01:52Z

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

            }
+            else {


similar comment as in https://github.com/trinodb/trino/pull/13573/files#r942932796 but reverse

lukasz-stec · 2022-08-11T14:37:16Z

Made it draft again as I'm gonna benchmark supporting more cases

lukasz-stec · 2022-08-12T12:08:07Z

Currently, our benchmarking infra is failing so the latest benchmark results will available next week probably. cc @sopel39 @gaurav8297

raunaqmorarka

Is BenchmarkBlockSerde#serializeLongDecimal or any other other JMH benchmark relevant here ?

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

...rc/main/java/io/trino/operator/aggregation/state/LongDecimalWithOverflowStateSerializer.java

lukasz-stec

Comments addressed

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

lukasz-stec · 2022-08-17T09:15:01Z

Is BenchmarkBlockSerde#serializeLongDecimal or any other other JMH benchmark relevant here ?

@raunaqmorarka I think BenchmarkDecimalAggregation.benchmarkEvaluateIntermediate is the best match for those changes although the first step and 50% of the benchmark time is the addInput.

I don't see many changes in this microbenchmark. The important thing to notice is that the number of groups thus random memory load times is the most important factor here.

Before
Benchmark                                                  (function)  (groupCount)  (type)  Mode  Cnt   Score   Error  Units
BenchmarkDecimalAggregation.benchmarkEvaluateIntermediate         avg          1000    LONG  avgt   20   7.190 ± 0.307  ns/op
BenchmarkDecimalAggregation.benchmarkEvaluateIntermediate         avg       1000000    LONG  avgt   20  99.130 ± 1.451  ns/op

After
Benchmark                                                  (function)  (groupCount)  (type)  Mode  Cnt   Score   Error  Units
BenchmarkDecimalAggregation.benchmarkEvaluateIntermediate         avg          1000    LONG  avgt   20   6.856 ± 0.037  ns/op
BenchmarkDecimalAggregation.benchmarkEvaluateIntermediate         avg       1000000    LONG  avgt   20  94.083 ± 1.077  ns/op

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

...rc/main/java/io/trino/operator/aggregation/state/LongDecimalWithOverflowStateSerializer.java

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

sopel39 · 2022-08-18T10:51:37Z

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

+            buffer[1] = decimalHighBytes;
+            buffer[2] = overflow;
+            buffer[3] = count;
+            // if decimalHighBytes == 0 and count == 1 and overflow == 0 we only write decimalLowBytes (bufferLength = 1)


You miss one important case: 3 longs -> decimal[offset] == 0
see #13573 (comment)

You can implement it by:

// append low buffer[0] = decimalLowBytes; offset = 1; // append high buffer[offset] = decimalHiBytes; offset = offset + (decimalHiBytes == 0 ? 0 : 1); // append overflow, count buffer[offset] = overflow; buffer[offset + 1] = count; offset = offset + (overflow == 0 & count == 1) ? 0 : 2 // will this be branchless really?

sopel39 · 2022-08-18T11:31:47Z

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

+            long high = 0;
+            long overflow = 0;
+            long count = 1;
+            if (slice.length() > Long.BYTES) {


branchless version

int sliceLength = slices.length(); long low = slice.getLong(0); long highOffset = sliceLength == 4 | sliceLength == 2 ? 1 : 0; long high = slice.getLong(highOffset) & (highOffset * -1L); // similar for count & overflow

Would be great to learn if JIT actually generates branchless versions

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

lukasz-stec · 2022-08-18T14:19:45Z

the latest version tpch/tpcds benchmarks

good improvement for tpch

Benchmarks_decimal_aggr_serde_opt.pdf

lukasz-stec

I implemented the additional case + UT.
For the branchless deserialization. I tested multiple ways (including offered by @sopel39 ) and it there is a little difference in jmh for mixed decimals cases (block with both small and large decimals) but if/switch approach is a little better for both small and large cases where the branch prediction works efficiently.

The tpc benchmarks look very good with the current code (switch).

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

lukasz-stec · 2022-08-25T07:55:13Z

tpch/tpcds orc part 1k on latest version (missing case implemented + using switch in deserailziation).
Big gains for tpch, some for tpcds.

Benchmarks_decimal_aggr_serde_opt_orc_part_sf1k_250822.pdf

sopel39

lgtm [Optimize decimal state serializers for small value case](https://github.com/trinodb/trino/pull/13573/commits/18708885459f1921b720148552a14d1ca8ff7c93) % comments

sopel39 · 2022-08-29T15:12:28Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -53,12 +54,12 @@
 @State(Scope.Thread)
 @OutputTimeUnit(TimeUnit.NANOSECONDS)
 @Fork(3)
-@Warmup(iterations = 10)
-@Measurement(iterations = 10)
+@Warmup(iterations = 20, time = 1)


nit: is time needed?

default is 10s, this means 1s.
That said, looking through generated assembly code i noticed that 1s may not be enough to trigger c2. let me check

yeah, 1s is not enough to trigger c2. 5s warmup does the trick

sopel39 · 2022-08-29T15:13:44Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+
+    @Benchmark
+    @OutputTimeUnit(TimeUnit.MICROSECONDS)
+    public Object benchmarkDeserialize(BenchmarkData data)


Why it's called benchmarkDeserialize? I think processPage does more than just deserialization

there is no way to benchmark deserialize only so I do processPage with unique group ids which makes combine behavior simple

Still, for a code reader this is confusing.

there is no way to benchmark deserialize only so

Sure there is: LongDecimalWithOverflowAndLongStateSerializer is a class and there is nothing stopping us from benchmarking it directly

benchmarking LongDecimalWithOverflowAndLongStateSerializer#deserialize in isolation has little value. it's always used in loop body in addIntermediate with combine and it's supposed to be inlined there (that's the reason for code generation)

sopel39 · 2022-08-29T15:14:12Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+        @Param({"SHORT", "LONG", "MIXED"})
+        private String decimalSize = "SHORT";
+
+        @Param({"true", "false"})


How does group ordering matter?

It doesn't matter for serialization, but for evaluateIntermediate the main bottleneck is actually jumping to random memory locations in case groups are located randomly.
For the partial aggregation adaptation case, the group ids are always consecutive.

For the partial aggregation adaptation case, the group ids are always consecutive.

This has little to do with serialization.

It doesn't matter for serialization, but for evaluateIntermediate the main bottleneck is actually jumping to random memory locations in case groups are located randomly.

For skip-cases groups are consecutive and number of them is small, so it probably doesn't matter too.

This has little to do with serialization.

That's what I said ^^

For skip-cases groups are consecutive and number of them is small, so it probably doesn't matter too.

For skip cases (I understand you mean PA disabled) yes, for other cases it does matter.

sopel39 · 2022-08-29T15:14:20Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -39,6 +39,7 @@
 import org.testng.annotations.Test;


this should be separate PR

all changes to BenchmarkDecimalAggregation?

@sopel39 ping

extracted BenchmarkDecimalAggregation changes to #13939

sopel39 · 2022-08-29T15:15:15Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -108,36 +130,66 @@ public Block benchmarkEvaluateFinal(BenchmarkData data)
        @Param({"10", "1000"})
        private int groupCount = 10;

+        @Param({"SHORT", "LONG", "MIXED"})
+        private String decimalSize = "SHORT";


decimalSize=LONG won't work with type=SHORT

in this case, I use larger long values (close to Long.MAX_VALUE)

in this case, I use larger long values (close to Long.MAX_VALUE)

It doesn't matter from perf perspective.

it can matter. it doesn't at the moment. I can also skip this case for type=SHORT if that bothers you.

sopel39 · 2022-08-29T15:15:44Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+                    values = createValues(functionResolution, type, (builder, value) -> {
+                        boolean writeShort = "SHORT".equals(decimalSize) || ("MIXED".equals(decimalSize) && random.nextBoolean());
+                        if (writeShort) {
+                            builder.writeLong(value);


both cases are long really writeLong

yes, but if we use varint those are different cases.

yes, but if we use varint those are different cases.

Here SHORT vs LONG is mixed with Short/Long decimal. This is confusing

replaced SHORT, LONG with SMALL, BIG

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

...rc/main/java/io/trino/operator/aggregation/state/LongDecimalWithOverflowStateSerializer.java

...a/io/trino/operator/aggregation/state/TestLongDecimalWithOverflowAndLongStateSerializer.java

sopel39 · 2022-08-29T16:00:37Z

Nice gains

lukasz-stec

comments addressed

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java

lukasz-stec · 2022-08-30T08:51:32Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+
+    @Benchmark
+    @OutputTimeUnit(TimeUnit.MICROSECONDS)
+    public Object benchmarkDeserialize(BenchmarkData data)


there is no way to benchmark deserialize only so I do processPage with unique group ids which makes combine behavior simple

lukasz-stec · 2022-08-30T08:54:24Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -108,36 +130,66 @@ public Block benchmarkEvaluateFinal(BenchmarkData data)
        @Param({"10", "1000"})
        private int groupCount = 10;

+        @Param({"SHORT", "LONG", "MIXED"})
+        private String decimalSize = "SHORT";


in this case, I use larger long values (close to Long.MAX_VALUE)

lukasz-stec · 2022-08-30T08:59:49Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+        @Param({"SHORT", "LONG", "MIXED"})
+        private String decimalSize = "SHORT";
+
+        @Param({"true", "false"})


It doesn't matter for serialization, but for evaluateIntermediate the main bottleneck is actually jumping to random memory locations in case groups are located randomly.
For the partial aggregation adaptation case, the group ids are always consecutive.

lukasz-stec · 2022-08-30T09:00:42Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+                    values = createValues(functionResolution, type, (builder, value) -> {
+                        boolean writeShort = "SHORT".equals(decimalSize) || ("MIXED".equals(decimalSize) && random.nextBoolean());
+                        if (writeShort) {
+                            builder.writeLong(value);


yes, but if we use varint those are different cases.

...a/io/trino/operator/aggregation/state/TestLongDecimalWithOverflowAndLongStateSerializer.java

lukasz-stec · 2022-08-30T09:07:19Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -53,12 +54,12 @@
 @State(Scope.Thread)
 @OutputTimeUnit(TimeUnit.NANOSECONDS)
 @Fork(3)
-@Warmup(iterations = 10)
-@Measurement(iterations = 10)
+@Warmup(iterations = 20, time = 1)


default is 10s, this means 1s.
That said, looking through generated assembly code i noticed that 1s may not be enough to trigger c2. let me check

...rc/main/java/io/trino/operator/aggregation/state/LongDecimalWithOverflowStateSerializer.java

lukasz-stec · 2022-08-30T11:14:30Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -39,6 +39,7 @@
 import org.testng.annotations.Test;


all changes to BenchmarkDecimalAggregation?

lukasz-stec · 2022-08-30T11:15:20Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -53,12 +54,12 @@
 @State(Scope.Thread)
 @OutputTimeUnit(TimeUnit.NANOSECONDS)
 @Fork(3)
-@Warmup(iterations = 10)
-@Measurement(iterations = 10)
+@Warmup(iterations = 20, time = 1)


yeah, 1s is not enough to trigger c2. 5s warmup does the trick

sopel39

reviewed again

lukasz-stec

comments addressed

lukasz-stec · 2022-08-30T13:05:25Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+
+    @Benchmark
+    @OutputTimeUnit(TimeUnit.MICROSECONDS)
+    public Object benchmarkDeserialize(BenchmarkData data)


benchmarking LongDecimalWithOverflowAndLongStateSerializer#deserialize in isolation has little value. it's always used in loop body in addIntermediate with combine and it's supposed to be inlined there (that's the reason for code generation)

lukasz-stec · 2022-08-30T13:07:00Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -108,36 +130,66 @@ public Block benchmarkEvaluateFinal(BenchmarkData data)
        @Param({"10", "1000"})
        private int groupCount = 10;

+        @Param({"SHORT", "LONG", "MIXED"})
+        private String decimalSize = "SHORT";


it can matter. it doesn't at the moment. I can also skip this case for type=SHORT if that bothers you.

lukasz-stec · 2022-08-30T13:14:52Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+        @Param({"SHORT", "LONG", "MIXED"})
+        private String decimalSize = "SHORT";
+
+        @Param({"true", "false"})


This has little to do with serialization.

That's what I said ^^

For skip-cases groups are consecutive and number of them is small, so it probably doesn't matter too.

For skip cases (I understand you mean PA disabled) yes, for other cases it does matter.

...a/io/trino/operator/aggregation/state/TestLongDecimalWithOverflowAndLongStateSerializer.java

lukasz-stec · 2022-08-30T20:53:39Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

+                    values = createValues(functionResolution, type, (builder, value) -> {
+                        boolean writeShort = "SHORT".equals(decimalSize) || ("MIXED".equals(decimalSize) && random.nextBoolean());
+                        if (writeShort) {
+                            builder.writeLong(value);


replaced SHORT, LONG with SMALL, BIG

lukasz-stec · 2022-08-30T20:54:03Z

core/trino-main/src/test/java/io/trino/operator/aggregation/BenchmarkDecimalAggregation.java

@@ -39,6 +39,7 @@
 import org.testng.annotations.Test;


@sopel39 ping

Given that many decimal aggregations (sum, avg) stay in the long range, aggregation state serializer can be optimized for this case, limiting the number of bytes per position significantly (3-4X) at the cost of small cpu overhead during serialization and deserialization.

cla-bot bot added the cla-signed label Aug 9, 2022

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch 3 times, most recently from ea406fd to 1f42004 Compare August 10, 2022 13:02

lukasz-stec marked this pull request as ready for review August 10, 2022 19:44

lukasz-stec requested review from sopel39 and gaurav8297 August 10, 2022 19:44

sopel39 reviewed Aug 10, 2022

View reviewed changes

lukasz-stec marked this pull request as draft August 11, 2022 14:36

lukasz-stec added the performance label Aug 12, 2022

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from da7883e to f213351 Compare August 12, 2022 12:03

lukasz-stec changed the title ~~Optimize decimal state serializers for low value case~~ Optimize decimal state serializers for small value case Aug 12, 2022

lukasz-stec marked this pull request as ready for review August 12, 2022 12:06

lukasz-stec requested a review from sopel39 August 12, 2022 12:06

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from f213351 to db86662 Compare August 16, 2022 14:05

raunaqmorarka reviewed Aug 16, 2022

View reviewed changes

lukasz-stec commented Aug 17, 2022

View reviewed changes

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from db86662 to 34ed7ef Compare August 17, 2022 09:09

lukasz-stec requested a review from raunaqmorarka August 17, 2022 09:15

raunaqmorarka reviewed Aug 17, 2022

View reviewed changes

raunaqmorarka approved these changes Aug 17, 2022

View reviewed changes

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from 34ed7ef to 87aafb6 Compare August 17, 2022 10:35

sopel39 reviewed Aug 18, 2022

View reviewed changes

lukasz-stec mentioned this pull request Aug 19, 2022

Use "union" in partial aggregation output #12155

Closed

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch 2 times, most recently from 29a2fae to 1870888 Compare August 25, 2022 07:47

lukasz-stec commented Aug 25, 2022

View reviewed changes

.../java/io/trino/operator/aggregation/state/LongDecimalWithOverflowAndLongStateSerializer.java Show resolved Hide resolved

lukasz-stec requested a review from sopel39 August 25, 2022 07:55

sopel39 reviewed Aug 29, 2022

View reviewed changes

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from 1870888 to ab4315c Compare August 30, 2022 11:23

lukasz-stec commented Aug 30, 2022

View reviewed changes

lukasz-stec requested a review from sopel39 August 30, 2022 11:24

sopel39 reviewed Aug 30, 2022

View reviewed changes

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from ab4315c to b506c13 Compare August 30, 2022 20:54

lukasz-stec commented Aug 30, 2022

View reviewed changes

lukasz-stec requested a review from sopel39 August 30, 2022 20:54

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from b506c13 to ac1f861 Compare August 31, 2022 09:40

lukasz-stec force-pushed the ls/034-paa-decimal-aggregation-serde branch from ac1f861 to 923fa7a Compare August 31, 2022 10:28

sopel39 approved these changes Aug 31, 2022

View reviewed changes

sopel39 merged commit 182b44e into trinodb:master Aug 31, 2022

sopel39 mentioned this pull request Aug 31, 2022

Release notes for 395 #13913

Closed

github-actions bot added this to the 395 milestone Aug 31, 2022

colebow mentioned this pull request Sep 2, 2022

Add Trino 395 release notes #13975

Merged

Optimize decimal state serializers for small value case #13573

Optimize decimal state serializers for small value case #13573

Conversation

lukasz-stec commented Aug 9, 2022 • edited Loading

Description

tpch/tpcds benchmarks

Related issues, pull requests, and links

Documentation

Release notes

sopel39 Aug 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukasz-stec commented Aug 11, 2022

lukasz-stec commented Aug 12, 2022

raunaqmorarka left a comment

Choose a reason for hiding this comment

lukasz-stec left a comment

Choose a reason for hiding this comment

lukasz-stec commented Aug 17, 2022

Choose a reason for hiding this comment

sopel39 Aug 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukasz-stec commented Aug 18, 2022

lukasz-stec left a comment

Choose a reason for hiding this comment

lukasz-stec commented Aug 25, 2022

sopel39 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sopel39 commented Aug 29, 2022

lukasz-stec left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sopel39 left a comment

Choose a reason for hiding this comment

lukasz-stec left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukasz-stec commented Aug 9, 2022 •

edited

Loading

sopel39 Aug 10, 2022 •

edited

Loading

sopel39 Aug 18, 2022 •

edited

Loading