KAFKA-15942: Implement ConsumerInterceptors in the async consumer #15000

lucasbru · 2023-12-13T12:41:46Z

We need to make sure to call the consumer interceptor and test its integration.

This is adding the required call in commitSync and commitAsync. The calls in commitAsync are executed using the same mechanism as commit callbacks, to ensure that we are calling the interceptors from a single thread, as is intended in the original KIP.

The interceptors also need to be invoked on auto-commits which are executed in the commit request manager. For this purpose, we share the OffsetCommitCallbackInvoker class with the background thread (it is already accessed implicitly from the background thread through a future lambda). This is done analogous to the RebalanceListenerInvoker.

Co-authored-by: John Doe [email protected]

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

lucasbru · 2023-12-19T17:10:35Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java

@@ -757,6 +757,14 @@ public void commitAsync(Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCo
        try {
            CompletableFuture<Void> future = commit(offsets, false);
            future.whenComplete((r, t) -> {
+                if (t == null && interceptors != null) {
+                    invoker.submit(new OffsetCommitCallbackTask(


Original KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors states:

onCommit() will be called when offsets get committed: just before OffsetCommitCallback.onCompletion() is called and in ConsumerCoordinator.commitOffsetsSync() on successful commit.

Since new consumer is single-threaded, ConsumerInterceptor API will be called from a single thread. Since interceptor callbacks are called for every record, the interceptor implementation should be careful about adding performance overhead to consumer.

That's why I implemented it using the Invoker.

lucasbru · 2024-01-03T08:52:48Z

@philipnee could you take a look please?

philipnee · 2024-01-03T19:42:27Z

hi @lucasbru - i think we might need to loosen the thread access restriction for the interceptor because i think we also need to trigger interceptors during autocommit. There are 2 ways to achieve this:

pass an event to the main thread to execute the interceptor on poll: the interceptor might never get triggered
loosen the thread access restriction - I think we might need to do that to guarantee the interceptors can always be invoked

#2 is a bit annoying because it against the contract as you mentioned in one of the comment. What do you think?

lucasbru · 2024-01-04T14:27:26Z

@philipnee Good point about auto-commits, I missed that. It's a pity that auto-commits aren't triggered from the main thread (I wonder if we could do that? Would be another architectural change).

I would go for solution num 1 because

Original KIP clearly states that it should only call into the interceptor from a single thread, so I think we'd easily break code if we start requiring them to be thread-safe. Yeah, you could write a KIP, but not sure if that's reasonable here.
When would we not trigger the interceptor in this case? If you are talking about Consumer.close(), could we make sure to empty the background event queue after closing the background thread, and then run the invoker?

Wdyt?

philipnee · 2024-01-04T17:17:19Z

hi @lucasbru - thanks. i personally prefer to make consumer interceptor thread safe because firstly, (i think) it simplifies the code quite a bit and secondly, it is possible to miss the interceptor invocation if user fail to poll/close after a commit is sent. latter is a bigger problem i think. do you have an idea of how to fix this?

When would we not trigger the interceptor in this case? i think we always want to trigger onCommit interceptor if provided. so I think after sending the last commit sync on close, we just need to clear the queue, close the network thread, and invoke the callbacks.

lucasbru · 2024-01-05T14:29:12Z

@philipnee some considerations:

Does the old consumer guarantee that the interceptor is called when poll or close are not called by the consumer anymore after an autocommit? I don't think so, because it's running as part of the application thread, so how would it provide that guarantee without even being called?
I think the interceptor is meant for monitoring / metrics -- it doesn't have the strict requirements as, for example, an ongoing commit shouldn't be dropped during close because it will result in data loss or duplicate processing.

"Make the consumer interceptor thread-safe" -- How would we even do that? Consumer intereceptors are implemented by the user, we cannot make them thread safe by just putting a lock on it. The only way would be to write a KIP, have a breaking change and a migration guide, and ask all users to upgrade their code. Not sure that is worth it, given the original consumer does not seem to provide the guarantee that you are asking for.

I think the best we can do is to insert the interceptor callback into the invoker queue upon commit completion, and make sure during close to execute all commit callbacks that are enqueued, no? If the user fails to call close, we are out of luck anyways.

AndrewJSchofield · 2024-01-09T11:20:20Z

The javadoc for ConsumerInterceptor says "ConsumerInterceptor callbacks are called from the same thread that invokes KafkaConsumer.poll(java.time.Duration)". As a result, it is quite clear what needs to happen here without a new KIP.

lucasbru · 2024-01-10T16:18:57Z

In terms of implementation of the autocommit-interceptor in the application thread, I see three options:

Move auto-commit triggering to application thread. That seems cleanest to me (all commits are triggered from the application thread), but it's the largest refactoring. Then the interceptor is trivial.
Share the Invoker queue with the background thread, and access it when we register the auto-commit callback. It's already shared with the background thread implicitly as part of the commitrequest completed future.
Add a new background event in the background event queue. Seems unclean, since we already have a common datastructure shared between application & background thread for things to be invoked after a commit, and that is the "Invoker" (option 2).

lianetm · 2024-01-11T17:54:50Z

My take would be option 2, using the invoker in the background thread and submitting a task for the interceptor. Seems like a clean way, re-using the mechanism of the invoker already in place, and without doing any major refactoring.

I would definitely leave the auto-commit logic in the background thread where it is, as it's truly an internal operation/request we want to perform without any direct relation with the app layer, needed from multiple places in the background even: auto-commit on the interval, but also auto-commit as part of the reconciliation process.

AndrewJSchofield · 2024-01-11T18:30:07Z

I agree with @lianetm that the second option seems best. The invoker mechanism already exists for commitAsync() and submitting to the invoker is already using a thread-safe object.

philipnee · 2024-01-11T20:59:35Z

hey @lucasbru - i assume invoker queue here you meant by OffsetCommitCallbackInvoker. It is not shared with the background thread. There are two invokers used by the async consumer, one is the offset commit callback invoker, the other one is the rebalance callback. The rebalance callback is shared by the two threads but the commit callback invoker is not.

my original thought was to use the background event to pass the interceptor onCommit event.

i think we are all on the same page about using the invoker to invoke the events.

lucasbru · 2024-01-12T16:00:27Z

@philipnee I meant that the OffsetCommitCallbackInvoker is implicitly shared with the background thread through the lambda passed into future.onComplete. That's why it's thread-safe. So we are just making this explicit.

lucasbru · 2024-01-12T16:01:00Z

I have updated the PR to run interceptors during auto-commits as well. Let me know what you think

lucasbru · 2024-01-12T16:03:30Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java

                if (callback == null) {
                    if (t != null) {
                        log.error("Offset commit with offsets {} failed", offsets, t);
                    }
                    return;
                }

-                invoker.submit(new OffsetCommitCallbackTask(callback, offsets, (Exception) t));
+                offsetCommitCallbackInvoker.submitUserCallback(callback, offsets, (Exception) t);


Refactored so that OffsetCommitCallbackTask can be an private class inside the invoker.

lucasbru · 2024-01-12T16:04:49Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java

-     * achieved by having the background thread register a {@link OffsetCommitCallbackTask} to the invoker upon the
-     * future completion, and execute the callbacks when user polls/commits/closes the consumer.
-     */
-    private class OffsetCommitCallbackInvoker {


Moved to a separate file since it's going to be shared across threads.

AndrewJSchofield

This PR looks about ready to merge. Please can you change PlaintextConsumerTest to run the tests for interceptors with the new consumer. They should pass once this PR is merged.

AndrewJSchofield · 2024-01-12T16:24:20Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+    public void submitCommitInterceptors(final Map<TopicPartition, OffsetAndMetadata> offsets) {
+        if (!interceptors.isEmpty()) {
+            callbackQueue.add(new OffsetCommitCallbackTask(
+                (o, e) -> interceptors.onCommit(o),


I think the use of single-character variable names here is a little confusing.

philipnee · 2024-01-12T16:55:28Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+
+    public void submitUserCallback(final OffsetCommitCallback callback,
+                                   final Map<TopicPartition, OffsetAndMetadata> offsets,
+                                   final Exception e) {


would be more consistent with the styling - exception instead of e.

philipnee · 2024-01-12T16:57:00Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+
+        public OffsetCommitCallbackTask(final OffsetCommitCallback callback,
+                                        final Map<TopicPartition, OffsetAndMetadata> offsets,
+                                        final Exception e) {


similarly to above.

lucasbru · 2024-01-15T13:11:39Z

This PR looks about ready to merge. Please can you change PlaintextConsumerTest to run the tests for interceptors with the new consumer. They should pass once this PR is merged.

The auto-commit one still fails, I debugged the problem and create a separate ticket (solution is clear, but we want to have this as a separate PR for potential backporting). The others are enabled.

lucasbru · 2024-01-15T13:12:06Z

@cadonna Can you please review this?

lucasbru · 2024-01-16T17:41:25Z

@cadonna The autoCommitIntercept test is flaky after this PR, but it's unrelated to this PR. I will create a separate ticket and disable it agian once your comments come in.

philipnee · 2024-01-16T18:53:44Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java

@@ -1902,65 +1912,14 @@ private void maybeThrowFencedInstanceException() {
    }

    private void maybeInvokeCommitCallbacks() {
-        if (callbacks() > 0) {
-            invoker.executeCallbacks();
+        if (offsetCommitCallbackInvoker.executeCallbacks()) {


the naming appears a bit misleading for me - if executeCallback() return true, it almost means the callbacks were executed correctly. I wonder if we could restructure the code like

invoker.executeCallbacks() isFenced = invoker.hasFencedException()

Good point, done. I could actually simplify the code by moving isFenced inside the invoker.

philipnee · 2024-01-16T18:56:04Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java

@@ -93,9 +94,11 @@ public CommitRequestManager(
            final SubscriptionState subscriptions,
            final ConsumerConfig config,
            final CoordinatorRequestManager coordinatorRequestManager,
+            final OffsetCommitCallbackInvoker offsetCommitCallbackInvoker,


looking back at the implementation - now i think it is rather unnecessary to have these finals in the parameters. I wonder if you think we should clean them up in the future...

This is very mixed across the clients codebase. Sometimes you put the final sometimes you don't. In the streams module there is a strict rule to do. Not sure, but as long as there is no guideline around this, and we are not completely repulsed by it, I'd suggest to just stick with whatever the existing code is doing for consistency to not mess up git blame too much.

philipnee · 2024-01-16T18:57:24Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java

            final String groupId,
            final Optional<String> groupInstanceId) {
-        this(time, logContext, subscriptions, config, coordinatorRequestManager, groupId,
+        this(time, logContext, subscriptions, config, coordinatorRequestManager,


i wonder if it would be more aesthetic to split each of them into its own line.... now it is spanning 3 lines with different widths.

philipnee · 2024-01-16T18:58:35Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+    /**
+     * @return true if an offset commit was fenced.
+     */
+    public boolean executeCallbacks() {


see comment above.

philipnee · 2024-01-16T19:02:32Z

clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java

@@ -844,6 +849,54 @@ public void testWakeupCommitted() {
        assertNull(consumer.wakeupTrigger().getPendingTask());
    }

+    @Test


what's the interceptor behavior on close? if we have inflight commits before closing the consumer, should the interceptors be invoked? can we add tests around that?

Added a test. Interceptors are called.

philipnee · 2024-01-16T19:04:06Z

clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java

@@ -367,6 +372,42 @@ public void testAutocommitEnsureOnlyOneInflightRequest() {
        assertPoll(1, commitRequestManger);
    }

+    @Test


similar to the comment above - we autocommit on close - should the interceptor be triggered?

I think they should. Looks to me like the legacy consumer called the interceptors when closing the consumer coordinator (ConsumerCoordinator.maybeAutoCommitOffsetsSync).

Yes, they should. This is tested in the integration test PlaintextConsumer.testAutoCommitIntercept - which works in terms of interceptors, but I have to keep disabled in this PR because of KAFKA-16155. This PR does call the interceptors after closing the network thread (I pinged you about it above). I can add a little unit test to AsyncKafkaConsumerTest. I don't think we can add a unit test for it in CommitRequestManagerTest, because the autocommit on close is triggered from the application thread, so in this class it does look very much like any normal commit.

philipnee

thanks for the changes @lucasbru - I left some comments.

philipnee · 2024-01-17T17:10:12Z

core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala

@@ -1336,8 +1336,9 @@ class PlaintextConsumerTest extends BaseConsumerTest {
    MockProducerInterceptor.resetCounters()
  }

+  // This is disabled for the the consumer group until KAFKA-16155 is resolved.


thanks for reporting this.

philipnee

Hi @lucasbru - Thanks a lot for the PR. I think this is good to go. Await for @cadonna for the last pass ...

cadonna

Thanks for the PR, @lucasbru !

Here my feedback.

cadonna · 2024-01-19T09:57:57Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+    // Thread-safe queue to store user-defined callbacks and interceptors to be executed
+    private final BlockingQueue<OffsetCommitCallbackTask> callbackQueue = new LinkedBlockingQueue<>();
+
+    public void submitCommitInterceptors(final Map<TopicPartition, OffsetAndMetadata> offsets) {


nit:
I find the name of this method quite confusing. It says that it submits a commit interceptor, but actually it submits a call to the commit interceptor.
Similar applies to submitUserCallback().

cadonna · 2024-01-19T10:07:39Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+                (innerOffsets, exception) -> interceptors.onCommit(innerOffsets),
+                offsets,


nit:
That is also a bit confusing. What are innerOffsets? I think it would be better to rename innerOffsets to offsetsParam. Additionally, offsets could be renamed to actualOffsets but that is actually not needed since after the renaming of innerOffsets it should be quite clear what offsets represents.

cadonna · 2024-01-19T10:15:50Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+                    hasFencedException = true;
+
+                task.callback.onComplete(task.offsets, task.exception);
+


nit:

Suggested change

cadonna · 2024-01-19T10:36:55Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+                if (task.exception instanceof FencedInstanceIdException)
+                    hasFencedException = true;


I think this is not completely correct.
The javadocs for commitAsync() (w/o callback) say:

@throws org.apache.kafka.common.errors.FencedInstanceIdException if this consumer instance gets fenced by broker.

If no callback is passed into commitAsync(), no offset commit callback invocation is submitted. However, we only check for a FencedInstanceIdException when we execute a callback. It seems to me that with commitAsync() we would not throw at all when the consumer gets fenced.
In any case, we need a unit test that verifies that the FencedInstanceIdException is thrown for each version of commitAsync().

You are right, but this is not related to interceptors but a separate commit bug. I opened KAFKA-16169.

cadonna · 2024-01-19T11:18:23Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java

@@ -271,6 +283,7 @@ public CompletableFuture<Void> maybeAutoCommitAllConsumedNow(
        return (response, throwable) -> {
            autoCommitState.ifPresent(autoCommitState -> autoCommitState.setInflightCommitStatus(false));
            if (throwable == null) {
+                offsetCommitCallbackInvoker.submitCommitInterceptors(allConsumedOffsets);


Why is this only called in the case of an auto commit? Or is the name of this method misleading and this method is called also in case of async commits?

This is not only called in the case of an auto commit, but also in commitAsync.

OK, but then the name of the method should be changed to better reflect what it does.

You asked why is "this" only called in the case of the auto commit. I think you were referring to submitCommitInterceptors. It is not only called in the case of auto-commit, but also in commitAsync.

Then you mentioned that the name of "this method" is misleading. I think you are referring to maybeAutocommitAllConsumedNow. I don't think it is, because it is only called in an auto-commit.

OK, I see it now that submitCommitInterceptors() (now enqueueInterceptorInvocation) is also called in commitAsync(). I do not know why I haven't seen this before.

cadonna · 2024-01-19T12:17:33Z

...c/test/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvokerTest.java

+    public void setup() {
+        offsetCommitCallbackInvoker = new OffsetCommitCallbackInvoker(consumerInterceptors);
+    }
+


I think there some edge cases missing like empty interceptors and empty user callbacks. Also only interceptors is also not tested.

lucasbru

@cadonna Thanks for the review. I addressed the comments that I believe should be addressed in this PR.

lucasbru · 2024-01-19T13:48:58Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java

@@ -271,6 +283,7 @@ public CompletableFuture<Void> maybeAutoCommitAllConsumedNow(
        return (response, throwable) -> {
            autoCommitState.ifPresent(autoCommitState -> autoCommitState.setInflightCommitStatus(false));
            if (throwable == null) {
+                offsetCommitCallbackInvoker.submitCommitInterceptors(allConsumedOffsets);


This is not only called in the case of an auto commit, but also in commitAsync.

lucasbru · 2024-01-19T13:54:20Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+    // Thread-safe queue to store user-defined callbacks and interceptors to be executed
+    private final BlockingQueue<OffsetCommitCallbackTask> callbackQueue = new LinkedBlockingQueue<>();
+
+    public void submitCommitInterceptors(final Map<TopicPartition, OffsetAndMetadata> offsets) {


lucasbru · 2024-01-19T13:54:38Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+                (innerOffsets, exception) -> interceptors.onCommit(innerOffsets),
+                offsets,


lucasbru · 2024-01-19T14:08:49Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+                if (task.exception instanceof FencedInstanceIdException)
+                    hasFencedException = true;


You are right, but this is not related to interceptors but a separate commit bug. I opened KAFKA-16169.

lucasbru · 2024-01-19T14:08:56Z

...s/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvoker.java

+                    hasFencedException = true;
+
+                task.callback.onComplete(task.offsets, task.exception);
+


lucasbru · 2024-01-19T14:16:22Z

...c/test/java/org/apache/kafka/clients/consumer/internals/OffsetCommitCallbackInvokerTest.java

+    public void setup() {
+        offsetCommitCallbackInvoker = new OffsetCommitCallbackInvoker(consumerInterceptors);
+    }
+


cadonna

Thanks @lucasbru !

I have one comment.

cadonna · 2024-01-29T14:17:17Z

clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java

+    }
+
+    @Test
+    public void testNoInterceptorCommitAsyncFailed() {


Is there a reason why we do not have the corresponding test for commitSync()?

I added one.

cadonna

Thanks for the update and the patience, @lucasbru !

LGTM!

lucasbru · 2024-01-29T20:25:40Z

Test failures are unrelated

…ache#15000) We need to make sure to call the consumer interceptor and test its integration. This is adding the required call in commitSync and commitAsync. The calls in commitAsync are executed using the same mechanism as commit callbacks, to ensure that we are calling the interceptors from a single thread, as is intended in the original KIP. The interceptors also need to be invoked on auto-commits which are executed in the commit request manager. For this purpose, we share the OffsetCommitCallbackInvoker class with the background thread (it is already accessed implicitly from the background thread through a future lambda). This is done analogous to the RebalanceListenerInvoker. Co-authored-by: John Doe [email protected] Reviewers: Bruno Cadonna <[email protected]>, Andrew Schofield <[email protected]>, Philip Nee <[email protected]>

vamossagar12 mentioned this pull request Dec 15, 2023

KAFKA-15942: Implement ConsumerInterceptor #14963

Closed

3 tasks

implement consumer interceptors

db405f1

lucasbru force-pushed the ctr_interceptors branch from e9e7ef7 to db405f1 Compare December 19, 2023 17:05

lucasbru requested a review from cadonna December 19, 2023 17:06

lucasbru commented Dec 19, 2023

View reviewed changes

lucasbru added 2 commits January 2, 2024 18:33

tests

8ca984f

Merge remote-tracking branch 'upstream/trunk' into ctr_interceptors

bc2a8da

lucasbru requested a review from philipnee January 3, 2024 08:52

autocommit logic

e103568

lucasbru commented Jan 12, 2024

View reviewed changes

AndrewJSchofield requested changes Jan 12, 2024

View reviewed changes

philipnee reviewed Jan 12, 2024

View reviewed changes

lucasbru added 3 commits January 15, 2024 10:39

Address comments

8ec95f9

update TODO

b6be0b9

move commit callback invocation on close

4985460

philipnee reviewed Jan 16, 2024

View reviewed changes

comments

31cf64c

philipnee reviewed Jan 17, 2024

View reviewed changes

philipnee approved these changes Jan 17, 2024

View reviewed changes

philipnee added consumer ctr Consumer Threading Refactor (KIP-848) labels Jan 17, 2024

do not fail close if fenced

68740cd

cadonna reviewed Jan 19, 2024

View reviewed changes

lucasbru added 5 commits January 19, 2024 15:18

Bruno's comments

4434f4b

Merge remote-tracking branch 'upstream/trunk' into ctr_interceptors

c6d07dd

Merge remote-tracking branch 'upstream/trunk' into ctr_interceptors

47b21e1

fix

4325279

fix

fd687dd

lucasbru commented Jan 19, 2024

View reviewed changes

cadonna reviewed Jan 29, 2024

View reviewed changes

lucasbru added 2 commits January 29, 2024 15:44

more testing

2d6f231

Merge remote-tracking branch 'upstream/trunk' into ctr_interceptors

4b1c1c4

cadonna approved these changes Jan 29, 2024

View reviewed changes

lucasbru merged commit 1a54c25 into apache:trunk Jan 29, 2024
1 check failed

		(innerOffsets, exception) -> interceptors.onCommit(innerOffsets),
		offsets,

		hasFencedException = true;

		task.callback.onComplete(task.offsets, task.exception);

		if (task.exception instanceof FencedInstanceIdException)
		hasFencedException = true;

KAFKA-15942: Implement ConsumerInterceptors in the async consumer #15000

KAFKA-15942: Implement ConsumerInterceptors in the async consumer #15000

Conversation

lucasbru commented Dec 13, 2023 • edited Loading

Committer Checklist (excluded from commit message)

lucasbru Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

lucasbru commented Jan 3, 2024

philipnee commented Jan 3, 2024

lucasbru commented Jan 4, 2024

philipnee commented Jan 4, 2024

lucasbru commented Jan 5, 2024

AndrewJSchofield commented Jan 9, 2024

lucasbru commented Jan 10, 2024

lianetm commented Jan 11, 2024

AndrewJSchofield commented Jan 11, 2024

philipnee commented Jan 11, 2024

lucasbru commented Jan 12, 2024

lucasbru commented Jan 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucasbru commented Jan 15, 2024

lucasbru commented Jan 15, 2024

lucasbru commented Jan 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucasbru Jan 17, 2024 • edited Loading

Choose a reason for hiding this comment

philipnee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philipnee left a comment

Choose a reason for hiding this comment

cadonna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucasbru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cadonna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cadonna left a comment

Choose a reason for hiding this comment

lucasbru commented Jan 29, 2024

lucasbru commented Dec 13, 2023 •

edited

Loading

lucasbru Dec 19, 2023 •

edited

Loading

lucasbru Jan 17, 2024 •

edited

Loading