feat(cdevents-notification): Implementing produce CDEvents using Notification #1295

rjalander · 2023-05-15T15:43:42Z

Implementing Produce CDEvents as per the RFC design
https://github.com/spinnaker/governance/blob/master/rfc/cdevents-spinnaker.md#produce-cdevents-1

This PR has the changes to implement a new CDEventsNotificationAgent, to handle the CDEvent type of Notifications and send them to address configured as events-broker URL using CDEvents-Java-SDK

Note:
Need some clarifications on type of CDEvents that Spinnaker needs to send,

Is this something will allow Spinnaker to send all type of CDEvents core, source code, CI, CD and Operations Events as a Notification
Or do we need to restrict to a subset of CDEvents(core, CD,..) which is suitable for Spinnaker

Current PR has the implementation to produce Core CDEvents for now, other events will be implemented after this PR review and discussion.

Dependent Spinnaker/deck PR - spinnaker/deck#9997

…fication agent

…ification

Signed-off-by: Jalander Ramagiri <[email protected]>

jasonmcintosh · 2023-09-06T14:10:13Z

...notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsSenderService.java

+   */
+  public HttpURLConnection sendCDEvent(CloudEvent ceToSend, URL url) throws IOException {
+    HttpURLConnection httpUrlConnection = createConnection(url);
+    MessageWriter messageWriter = createMessageWriter(httpUrlConnection);


A concern here is this is REALLY raw java layer for HTTP communications. The standard in spinnaker is to use retrofit & ok-http client libs. These automatically add things LIKE timeouts, URL validation, etc. that are pretty key settings for most environments. I've not looked at the CloudEvents MessageWriter system... so may HAVE to do this, but definitely have concerns.

+1 on this. Let's use an actual HTTP client, please.

I am trying to use Retrofit client to send CDEvent but facing difficulties in deserializing as Retrofit doesn't directly use CloudEventHttpMessageConverter is the issue.
Can I use spring RestTemplate to send the CDEvent instead, please share your ideas?

With retrofit and a return type of Response (e.g. here), I believe you'll get flexibility on converting the response.

I could able to solve this issue by creating a custom CDEventsHTTPMessageConverter as retrofit doesn't directly support CloudEventHttpMessageConverter, this converter uses Jackson converter to map the CloudEvent as String and send this String with the Requestbody

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

jasonmcintosh · 2023-09-06T15:09:24Z

...tions/src/main/groovy/com/netflix/spinnaker/echo/notification/CDEventsNotificationAgent.java

+
+    } catch (Exception e) {
+      log.error("Exception occurred while sending CDEvent {}", e);
+      throw new CDEventsException("Exception occurred while sending CDEvent", e);


As a rule, we should be throwing spinnaker exceptions not custom exceptions for these.

One thing to call out here, is unless you are modifying the retrofit exception handler, these custom exceptions may not be handled properly (I dont think). One thing we could consider doing is having some base spinnaker exception that custom exceptions could inherit from which could make throwing custom exceptions a little more flexible

CDEventsException is not needed now as using the retrofit request/response to send CDEvents notification

xibz · 2023-10-03T21:52:33Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+      String status,
+      String spinnakerUrl) {
+
+    String configType = Optional.ofNullable(config).map(c -> (String) c.get("type")).orElse(null);


is null valid? We probably want to throw an exception here?

yeah, throwing an existing FieldNotFoundException here

xibz · 2023-10-03T21:53:09Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+      String spinnakerUrl) {
+
+    String configType = Optional.ofNullable(config).map(c -> (String) c.get("type")).orElse(null);
+    String configLink = Optional.ofNullable(config).map(c -> (String) c.get("link")).orElse(null);


This is odd too, considering both these fields could not be present, but we would still try to build an executionUrl from it. I think we should probably throw an exception, unless we have proper defaults

yeah, throwing an existing FieldNotFoundException here

xibz · 2023-10-03T21:54:38Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+      String status) {
+    CloudEvent ceToSend = null;
+    switch (cdEventsType) {
+      case "dev.cdevents.pipelinerun.queued":


Aren't these defined in the CDEvents SDK? I'd rather pull the enums from there

Yes I can pull that, there are Enum constants created with event types in the CDEvents SDK,

PipelineRunFinishedEvent("dev.cdevents.pipelinerun.finished."), PipelineRunQueuedEvent("dev.cdevents.pipelinerun.queued."),

May be I need to compare with substring of a cdEventsType to use with Enum constants

Yes, please do that :)

xibz · 2023-10-03T21:56:59Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+                executionId, executionUrl, executionName, spinnakerUrl, status);
+        break;
+      default:
+        throw new CDEventsException(


Are we only interesting int he pipelinerun and taskrun events?

Also, most of these have the same parameter, is there an interface that can be used here? That way we can just have a map of those interfaces. Should simplify this some and makes adding to that map easier in the future

As per my comment in the PR description, I will update the same PR/create another PR for all other events once this review is closed.
Current PR has the implementation to produce [Core CDEvents](https://github.com/cdevents/spec/blob/v0.1.2/core.md) for now, other events will be implemented after this PR review and discussion.

My question here is does Spinnaker needs all types of events which CDEvents spec provides(https://cdevents.dev/docs/) or needs a subset of event types based on any Spinnaker requirements.
Like I am not sure whether Source Code Control Events are required for Spinnaker to send.

There is no such interface to create CDEvents(might be a good idea to propose this change from SDK itself as that interface/implementations can be used as common),
I can update the current switch case to a map for flexibility and extensibility purpose.

There is no such interface to create CDEvents(might be a good idea to propose this change from SDK itself as that interface/implementations can be used as common)

What I was meaning though is:

public interface CDEventCreateTask { public CloudEvent create(String executionId, String executionUrl, String executionName, String spinnakerUrl, String status); }

public class CDEventCreateTaskRunFinished implements CDEventCreateTask { // Code here }

Then in this class

private final Map<CDEventTypeEnum, CDEventCreateTask> createTasks = Map.of ( CDEventTaskRunFinished, new CDEventCreateTaskRunFinished(), );

That way the switch statement goes away and instead becomes something like

CDEventCreateTask createTask = createTasks.get(eventTypeEnum); if (createTask == null) { throw new SomeException("Invalid CDEvent create task: " + eventTypeEnum.toString()); }

yes I'm in the same direction, was thinking aloud rather implementing in the Spinnaker is that a good idea to create these interface and classes in the CDEvents Java-SDK itself,
but I know It will restrict to create CDEvents the way Spinnaker wants with any additional changes.
Will create these classes with little modifications here.

public interface CDEventCreator { //other type of events can have different parameters, so will remove the params CloudEvent createCDEvent(); }

Implement the classes for each event

public class CDEventPipelineRunQueued implements CDEventCreator { //each event can have different parameters, can go here private String source; private String subjectId; private String subjectSource; private String subjectPipelineName; private String subjectUrl; public CDEventPipelineRunQueued(String executionId, String executionUrl, String executionName, String spinnakerUrl) { this.source = spinnakerUrl; this.subjectId = executionId; this.subjectSource = spinnakerUrl; this.subjectPipelineName = executionName; this.subjectUrl = executionUrl; } @Override public CloudEvent createCDEvent() { PipelineRunQueuedCDEvent cdEvent = new PipelineRunQueuedCDEvent(); //set the params }

creating a map of different events in this class,

Map<CDEventTypeEnum, CDEventCreator> createTasks = Map.of ( PipelineRunQueuedEnum, new CDEventPipelineRunQueued(executionId, executionUrl, executionName, spinnakerUrl) );

xibz · 2023-10-03T21:57:58Z

...notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsSenderService.java

+   */
+  public HttpURLConnection sendCDEvent(CloudEvent ceToSend, URL url) throws IOException {
+    HttpURLConnection httpUrlConnection = createConnection(url);
+    MessageWriter messageWriter = createMessageWriter(httpUrlConnection);


+1 on this. Let's use an actual HTTP client, please.

xibz · 2023-10-03T22:00:17Z

...tions/src/main/groovy/com/netflix/spinnaker/echo/notification/CDEventsNotificationAgent.java

+
+    } catch (Exception e) {
+      log.error("Exception occurred while sending CDEvent {}", e);
+      throw new CDEventsException("Exception occurred while sending CDEvent", e);


One thing to call out here, is unless you are modifying the retrofit exception handler, these custom exceptions may not be handled properly (I dont think). One thing we could consider doing is having some base spinnaker exception that custom exceptions could inherit from which could make throwing custom exceptions a little more flexible

…otification_cdevents

xibz · 2023-10-14T08:02:54Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+      String executionName,
+      String spinnakerUrl,
+      String status) {
+    TaskRunFinishedCDEvent cdEvent = new TaskRunFinishedCDEvent();


Then all these classes can just implement that interface I mentioned above

xibz · 2023-10-17T18:44:46Z

...notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventTaskRunStarted.java

+
+public class CDEventTaskRunStarted implements CDEventCreator {
+
+  private String source;


We can probably move a lot of these to some base CDEvent class, since I imagine some of these will always be required for all CDEventCreators.

yes, making CDEventCreator class as an abstract and moving the common properties to this class

xibz · 2023-10-23T14:39:41Z

echo-notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventCreator.java

+    this.subjectUrl = subjectUrl;
+  }
+
+  public String getSource() {


Lombok can be used here

See
@Getter
@Setter
Makes the code a little more clean :)

So you'd have something like

public abstract class CDEventCreator { @Getter private String source; }

yeah that looks good, using Lombok in other sub classes too.

xibz · 2023-10-23T14:42:10Z

echo-notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventCreator.java

+
+import io.cloudevents.CloudEvent;
+
+public abstract class CDEventCreator {


This name is a little confusing. CDEventCreator? It's not really creating a CDEvent. It's more of a BaseCDEvent or something of that nature. Im sure there are better names than BaseCDEvent, but I think I'd much prefer that over CDEventCreator since no creating is happening here.

renamed to BaseCDEvent

xibz

I just had a couple comments, but overall looks good. Can we also add docs around the CDEventCreator? Just something to outline what it is and link to CDEvents in general

nemesisOsorio · 2023-10-23T16:15:00Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+    String configType =
+        Optional.ofNullable(config)
+            .map(c -> (String) c.get("type"))
+            .orElseThrow(FieldNotFoundException::new);
+    String configLink =
+        Optional.ofNullable(config)
+            .map(c -> (String) c.get("link"))
+            .orElseThrow(FieldNotFoundException::new);
+
+    String executionId =
+        Optional.ofNullable(event.content)
+            .map(e -> (Map) e.get("execution"))
+            .map(e -> (String) e.get("id"))
+            .orElseThrow(FieldNotFoundException::new);


can we add the field name to the exceptions?
or log what is the missing field

nemesisOsorio · 2023-10-23T16:19:37Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+    } else {
+      return cdEventCreator.createCDEvent();
+    }


else not needed

nemesisOsorio · 2023-10-23T16:21:02Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java

+    for (String keyType : cdEventsMap.keySet()) {
+      if (keyType.contains(cdEventsType)) {
+        cdEventCreator = cdEventsMap.get(keyType);
+        break;
+      }
+    }


can we use a stream and a filter?

updated to use stream and filter

nemesisOsorio · 2023-10-23T16:36:17Z

...ations/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsHTTPMessageConverter.java

+
+  private final ObjectMapper objectMapper;
+
+  private static final String MIME_TYPE = "application/json";


I believe there is a constant in spring for this

yes, using MediaType.APPLICATION_JSON_VALUE from spring

nemesisOsorio · 2023-10-23T16:39:43Z

...ations/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsHTTPMessageConverter.java

+    } catch (JsonParseException e) {
+      throw new ConversionException(e);
+    } catch (JsonMappingException e) {
+      throw new ConversionException(e);
+    } catch (IOException e) {
+      throw new ConversionException(e);
+    }


seems like we can use catch (JsonParseException | JsonMappingException | IOException e)

IOException can not be used with multi-catch as the other Exception classes are subclass of IOException

Using multi-catch for subclasses alone,
catch (JsonParseException | JsonMappingException e)

nemesisOsorio · 2023-10-23T16:40:25Z

...-notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsSenderClient.java

+package com.netflix.spinnaker.echo.cdevents;
+
+import retrofit.client.Response;
+import retrofit.http.*;


I believe we should avoid wildcards

yeah corrected, it was auto imported,

nemesisOsorio · 2023-10-23T16:41:13Z

...notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsSenderService.java

+            request.addHeader("Ce-Specversion", cdEvent.getSpecVersion().V1.toString());
+            request.addHeader("Ce-Source", cdEvent.getSource().toString());
+            request.addHeader("Ce-Type", cdEvent.getType());
+            request.addHeader("Content-Type", "application/json");


reuse a constant

…otification_cdevents

nemesisOsorio · 2023-10-24T15:52:39Z

...ications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventPipelineRunFinished.java

+    if (subjectError.equals("complete")) {
+      cdEvent.setSubjectOutcome(CDEventConstants.Outcome.SUCCESS);
+    } else if (subjectError.equals("failed")) {
+      cdEvent.setSubjectOutcome(CDEventConstants.Outcome.FAILURE);
+    }


these could produce a NPE let's change them to "complete".equals(subjectError)

nemesisOsorio · 2023-10-24T15:54:50Z

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventTaskRunFinished.java

+    if (getSubjectError().equals("complete")) {
+      cdEvent.setSubjectOutcome(CDEventConstants.Outcome.SUCCESS);
+    } else if (getSubjectError().equals("failed")) {
+      cdEvent.setSubjectOutcome(CDEventConstants.Outcome.FAILURE);
+    }


avoid NPE using "failed".equals(getSubjectError())

nemesisOsorio · 2023-10-24T16:00:15Z

...notifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsSenderService.java

+      return endPointURL;
+    } catch (MalformedURLException e) {
+      throw new InvalidRequestException(
+          "Unable to determine base URL from Microsoft Teams webhook URL.", e);


is this only related to Microsoft Teams?

Apologies for the typo, corrected with valid message

nemesisOsorio · 2023-10-24T16:01:29Z

...tions/src/main/groovy/com/netflix/spinnaker/echo/notification/CDEventsNotificationAgent.java

@@ -0,0 +1,85 @@
+/*
+ * Copyright 2020 Cerner Corporation


2023 Nordix?

Yes Updated

nemesisOsorio · 2023-10-24T16:03:26Z

...src/test/groovy/com/netflix/spinnaker/echo/notification/CDEventsNotificationAgentSpec.groovy

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2016 Google, Inc.


2023 Nordix?

yes, updated

jasonmcintosh

Overall better. Frankly the APIs are a TOUCh rough on cdevents side. Example:

EVERY event right now looks like it duplicates that same call cdEvent.setSource(URI.create(getSource())); uniquely. There isn't a base object that could handle this and reduce the duplication. There appear to be about 3 or 4 fields which are duplicated across ALL cdevents stuff
The URI parsing has a few places of "hey this is probably non ideal" on things like the format of spinnaker URls & such, but... for the moment it'd work. Just a personal preference on UI urls vs. API urls and encoding. Just depends on how downstream handles it, not enough to "block" the PR at this time.

xibz · 2023-10-26T16:57:27Z

EVERY event right now looks like it duplicates that same call cdEvent.setSource(URI.create(getSource())); uniquely. There isn't a base object that could handle this and reduce the duplication. There appear to be about 3 or 4 fields which are duplicated across ALL cdevents stuff

@jasonmcintosh The SDKs are very low level currently for CDEvents. So no abstractions have been added to make things much easier. Generally when you generating from openapi, smithy, and any other API language, they may not generate base classes especially for languages like golang. It's completely up to the generator. However, what most people do is use that to generate clients but are not meant to be used as the whole SDK. You see this in AWS SDKs where the clients are actually very low level, but there are higher level abstractions that make it easier. Due to CDEvents being still early in its life, these abstractions have no been made yet, unfortunately. This should get better with time though :).

jasonmcintosh · 2023-10-26T16:58:49Z

Exactly :) Let's get this merged, and see how it gets better!

xibz

LGTM! 🚢

Jalander Ramagiri added 3 commits May 15, 2023 16:28

feat(cdevents-notification): Implementing produce CDEvents using Noti…

483a461

…fication agent

feat(cdevents-notification): unit tests to produce CDEvents using Not…

3b759ee

…ification

feat(cdevents-notification): update variable name to cdEventsType

576e7d1

rjalander mentioned this pull request May 19, 2023

feat(cdevents-notification): CDEvents notification to produce CDEvents spinnaker/deck#9997

Merged

rjalander added 5 commits May 22, 2023 10:24

Merge branch 'master' into notification_cdevents

b943650

Merge branch 'master' into notification_cdevents

68ff04e

Merge branch 'master' into notification_cdevents

174e51b

Merge branch 'master' into notification_cdevents

25e6b7a

Merge branch 'master' into notification_cdevents

09237b9

rjalander mentioned this pull request Jul 24, 2023

Generate events using jsonschema2pojo and mustache template cdevents/sdk-java#58

Merged

Jalander Ramagiri and others added 5 commits July 26, 2023 09:59

Updating cdevents-sdk-java dependency version

de76a8c

Signed-off-by: Jalander Ramagiri <[email protected]>

fix: unit test failure with sdk

02a6610

Merge branch 'master' into notification_cdevents

f687deb

Merge branch 'master' into notification_cdevents

34c40c9

Merge branch 'master' into notification_cdevents

6ff8064

rjalander mentioned this pull request Aug 21, 2023

feat(cdevents-webhooks) : Consume CDEvents webhook API implementation spinnaker/gate#1651

Merged

Merge branch 'master' into notification_cdevents

71f5c8f

jasonmcintosh reviewed Sep 6, 2023

View reviewed changes

...otifications/src/main/groovy/com/netflix/spinnaker/echo/cdevents/CDEventsBuilderService.java Outdated Show resolved Hide resolved

jasonmcintosh reviewed Sep 6, 2023

View reviewed changes

Merge branch 'master' into notification_cdevents

e7cf714

xibz suggested changes Oct 3, 2023

View reviewed changes

rjalander and others added 5 commits October 12, 2023 09:43

Merge branch 'master' into notification_cdevents

cb9e3a7

using retrofit to send cdevent

6470546

Merge branch 'notification_cdevents' of github.com:Nordix/echo into n…

ac7262a

…otification_cdevents

fixing format violations

65e8b33

addressing review comments

f573844

xibz reviewed Oct 14, 2023

View reviewed changes

Jalander Ramagiri added 2 commits October 16, 2023 16:48

Interface to create CDEvents

a24cabd

update Copyright information

f9fa051

rjalander requested a review from dbyron-sf October 17, 2023 08:43

xibz reviewed Oct 17, 2023

View reviewed changes

Jalander Ramagiri and others added 2 commits October 18, 2023 13:19

Making CDEventCreator as an abstract

2eecfe0

Merge branch 'master' into notification_cdevents

af917bb

rjalander requested a review from xibz October 18, 2023 13:58

Merge branch 'master' into notification_cdevents

7b832ee

xibz reviewed Oct 23, 2023

View reviewed changes

xibz suggested changes Oct 23, 2023

View reviewed changes

nemesisOsorio reviewed Oct 23, 2023

View reviewed changes

nemesisOsorio suggested changes Oct 23, 2023

View reviewed changes

Jalander Ramagiri added 3 commits October 24, 2023 10:30

Class name changed to BaseCDEvent and uing lombok.Getter

fc31963

Merge branch 'notification_cdevents' of github.com:Nordix/echo into n…

0ebb3ce

…otification_cdevents

addressing review comments

1f783f6

rjalander requested review from xibz and nemesisOsorio October 24, 2023 14:27

addressing minor review comments

621da6b

nemesisOsorio suggested changes Oct 24, 2023

View reviewed changes

nemesisOsorio mentioned this pull request Oct 24, 2023

REQUEST: New Approver status for nemesisOsorio spinnaker/governance#318

Open

6 tasks

rjalander requested a review from nemesisOsorio October 24, 2023 16:59

nemesisOsorio approved these changes Oct 24, 2023

View reviewed changes

jasonmcintosh approved these changes Oct 26, 2023

View reviewed changes

xibz approved these changes Oct 26, 2023

View reviewed changes

jasonmcintosh added the ready to merge Approved and ready for merge label Oct 26, 2023

mergify bot added the auto merged label Oct 26, 2023

mergify bot merged commit 8bdcaa5 into spinnaker:master Oct 26, 2023
4 checks passed

spinnakerbot added the target-release/1.33 label Oct 26, 2023

aman-agrawal mentioned this pull request Jun 7, 2024

Upgrade 1.30.x to 1.33.x OpsMx/echo-oes#27

Closed


		public class CDEventTaskRunStarted implements CDEventCreator {

		private String source;


		import io.cloudevents.CloudEvent;

		public abstract class CDEventCreator {


		private final ObjectMapper objectMapper;

		private static final String MIME_TYPE = "application/json";

feat(cdevents-notification): Implementing produce CDEvents using Notification #1295

feat(cdevents-notification): Implementing produce CDEvents using Notification #1295

Conversation

rjalander commented May 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xibz Oct 14, 2023 • edited Loading

Choose a reason for hiding this comment

rjalander Oct 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xibz Oct 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xibz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonmcintosh left a comment

Choose a reason for hiding this comment

xibz commented Oct 26, 2023 • edited Loading

jasonmcintosh commented Oct 26, 2023

xibz left a comment

Choose a reason for hiding this comment

rjalander commented May 15, 2023 •

edited

Loading

xibz Oct 14, 2023 •

edited

Loading

rjalander Oct 16, 2023 •

edited

Loading

xibz Oct 23, 2023 •

edited

Loading

xibz commented Oct 26, 2023 •

edited

Loading