Problem with setting SQS queue #1068

tomaszkrys · 2021-02-17T16:19:18Z

Describe the bug
We are creating EventSource on EKS k8s cluster that should consume from SQS

To Reproduce
Steps to reproduce the behavior:

kind: EventSource
metadata:
  name: aws-sqs
spec:
  sqs:
    sqs-git:
      jsonBody: true
      accessKey:
        key: accesskey
        name: aws-secret
      secretKey:
        key: secretkey
        name: aws-secret
      region: eu-west-1
      queue: test

we also have aws secrets as in the example:

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
type: Opaque
data:
   accesskey: XXXXX
   secretkey: XXXXX

we can see in logs that it is trying to generate aws sqs url, but because of some reason it is failling:

(...)
2021-02-17T16:04:45.743Z	INFO	argo-events.eventsource	awssqs/start.go:76	fetching queue url...	{"eventSourceName": "aws-sqs", "eventSourceType": "sqs", "eventName": "sqs-git"}
2021-02-17T16:04:46.064Z	ERROR	argo-events.eventsource	eventsources/eventing.go:355	failed to start listening eventsource	{"eventSourceName": "aws-sqs", "eventSourceType": "sqs", "eventName": "sqs-git", "error": "timed out waiting for the condition"}
github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start.func3
	/home/runner/work/argo-events/argo-events/eventsources/eventing.go:355
2021-02-17T16:04:46.064Z	ERROR	argo-events.eventsource	eventsources/eventing.go:378	Erroring out, no active event server running	{"eventSourceName": "aws-sqs"}
github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start
	/home/runner/work/argo-events/argo-events/eventsources/eventing.go:378
main.main
	/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:58
runtime.main
	/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/proc.go:203
2021-02-17T16:04:46.064Z	INFO	argo-events.eventsource	eventsources/eventing.go:291	exiting eventbus connection daemon...	{"eventSourceName": "aws-sqs"}
2021-02-17T16:04:46.066Z	ERROR	argo-events.eventsource	driver/nats.go:84	NATS connection lost	{"eventSourceName": "aws-sqs", "clientID": "aws-sqs-eventsource-pvg7l-6d5f58f56b-bssz5"}
github.com/argoproj/argo-events/eventbus/driver.(*natsStreaming).Connect.func1
	/home/runner/work/argo-events/argo-events/eventbus/driver/nats.go:84
github.com/nats-io/nats%2ego.(*Conn).close.func1
	/home/runner/go/pkg/mod/github.com/nats-io/[email protected]/nats.go:3722
github.com/nats-io/nats%2ego.(*asyncCallbacksHandler).asyncCBDispatcher
	/home/runner/go/pkg/mod/github.com/nats-io/[email protected]/nats.go:2008
2021-02-17T16:04:46.066Z	FATAL	argo-events.eventsource	cmd/main.go:59	failed to start eventsource server	{"eventSourceName": "aws-sqs", "error": "no active event server running", "errorVerbose": "no active event server running\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:381\nmain.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:58\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/proc.go:203\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/asm_amd64.s:1373"}
main.main
	/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:59
runtime.main
	/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/proc.go:203

Expected behavior
Consume SQS messages

Environment (please complete the following information):

Kubernetes: v1.16.15-eks-ad4801
Argo Events: v1.2.2

Additional context
Webhooks are working without any problems

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

The text was updated successfully, but these errors were encountered:

whynowy · 2021-02-18T08:48:13Z

Can you try to specify queueAccountId like https://github.com/argoproj/argo-events/blob/master/examples/event-sources/aws-sqs.yaml#L51?

// To access a queue that belongs to another AWS account, use the QueueOwnerAWSAccountId
// parameter to specify the account ID of the queue's owner. The queue's owner
// must grant you permission to access the queue. For more information about
// shared queue access, see AddPermission or see Allow Developers to Write Messages
// to a Shared Queue (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-writing-an-sqs-policy.html#write-messages-to-shared-queue)
// in the Amazon Simple Queue Service Developer Guide.

tomaszkrys · 2021-02-18T09:22:40Z

I did created this sqs queue with same AWS credentials witch i'm using to argo eventsource.
But i did tried and got same error as before, also maybe worth to mention this part of log:

{"level":"info","ts":1613639602.992304,"logger":"argo-events.eventsource","caller":"awssqs/start.go:64","msg":"started processing the AWS SQS event source...","eventSourceName":"aws-sqs","eventSourceType":"sqs","eventName":"sqs-git"}
{"level":"info","ts":1613639602.9925158,"logger":"argo-events.eventsource","caller":"awssqs/start.go:76","msg":"fetching queue url...","eventSourceName":"aws-sqs","eventSourceType":"sqs","eventName":"sqs-git"}

is repeated few times before this error. I did also added env DEBUG_LOG:true but log looks same as before.

What i also did is entered aws-sqs-eventsource pod with some hack [https://github.com/kubernetes-retired/contrib/blob/master/scratch-debugger/debug.sh] and i did checked AWS credentials:

/argo-events/secrets/aws-secret # ls
accesskey  secretkey
/argo-events/secrets/aws-secret # cat accesskey
XXXX
/argo-events/secrets/aws-secret # cat secretkey
XXXX

and they are present and ofc XXXX have correct values.

Whats more interesting i did passed empty or not correct aws credentials, and the log looks still the same.

whynowy · 2021-02-18T16:27:32Z

Have you added queueAccountId to your spec?

tomaszkrys · 2021-02-18T21:28:55Z

Yep, as pointed in previous message i did tried that, but no luck:

But i did tried and got same error as before ...

santoshsb · 2021-03-23T20:33:46Z

We are also facing the same issue, let us know if more information is needed.

santoshsb · 2021-03-24T21:36:55Z

We are also facing the same issue, let us know if more information is needed.

Fixed it by creating a new user and attaching a sqs access policy. Using the access key and secret from this user in the aws-secret.yaml. Initially we were trying to connect using a AWS role.

whynowy · 2021-03-24T22:20:57Z

@santoshsb - thanks for the update!

I would assume the problem is still the permission issue, @tomaszkrys - could you check it again?

phelinor · 2021-03-25T01:14:18Z

I'm still having the issue even after checking the keys and the permissions in AWS

@santoshsb could you please share your secrets.yaml (of course without the keys, I only would like to see the layout), and also the eventsource.yaml?

Also what permissions did you provide to the aws user for SQS?

Another thing, what version of Argo Events are you using?

santoshsb · 2021-03-25T04:37:07Z

@phelinor here is the secrets.yaml file

kind: Secret
metadata:
  name: aws-secret
type: Opaque
data:
  accesskey: <Base64EncodedAccessKey>
  secretkey: <Base64EncodedSecretKey>

eventsource.yaml

kind: EventSource
metadata:
  name: aws-sqs
spec:
  sqs:
    example:
      # jsonBody specifies that all event body payload coming from this
      # source will be JSON
      jsonBody: true
      # accessKey contains information about K8s secret that stores the access key
      accessKey:
        # Key within the K8s secret whose corresponding value (must be base64 encoded) is access key
        key: accesskey
        # Name of the K8s secret that contains the access key
        name: aws-secret
      # secretKey contains information about K8s secret that stores the secret key
      secretKey:
        # Key within the K8s secret whose corresponding value (must be base64 encoded) is secret key
        key: secretkey
        # Name of the K8s secret that contains the secret key
        name: aws-secret
      # aws region
      region: us-west-2
      # name of the queue. The eventsource resolves the url of the queue from the queue name.
      queue: test
      # The duration (in seconds) for which the call waits for a message to arrive in the queue before returning.
      # MUST BE > 0 AND <= 20
      waitTimeSeconds: 20

We created a new user called argo-events in IAM, with the following policy attached, than created the access key and secret key for this user to use in the aws-secret.yaml

We added this user ARN to the SQS queue access policy,

{
      "Sid": "__receiver_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::myAwsAcc:user/argo-events",
          "arn:aws:iam::myAwsAcc:root",
        ]
      },
      "Action": [
        "SQS:ChangeMessageVisibility",
        "SQS:DeleteMessage",
        "SQS:ReceiveMessage"
      ],
      "Resource": "arn:aws:sqs:us-west-2:<myAwsAcc>:test"
    }

Creation of the new user with the AmazonSQSFullAccess policy and attaching this user to the SQS queue policy receiver statement got it up and running. Hope this helps.

phelinor · 2021-03-25T17:11:40Z

@santoshsb thanks much for your answers. just one more, what Argo Events Version are you using?

santoshsb · 2021-03-25T17:46:23Z

@phelinor here is what I found by looking at the deployed controller files,

image: argoproj/eventsource-controller:v1.2.2
image: argoproj/sensor-controller:v1.2.2
image: argoproj/eventbus-controller:v1.2.2

If there is a command to get the overall Argo Events version do let us know, will run the same.

argo: v2.9.1
BuildDate: 2020-07-03T15:06:40Z
GitCommit: 6b967d08c0a142aaa278538f2407c28de467262e
GitTreeState: clean
GitTag: v2.9.1
GoVersion: go1.13.4
Compiler: gc
Platform: linux/amd64

phelinor · 2021-03-25T18:07:08Z

@santoshsb thanks again, that is what I need

phelinor · 2021-03-25T21:22:26Z

@whynowy sorry to keep bugging but after trying the above suggestions I continue to have the same issue with the EventSource in my LOCAL cluster.

This is my setup:

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
type: Opaque
data:
  accesskey: akey in Base64
  secretkey: skey in Base64
---
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: aws-sqs-event-source-local
spec:
  sqs:
    sqs-connection:
      namespace: "hugo"
      region: "us-west-2"
      queue: "phelinor.fifo"
      waitTimeSeconds: 10
      jsonBody: true
      queueAccountId: "164506......."
      accessKey:
        key: accesskey
        name: aws-secret
      secretKey:
        key: secretkey
        name: aws-secret

The main difference is I'm providing the AWS queueAccountId: "164506......." (also tried not providing it, same results)

These are the last error messages I get:

{"level":"info","ts":1616705287.6004927,"logger":"argo-events.eventsource","caller":"awssqs/start.go:64","msg":"started processing the AWS SQS event source...","eventSourceName":"aws-sqs-event-source-local","eventSourceType":"sqs","eventName":"sqs-connection"}
{"level":"info","ts":1616705287.600797,"logger":"argo-events.eventsource","caller":"awssqs/start.go:76","msg":"fetching queue url...","eventSourceName":"aws-sqs-event-source-local","eventSourceType":"sqs","eventName":"sqs-connection"}
{"level":"error","ts":1616705287.9258828,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:355","msg":"failed to start listening eventsource","eventSourceName":"aws-sqs-event-source-local","eventSourceType":"sqs","eventName":"sqs-connection","error":"timed out waiting for the condition","stacktrace":"github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start.func3\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:355"}
{"level":"error","ts":1616705287.9260561,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:378","msg":"Erroring out, no active event server running","eventSourceName":"aws-sqs-event-source-local","stacktrace":"github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:378\nmain.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:58\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/proc.go:203"}
{"level":"info","ts":1616705287.9261007,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:291","msg":"exiting eventbus connection daemon...","eventSourceName":"aws-sqs-event-source-local"}
{"level":"fatal","ts":1616705287.9438224,"logger":"argo-events.eventsource","caller":"cmd/main.go:59","msg":"failed to start eventsource server","eventSourceName":"aws-sqs-event-source-local","error":"no active event server running","errorVerbose":"no active event server running\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:381\nmain.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:58\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/proc.go:203\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/asm_amd64.s:1373","stacktrace":"main.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:59\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.13/x64/src/runtime/proc.go:203"}

whynowy · 2021-03-25T21:35:59Z

@phelinor - What is the cluster you are running? Could you confirm if it has the connectivity to AWS?

Also, if you could try out the latest v1.3.0-rc1 release, it prints out the error log if there's an issue with fetching queue URL.

phelinor · 2021-03-25T22:47:40Z

Macos Catalina ver 10.15.6 -> Docker Desktop 3.2.2 (61853) -> K8s v1.19.3

I can confirm that my K8s Local setup is able to connect to AWS because I just ran a Workflow from the Argo UI that uses docker images stored in the Elastic Container Registry like this:

I will also try running Argo v1.3.0-rc1 to see what other errors I get

phelinor · 2021-03-26T19:56:55Z

New Errors:

{"level":"info","ts":1616721784.276223,"logger":"argo-events.eventsource","caller":"awssqs/start.go:66","msg":"started processing the AWS SQS event source...","eventSourceName":"aws-sqs-event-source-local","eventSourceType":"sqs","eventName":"sqs-connection"}
{"level":"info","ts":1616721784.2775762,"logger":"argo-events.eventsource","caller":"awssqs/start.go:79","msg":"fetching queue url...","eventSourceName":"aws-sqs-event-source-local","eventSourceType":"sqs","eventName":"sqs-connection"}
{"level":"error","ts":1616721784.5910778,"logger":"argo-events.eventsource","caller":"awssqs/start.go:89","msg":"Error getting SQS Queue URL","eventSourceName":"aws-sqs-event-source-local","eventSourceType":"sqs","eventName":"sqs-connection","error":"RequestError: send request failed\ncaused by: Post "https://sqs.us-west-2.amazonaws.com/\": net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=AKIASMTKSX7MFVOYAM6E\n/20210326/us-west-2/sqs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=a205d08be251cedc482f2225ce293f7d1583c2100d3b5faa1fe5f526991605d6" for key Authorization","stacktrace":"github.com/argoproj/argo-events/eventsources/sources/awssqs.(*EventListener).StartListening\n\t/home/runner/work/argo-events/argo-events/eventsources/sources/awssqs/start.go:89\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start.func3.1\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:345\ngithub.com/argoproj/argo-events/common.Connect.func1\n\t/home/runner/work/argo-events/argo-events/common/retry.go:98\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:211\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:399\ngithub.com/argoproj/argo-events/common.Connect\n\t/home/runner/work/argo-events/argo-events/common/retry.go:97\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start.func3\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:344"}
{"level":"error","ts":1616721784.5912366,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:375","msg":"failed to start listening eventsource","eventSourceName":"aws-sqs-event-source-local","eventSourceType":"sqs","eventName":"sqs-connection","error":"timed out waiting for the condition","stacktrace":"github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start.func3\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:375"}
{"level":"error","ts":1616721784.5913296,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:398","msg":"Erroring out, no active event server running","eventSourceName":"aws-sqs-event-source-local","stacktrace":"github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:398\nmain.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:62\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/proc.go:203"}
{"level":"info","ts":1616721784.591635,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:296","msg":"exiting eventbus connection daemon...","eventSourceName":"aws-sqs-event-source-local"}
{"level":"error","ts":1616721784.5956926,"logger":"argo-events.eventsource","caller":"driver/nats.go:84","msg":"NATS connection lost","eventSourceName":"aws-sqs-event-source-local","clientID":"client-aws-sqs-event-source-local-eventsource-bxjff-69856975d5-x5rjc-39","stacktrace":"github.com/argoproj/argo-events/eventbus/driver.(*natsStreaming).Connect.func1\n\t/home/runner/work/argo-events/argo-events/eventbus/driver/nats.go:84\ngithub.com/nats-io/nats%2ego.(*Conn).close.func1\n\t/home/runner/go/pkg/mod/github.com/nats-io/[email protected]/nats.go:3722\ngithub.com/nats-io/nats%2ego.(*asyncCallbacksHandler).asyncCBDispatcher\n\t/home/runner/go/pkg/mod/github.com/nats-io/[email protected]/nats.go:2008"}
{"level":"fatal","ts":1616721784.595863,"logger":"argo-events.eventsource","caller":"cmd/main.go:63","msg":"failed to start eventsource server","eventSourceName":"aws-sqs-event-source-local","error":"no active event server running","errorVerbose":"no active event server running\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:401\nmain.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:62\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/proc.go:203\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/asm_amd64.s:1373","stacktrace":"main.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:63\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/proc.go:203"}

whynowy · 2021-03-26T21:14:51Z

@phelinor - From the error log, I can tell the problem is caused by \n in the Authorization header, which is generated by the credentials from the Secrets. Do you have extra \n in your credentials (before it was stored in the secret)?

net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=AKIASMTKSX7MFVOYAM6E\n/20210326/us-west-2/sqs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=a205d08be251cedc482f2225ce293f7d1583c2100d3b5faa1fe5f526991605d6" for key Authorization"

phelinor · 2021-03-26T21:23:40Z

@whynowy In fact "AKIASMTKSX7MFVOYAM6E" is the Access Key that I'm using to access AWS, after decoding the base64 text I can see that it does not contain any "\n" in it

Same for the Secret Key

I don't know what "\n/20210326" stands for.

whynowy · 2021-03-26T21:32:23Z

I recommend you to create secrets with command like following instead of doing encoding by yourself.

kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub

whynowy · 2021-03-30T18:18:39Z

@phelinor - did you get any luck? If not, it might be caused by the bug of #1146.

phelinor · 2021-04-07T15:22:17Z

@whynowy, @tomaszkrys Just to mention that with version v1.3.0-rc2 it worked fine in my local cluster. thanks for all the help.

apiVersion: v1
kind: Namespace
metadata:
  name: my-test
---
apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: my-test
type: Opaque
data:
  accesskey: accesskey
  secretkey: secretkey
---
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: aws-sqs-event-source-local
  namespace: my-test
spec:
  sqs:
    sqs-connection:
      region: us-west-2
      queue: phelinor.fifo
      waitTimeSeconds: 1
      jsonBody: true
      accessKey:
        key: accesskey
        name: aws-secret
      secretKey:
        key: secretkey
        name: aws-secret

Argo and Argo Events in cluster wide installations

rajendragosavi · 2021-08-04T12:03:17Z

Thanks @phelinor . Even I was facing the same issue.

ontor79 · 2021-12-02T12:39:29Z

I am completely new to this.Can anyone help me setup sqs for argo events from scratch please @phelinor

tomaszkrys added the bug Something isn't working label Feb 17, 2021

whynowy self-assigned this Mar 18, 2021

whynowy mentioned this issue Mar 30, 2021

fix: Trim newline of the content read from Secrets or Configmaps #1146

Merged

1 task

whynowy closed this as completed in #1146 Mar 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem with setting SQS queue #1068

Problem with setting SQS queue #1068

tomaszkrys commented Feb 17, 2021

whynowy commented Feb 18, 2021

tomaszkrys commented Feb 18, 2021 •

edited

Loading

whynowy commented Feb 18, 2021

tomaszkrys commented Feb 18, 2021 •

edited

Loading

santoshsb commented Mar 23, 2021

santoshsb commented Mar 24, 2021

whynowy commented Mar 24, 2021

phelinor commented Mar 25, 2021 •

edited

Loading

santoshsb commented Mar 25, 2021

phelinor commented Mar 25, 2021

santoshsb commented Mar 25, 2021 •

edited

Loading

phelinor commented Mar 25, 2021

phelinor commented Mar 25, 2021

whynowy commented Mar 25, 2021

phelinor commented Mar 25, 2021

phelinor commented Mar 26, 2021

whynowy commented Mar 26, 2021

phelinor commented Mar 26, 2021 •

edited

Loading

whynowy commented Mar 26, 2021

whynowy commented Mar 30, 2021

phelinor commented Apr 7, 2021

rajendragosavi commented Aug 4, 2021

ontor79 commented Dec 2, 2021 •

edited

Loading

Problem with setting SQS queue #1068

Problem with setting SQS queue #1068

Comments

tomaszkrys commented Feb 17, 2021

whynowy commented Feb 18, 2021

tomaszkrys commented Feb 18, 2021 • edited Loading

whynowy commented Feb 18, 2021

tomaszkrys commented Feb 18, 2021 • edited Loading

santoshsb commented Mar 23, 2021

santoshsb commented Mar 24, 2021

whynowy commented Mar 24, 2021

phelinor commented Mar 25, 2021 • edited Loading

santoshsb commented Mar 25, 2021

phelinor commented Mar 25, 2021

santoshsb commented Mar 25, 2021 • edited Loading

phelinor commented Mar 25, 2021

phelinor commented Mar 25, 2021

whynowy commented Mar 25, 2021

phelinor commented Mar 25, 2021

phelinor commented Mar 26, 2021

whynowy commented Mar 26, 2021

phelinor commented Mar 26, 2021 • edited Loading

whynowy commented Mar 26, 2021

whynowy commented Mar 30, 2021

phelinor commented Apr 7, 2021

rajendragosavi commented Aug 4, 2021

ontor79 commented Dec 2, 2021 • edited Loading

tomaszkrys commented Feb 18, 2021 •

edited

Loading

tomaszkrys commented Feb 18, 2021 •

edited

Loading

phelinor commented Mar 25, 2021 •

edited

Loading

santoshsb commented Mar 25, 2021 •

edited

Loading

phelinor commented Mar 26, 2021 •

edited

Loading

ontor79 commented Dec 2, 2021 •

edited

Loading