chore(plugins-view): plugin list was not displayed for DevWorkspace instances #1225

vitaliy-guliy · 2021-10-05T09:44:52Z

Signed-off-by: Vitaliy Gulyy [email protected]

What does this PR do?

Implements K8sWorkspaceServiceImpl.getWorkspaceSettings() method to be able Che-Theia get plugin registry internal/external URLs.

Screenshot/screencast of this PR

What issues does this PR fix or reference?

eclipse-che/che#20448

How to test this PR?

Devfile to create a workspace

schemaVersion: 2.1.0
attributes:
  che-editor.yaml: |
    schemaVersion: 2.1.0
    metadata:
      name: theia-ide
    commands:
      - id: init-container-command
        apply:
          component: remote-runtime-injector
    events:
      preStart:
        - init-container-command
    components:
      - name: theia-ide
        container:
          image: 'quay.io/crw_pr/che-theia:1225'
          env:
            - name: THEIA_PLUGINS
              value: 'local-dir:///plugins'
            - name: HOSTED_PLUGIN_HOSTNAME
              value: 0.0.0.0
            - name: HOSTED_PLUGIN_PORT
              value: '3130'
            - name: THEIA_HOST
              value: 0.0.0.0
          volumeMounts:
            - name: plugins
              path: /plugins
            - name: theia-local
              path: /home/theia/.theia
          mountSources: true
          memoryLimit: 2500M
          cpuLimit: 1500m
          cpuRequest: 100m
          endpoints:
            - name: theia
              attributes:
                type: main
                cookiesAuthEnabled: true
                discoverable: false
                urlRewriteSupported: true
              targetPort: 3100
              exposure: public
              secure: false
              protocol: https
            - name: webviews
              attributes:
                type: webview
                cookiesAuthEnabled: true
                discoverable: false
                unique: true
                urlRewriteSupported: true
              targetPort: 3100
              exposure: public
              secure: false
              protocol: https
            - name: mini-browser
              attributes:
                type: mini-browser
                cookiesAuthEnabled: true
                discoverable: false
                unique: true
                urlRewriteSupported: true
              targetPort: 3100
              exposure: public
              secure: false
              protocol: https
            - name: theia-dev
              attributes:
                type: ide-dev
                discoverable: false
                urlRewriteSupported: true
              targetPort: 3130
              exposure: public
              protocol: http
            - name: theia-redirect-1
              attributes:
                discoverable: false
                urlRewriteSupported: true
              targetPort: 13131
              exposure: public
              protocol: http
            - name: theia-redirect-2
              attributes:
                discoverable: false
                urlRewriteSupported: true
              targetPort: 13132
              exposure: public
              protocol: http
            - name: theia-redirect-3
              attributes:
                discoverable: false
                urlRewriteSupported: true
              targetPort: 13133
              exposure: public
              protocol: http
            - name: terminal
              attributes:
                type: collocated-terminal
                discoverable: false
                cookiesAuthEnabled: true
                urlRewriteSupported: true
              targetPort: 3333
              exposure: public
              secure: false
              protocol: wss
        attributes:
          app.kubernetes.io/component: che-theia
          app.kubernetes.io/part-of: che-theia.eclipse.org
      - name: plugins
        volume: {}
      - name: theia-local
        volume: {}
      - name: che-machine-exec
        container:
          image: 'quay.io/eclipse/che-machine-exec:next'
          command:
            - /go/bin/che-machine-exec
            - '--url'
            - '0.0.0.0:3333'
          memoryLimit: 128Mi
          memoryRequest: 32Mi
          cpuLimit: 500m
          cpuRequest: 30m
        attributes:
          app.kubernetes.io/component: machine-exec
          app.kubernetes.io/part-of: che-theia.eclipse.org
      - name: remote-runtime-injector
        container:
          image: 'quay.io/eclipse/che-theia-endpoint-runtime-binary:next'
          env:
            - name: PLUGIN_REMOTE_ENDPOINT_EXECUTABLE
              value: /remote-endpoint/plugin-remote-endpoint
            - name: REMOTE_ENDPOINT_VOLUME_NAME
              value: remote-endpoint
          volumeMounts:
            - name: remote-endpoint
              path: /remote-endpoint
          memoryLimit: 128Mi
          memoryRequest: 32Mi
          cpuLimit: 500m
          cpuRequest: 30m
        attributes:
          app.kubernetes.io/component: remote-runtime-injector
          app.kubernetes.io/part-of: che-theia.eclipse.org
      - name: remote-endpoint
        volume:
          ephemeral: true
metadata:
  name: nodejs-web-app-rbuz
  namespace: admin-che
projects:
  - git:
      checkoutFrom:
        revision: devfilev2
      remotes:
        origin: 'https://github.com/che-samples/web-nodejs-sample.git'
    name: web-nodejs-sample
components:
  - attributes:
      app.kubernetes.io/name: nodejs
      che-theia.eclipse.org/vscode-extensions:
        - 'https://open-vsx.org/api/vscode/typescript-language-features/1.49.3/file/vscode.typescript-language-features-1.49.3.vsix'
        - 'https://open-vsx.org/api/ms-vscode/js-debug/1.52.2/file/ms-vscode.js-debug-1.52.2.vsix'
    container:
      args:
        - sh
        - '-c'
        - '${PLUGIN_REMOTE_ENDPOINT_EXECUTABLE}'
      endpoints:
        - exposure: public
          name: nodejs
          protocol: http
          targetPort: 3000
      env:
        - name: PLUGIN_REMOTE_ENDPOINT_EXECUTABLE
          value: /remote-endpoint/plugin-remote-endpoint
        - name: THEIA_PLUGINS
          value: 'local-dir:///plugins/sidecars/nodejs'
      image: 'quay.io/devfile/universal-developer-image:ubi8-b452131'
      memoryLimit: 1G
      mountSources: true
      sourceMapping: /projects
      volumeMounts:
        - name: remote-endpoint
          path: /remote-endpoint
        - name: plugins
          path: /plugins
    name: nodejs
commands:
  - exec:
      commandLine: npm install
      component: nodejs
      group:
        isDefault: true
        kind: build
      label: Download dependencies
      workingDir: '${PROJECTS_ROOT}/web-nodejs-sample/app'
    id: dependencies
  - exec:
      commandLine: nodemon app.js
      component: nodejs
      group:
        kind: run
      label: Run the web app
      workingDir: '${PROJECTS_ROOT}/web-nodejs-sample/app'
    id: runapp
  - exec:
      commandLine: nodemon --inspect app.js
      component: nodejs
      group:
        isDefault: true
        kind: debug
      label: Run the web app (debugging enabled)
      workingDir: '${PROJECTS_ROOT}/web-nodejs-sample/app'
    id: debug
  - exec:
      commandLine: 'node_server_pids=$(pgrep -fx ''.*nodemon (--inspect )?app.js'' | tr "\\n" " ") && echo "Stopping node server with PIDs: ${node_server_pids}" &&  kill -15 ${node_server_pids} &>/dev/null && echo ''Done.'''
      component: nodejs
      group:
        kind: run
      label: Stop the web app
    id: stopapp

To create a workspace use this public Gist https://gist.githubusercontent.com/vitaliy-guliy/c3eadb47d3c241d9a6632029e580bf91/raw/802bab8a45fd388c3e91a26cc98a1f083adaf2dc

PR Checklist

As the author of this Pull Request I made sure that:

Reviewers

Reviewers, please comment how you tested the PR when approving it.

Happy Path Channel

HAPPY_PATH_CHANNEL=stable

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

che-bot · 2021-10-05T10:35:11Z

✅ E2E Happy path tests succeed 🎉

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:1225
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:1225

Test product:

tested with Eclipse Che on K8S (minikube v1.22.0)
E2E test scenario
E2E test pipeline source code

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

benoitf · 2021-10-05T12:31:32Z

extensions/eclipse-che-theia-remote-impl-k8s/src/node/k8s-workspace-service-impl.ts

+    console.log('workspaceService.getWorkspaceSettings() is implemented partially');
+
+    return {
+      CHE_PLUGIN_REGISTRY_URL: this.env.getPluginRegistryURL(),


note that these variables are available in Che/DevWorkspaceOperator mode but not in pure DevWorkspace Operator mode so please check that there is no error on pure DevWorkspace

then, how can we get links to plugin registry in pure DevWorkspace Operator mode?

it requires ad-hoc feature but usually everything is provided by config map

config map does not contain internal UR

Pure DevWorkspace Operator mode means, that there is no plugin registry, right?
Then how Che-Theia should work in that case? Is it OK to display the same notification ( Your registry is invalid ) as we have now?

we could display the notification or use online instance of the registry at github.io

Signed-off-by: Vitaliy Gulyy <[email protected]>

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

codecov · 2021-10-11T16:07:11Z

Codecov Report

Merging #1225 (d5e5622) into main (c299f59) will decrease coverage by 0.46%.
The diff coverage is 22.34%.

@@            Coverage Diff             @@
##             main    #1225      +/-   ##
==========================================
- Coverage   32.78%   32.31%   -0.47%     
==========================================
  Files         290      296       +6     
  Lines        9885     9849      -36     
  Branches     1457     1322     -135     
==========================================
- Hits         3241     3183      -58     
- Misses       6641     6662      +21     
- Partials        3        4       +1

Impacted Files	Coverage Δ
...theia-about/src/browser/about-che-theia-dialog.tsx	`0.00% <0.00%> (ø)`
...credentials/src/browser/che-credentials-service.ts	`0.00% <0.00%> (ø)`
...entials/src/browser/credentials-frontend-module.ts	`0.00% <0.00%> (ø)`
...eia-credentials/src/common/credentials-protocol.ts	`0.00% <0.00%> (ø)`
...eia-credentials/src/node/che-credentials-server.ts	`0.00% <0.00%> (ø)`
...s/src/node/che-theia-credentials-backend-module.ts	`0.00% <0.00%> (ø)`
...rowser/src/browser/che-mini-browser-environment.ts	`0.00% <0.00%> (ø)`
...ia-plugin-remote/src/node/hosted-plugin-service.ts	`0.00% <0.00%> (ø)`
...in-remote/src/node/plugin-remote-backend-module.ts	`0.00% <0.00%> (ø)`
...theia-plugin-remote/src/node/plugin-remote-init.ts	`0.00% <0.00%> (ø)`
... and 208 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2e6988...d5e5622. Read the comment docs.

che-bot · 2021-10-11T17:30:00Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:1225
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:1225

Test product:

tested with Eclipse Che on K8S (minikube v1.22.0)
E2E test scenario
E2E test pipeline source code

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

Signed-off-by: Vitaliy Gulyy <[email protected]>

che-bot · 2021-10-12T11:04:14Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:1225
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:1225

Test product:

tested with Eclipse Che on K8S (minikube v1.22.0)
E2E test scenario
E2E test pipeline source code

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

vitaliy-guliy · 2021-10-12T14:17:44Z

The plugin list is displayed, but it's not possible to install any plugin due to different format and architecture between devfile V1 and devfile V2.

che-bot · 2021-10-12T14:28:26Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:1225
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:1225

Test product:

tested with Eclipse Che on K8S (minikube v1.22.0)
E2E test scenario
E2E test pipeline source code

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

extensions/eclipse-che-theia-remote-impl-k8s/src/node/k8s-devworkspace-env-variables.ts

azatsarynnyy · 2021-10-13T10:14:21Z

extensions/eclipse-che-theia-remote-impl-k8s/src/node/k8s-workspace-service-impl.ts

+    if (publicURI && privateURI) {
+      return {
+        CHE_PLUGIN_REGISTRY_URL: publicURI,
+        CHE_PLUGIN_REGISTRY_INTERNAL_URL: privateURI,


If we call these keys without CHE_ prefix, as PLUGIN_REGISTRY_URL and PLUGIN_REGISTRY_INTERNAL_URL, then we don't need any changes in che-plugin-service.ts. Isn't it?

PLUGIN_REGISTRY_URL in che-plugin-service.ts means a bit different

const PLUGIN_REGISTRY_URL = 'cheWorkspacePluginRegistryUrl';

I would use CHE_PLUGIN_REGISTRY_URL for both cases and remove cheWorkspacePluginRegistryUrl, but the value is given from che-server.

PLUGIN_REGISTRY_URL in che-plugin-service.ts means a bit different

Actually, it means the same - URL of the plugin registry. It doesn't matter how this registry is implemented and where it's located, etc.

workspaceService.getWorkspaceSettings() allows to abstract from where it's stored: either Che-Server WS settings or DevWorkspace env. variables/config map.

So, if we get rid of the CHE_ prefixes, we can make it more transparent in che-plugin-service.ts.

PLUGIN_REGISTRY_URL it's only name of the variable, but yes, it means the same - URI to the plugin registry.

const PLUGIN_REGISTRY_URL = 'cheWorkspacePluginRegistryUrl';

But, if you prefer to have less changes, which is reasonable, we can return the object with allready-known keys

return { cheWorkspacePluginRegistryUrl: publicURI, cheWorkspacePluginRegistryUrl: privateURI, };

looks like the returned object has the same key but with different values ?

Changes in che-plugin-service.ts were reverted, k8s-workspace-service-impl.ts returns

return { cheWorkspacePluginRegistryUrl: publicURI, cheWorkspacePluginRegistryUrl: privateURI, };

looks like the returned object has the same key but with different values ?

I would say different keys with the same meaning.

azatsarynnyy · 2021-10-13T10:24:12Z

extensions/eclipse-che-theia-remote-impl-k8s/src/node/k8s-workspace-service-impl.ts

-    console.log('workspaceService.getWorkspaceSettings() not supported');
-    return {};
+    const publicURI = this.env.getPluginRegistryURL();
+    const privateURI = this.env.getPluginRegistryInternalURL() || publicURI;


Is there a real reason for using a public URI as a private one if it is not set?
Considering we're already checking that in

che-theia/extensions/eclipse-che-theia-plugin-ext/src/node/che-plugin-service.ts

Line 103 in f677d11

const uri = workspaceSettings[PLUGIN_REGISTRY_INTERNAL_URL] || publicUri;

We use private URI for cross-container communication, to allow che-theia get the plugin list from apache server, that is running in che-plugin-registry.
When internal URI is not defined, we use public URI. That for cases when user defined his plugin registry outside Che and added it to che-theia, using only one URI.

Thanks for the explanation!
Do we need to check it twice? In both k8s-workspace-service-impl.ts and che-plugin-service.ts.

Yes, we still need both of them.
Private URI is used to get the plugin list on the backend, public URI is used by frontend to load plugin icons.

I'm not asking about both URIs.
I'm asking about both places of using it:

che-theia/extensions/eclipse-che-theia-remote-impl-k8s/src/node/k8s-workspace-service-impl.ts

Line 75 in 4f6995d

const privateURI = this.env.getPluginRegistryInternalURL() || publicURI;

che-theia/extensions/eclipse-che-theia-plugin-ext/src/node/che-plugin-service.ts

Line 103 in f677d11

const uri = workspaceSettings[PLUGIN_REGISTRY_INTERNAL_URL] || publicUri;

In other words, if I want to get and see the "raw" settings returned by workspaceService.getWorkspaceSettings(), I should be able to get it as it is. Without any substitutions inside, like in 1.
And only when I get the "raw" settings, I can do with them whatever I want - like making such checks as in 2.

Private URI is used to get the plugin list on the backend, public URI is used by frontend to load plugin icons

workspaceService.getWorkspaceSettings() knows nothing about that ^. And it must not know that.
It's only ChePluginService who has this knowledge about plug-ins, icons and so on.

Let's imagine, in some other place in the code, I call workspaceService.getWorkspaceSettings()
in order to see if privateURI is absent. Despite the presence of publicURI.
How I can know that? There's no chance.
That's why I'm proposing to return the WorkspaceSettings as it is. And keep such checking only in the place where it's actually important - like in ChePluginService.

Let's imagine, in some other place in the code, I call workspaceService.getWorkspaceSettings() in order to see if privateURI is absent. Despite the presence of publicURI. How I can know that? There's no chance. That's why I'm proposing to return the WorkspaceSettings as it is. And keep such checking only in the place where it's actually important - like in ChePluginService.

Got it, reworking.

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

che-bot · 2021-10-13T16:08:03Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:1225
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:1225

Test product:

tested with Eclipse Che on K8S (minikube v1.22.0)
E2E test scenario
E2E test pipeline source code

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

Signed-off-by: Vitaliy Gulyy <[email protected]>

che-bot · 2021-10-15T10:56:59Z

✅ E2E Happy path tests succeed 🎉

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:1225
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:1225

Test product:

tested with Eclipse Che on K8S (minikube v1.22.0)
E2E test scenario
E2E test pipeline source code

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

chore(plugins-view): plugin list was not displayed for DevWorkspace i…

7f2b363

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

benoitf reviewed Oct 5, 2021

View reviewed changes

vitaliy-guliy added 2 commits October 11, 2021 18:16

Merge branch 'main' into che-20448

7461ee9

Signed-off-by: Vitaliy Gulyy <[email protected]>

chore(plugins-view): plugin list was not displayed for DevWorkspace i…

b9fb28d

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

Merge branch 'main' into che-20448

058920a

Signed-off-by: Vitaliy Gulyy <[email protected]>

chore(plugins-view): plugin list was not displayed for DevWorkspace i…

4f6995d

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

vitaliy-guliy marked this pull request as ready for review October 12, 2021 15:01

vitaliy-guliy requested review from azatsarynnyy, RomanNikitenko and vzhukovs as code owners October 12, 2021 15:01

vitaliy-guliy requested a review from svor October 12, 2021 15:02

azatsarynnyy reviewed Oct 13, 2021

View reviewed changes

chore(plugins-view): plugin list was not displayed for DevWorkspace i…

30ae7c9

…nstances Signed-off-by: Vitaliy Gulyy <[email protected]>

chore(plugins-view): code refactoring

d5e5622

Signed-off-by: Vitaliy Gulyy <[email protected]>

azatsarynnyy approved these changes Oct 15, 2021

View reviewed changes

vitaliy-guliy merged commit 0008e48 into main Oct 15, 2021

vitaliy-guliy deleted the che-20448 branch October 15, 2021 12:16

che-bot added this to the 7.38 milestone Oct 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(plugins-view): plugin list was not displayed for DevWorkspace instances #1225

chore(plugins-view): plugin list was not displayed for DevWorkspace instances #1225

vitaliy-guliy commented Oct 5, 2021 •

edited

Loading

che-bot commented Oct 5, 2021

benoitf Oct 5, 2021

vitaliy-guliy Oct 5, 2021

benoitf Oct 5, 2021

vitaliy-guliy Oct 5, 2021

vitaliy-guliy Oct 5, 2021

benoitf Oct 5, 2021

codecov bot commented Oct 11, 2021 •

edited

Loading

che-bot commented Oct 11, 2021

che-bot commented Oct 12, 2021

vitaliy-guliy commented Oct 12, 2021

che-bot commented Oct 12, 2021

azatsarynnyy Oct 13, 2021

vitaliy-guliy Oct 13, 2021

azatsarynnyy Oct 13, 2021 •

edited

Loading

vitaliy-guliy Oct 13, 2021

benoitf Oct 13, 2021

vitaliy-guliy Oct 13, 2021

vitaliy-guliy Oct 13, 2021

azatsarynnyy Oct 13, 2021

vitaliy-guliy Oct 13, 2021

azatsarynnyy Oct 13, 2021

vitaliy-guliy Oct 13, 2021 •

edited

Loading

azatsarynnyy Oct 13, 2021

azatsarynnyy Oct 13, 2021

azatsarynnyy Oct 13, 2021

vitaliy-guliy Oct 15, 2021

che-bot commented Oct 13, 2021

che-bot commented Oct 15, 2021

chore(plugins-view): plugin list was not displayed for DevWorkspace instances #1225

chore(plugins-view): plugin list was not displayed for DevWorkspace instances #1225

Conversation

vitaliy-guliy commented Oct 5, 2021 • edited Loading

What does this PR do?

Screenshot/screencast of this PR

What issues does this PR fix or reference?

How to test this PR?

PR Checklist

Reviewers

Happy Path Channel

che-bot commented Oct 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 11, 2021 • edited Loading

Codecov Report

che-bot commented Oct 11, 2021

che-bot commented Oct 12, 2021

vitaliy-guliy commented Oct 12, 2021

che-bot commented Oct 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azatsarynnyy Oct 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitaliy-guliy Oct 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

che-bot commented Oct 13, 2021

che-bot commented Oct 15, 2021

vitaliy-guliy commented Oct 5, 2021 •

edited

Loading

codecov bot commented Oct 11, 2021 •

edited

Loading

azatsarynnyy Oct 13, 2021 •

edited

Loading

vitaliy-guliy Oct 13, 2021 •

edited

Loading