Skip to content
This repository has been archived by the owner on Mar 17, 2021. It is now read-only.

[Infra issue] Workspaces briefly switching to offline during test run #1848

Closed
ScrewTSW opened this issue Mar 30, 2020 · 24 comments
Closed

[Infra issue] Workspaces briefly switching to offline during test run #1848

ScrewTSW opened this issue Mar 30, 2020 · 24 comments

Comments

@ScrewTSW
Copy link
Member

Sometimes it happens that workspaces switch temporarily to offline state during workspace readiness stage of tests.
There are no evicted events present
image
image

@ScrewTSW
Copy link
Member Author

@Katka92
Copy link
Collaborator

Katka92 commented Mar 30, 2020

It also seems there is nothing suspicious in kibana logs.
Search command used: workspace8m9jip0n7hhb1jx3 AND kubernetes.conatiner_name=rhche

Screenshot from 2020-03-30 12-32-00

@Katka92 Katka92 closed this as completed Apr 20, 2020
@Katka92
Copy link
Collaborator

Katka92 commented May 18, 2020

Reopening this issue as a root of of failures for those issues:
Workspace reloaded twice during startup #1837
Maven terminal reset during build #1633

@Katka92
Copy link
Collaborator

Katka92 commented May 18, 2020

@Katka92
Copy link
Collaborator

Katka92 commented May 19, 2020

Workspace switched to offline on prod-preview cluster 2a 1x
https://ci.centos.org/job/devtools-rh-che-periodic-prod-preview-2a/2872/console

@Katka92
Copy link
Collaborator

Katka92 commented Jun 2, 2020

Seen 7 days ago:
Seen again on production cluster 2 x1:
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/2912/console

@ScrewTSW
Copy link
Member Author

image
Seen again on production cluster 2 x1:
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/3081/console
on production cluster 1a x1:
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-1a/2874/console
(workspace switched to offline briefly after starting up)

@ScrewTSW ScrewTSW reopened this Jun 23, 2020
@ScrewTSW ScrewTSW changed the title [Infra issue] Workspaces briefly switching to offline during workspace startup [Infra issue] Workspaces briefly switching to offline during test run Jun 23, 2020
@Katka92 Katka92 closed this as completed Jul 13, 2020
@Katka92
Copy link
Collaborator

Katka92 commented Jul 15, 2020

Seen again on cluster 2 - workspace switched to offline, that broke running command and LS stopped working.
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/3212/console

@Katka92 Katka92 reopened this Jul 15, 2020
@Katka92
Copy link
Collaborator

Katka92 commented Jul 28, 2020

Workspace switched to offline. After "restarting" to online again, LS was not working and command was not able to run.
image

https://ci.centos.org/job/devtools-rh-che-periodic-prod-2aProd/3065/console

@ScrewTSW
Copy link
Member Author

@Katka92
Copy link
Collaborator

Katka92 commented Aug 17, 2020

Seen on production cluster 1b 1x
https://ci.centos.org/job/devtools-rh-che-periodic-prod-1b/3136/console

@Katka92
Copy link
Collaborator

Katka92 commented Aug 20, 2020

Seen on production cluster 2 1x
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/3370/console

@Katka92
Copy link
Collaborator

Katka92 commented Aug 26, 2020

Seen on productin cluster 1b 1x
https://ci.centos.org/job/devtools-rh-che-periodic-prod-1b/3168/console

@ScrewTSW
Copy link
Member Author

ScrewTSW commented Aug 31, 2020

Seen again on production cluster 2 x1:
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-2/3413/console
resulting in maven build fail - connection is closed
mpc-hc64_bqjdRWUcJQ
on production cluster 1b x1:
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-1b/3191/console
causing the maven build console to loose output and test to fail
mpc-hc64_Sh69jyNuI7
mpc-hc64_J2s0mOeSeP

@ScrewTSW
Copy link
Member Author

ScrewTSW commented Sep 1, 2020

Seen again on production cluster 2a x1:
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-2aProd/3208/console
causing project to not be imported and test refresh

@ScrewTSW
Copy link
Member Author

Seen again on production cluster 2 x1:
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/3476/console

@Katka92 Katka92 closed this as completed Oct 2, 2020
@Katka92
Copy link
Collaborator

Katka92 commented Oct 21, 2020

@Katka92 Katka92 reopened this Oct 21, 2020
@ScrewTSW
Copy link
Member Author

Seen again on prod-preview cluster 2a x1:
https://ci.centos.org/job/devtools-rh-che-periodic-prod-preview-2a/3636/console

@ScrewTSW
Copy link
Member Author

ScrewTSW commented Nov 9, 2020

Seen again on production cluster 2a x2:
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-2aProd/3472/console
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-2aProd/3473/console
on prod-preview cluster 2a x2:
https://ci.centos.org/job/devtools-rh-che-periodic-prod-preview-2a/3685/console
https://ci.centos.org/job/devtools-rh-che-periodic-prod-preview-2a/3687/console
project was imported after refresh of workspace, but it caused the tests to skip build task, therefore LS didn't have enough time to initialize

@Katka92
Copy link
Collaborator

Katka92 commented Nov 30, 2020

Seen again on cluster 2a, LS was not working at all because of that.
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2aProd/3559/console

@Katka92
Copy link
Collaborator

Katka92 commented Feb 3, 2021

Probably seen again on cluster 2. https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/4036/console

@ibuziuk
Copy link
Member

ibuziuk commented Feb 25, 2021

Closing as outdated

@ibuziuk ibuziuk closed this as completed Feb 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants