You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Whenever there is a change that causes the jenkins main node to be replaced we see intermittent failure during instance initialisation.
Cause:
This is due to hardcoded log-stream value here.
Whenever a change causes a new jenkins main node to be brought up, the moment the new main node is brought up and docker-compose is run, the new container tries to write its logs to an existing log-stream which is already in use by the previous jenkins main node container which is yet to be brought down.
As per https://docs.docker.com/config/containers/logging/awslogs/#awslogs-stream two containers writing to the same log-stream is not a good practise and results in degraded performance. This is the reason we see intermittent failures during new instance bring up.
To reproduce
Bring up a new CI stack, make a change that will potentially result in jenkins main node replacement and redeploy the stack.
Expected behavior
No response
Screenshots
If applicable, add screenshots to help explain your problem.
Host / Environment
No response
Additional context
No response
Relevant log output
May 26 15:03:03 ip-10-0-145-33 dockerd: time="2022-05-26T15:03:03.838804152Z" level=error msg="Failed to put log events" errorCode=InvalidSequenceTokenException logGroupName=JenkinsMainNode/jenkins.log logStreamName=jenkins.log message="The given sequenceToken is invalid. The next expected sequenceToken is: 49629487109106147829027718688550912310461956053603975282" origError="<nil>"
May 26 15:03:04 ip-10-0-145-33 systemd: Unmounted /run/docker/netns/f01cf084571d.
The text was updated successfully, but these errors were encountered:
Describe the bug
Whenever there is a change that causes the jenkins main node to be replaced we see intermittent failure during instance initialisation.
Cause:
This is due to hardcoded log-stream value here.
Whenever a change causes a new jenkins main node to be brought up, the moment the new main node is brought up and docker-compose is run, the new container tries to write its logs to an existing log-stream which is already in use by the previous jenkins main node container which is yet to be brought down.
As per https://docs.docker.com/config/containers/logging/awslogs/#awslogs-stream two containers writing to the same log-stream is not a good practise and results in degraded performance. This is the reason we see intermittent failures during new instance bring up.
To reproduce
Bring up a new CI stack, make a change that will potentially result in jenkins main node replacement and redeploy the stack.
Expected behavior
No response
Screenshots
If applicable, add screenshots to help explain your problem.
Host / Environment
No response
Additional context
No response
Relevant log output
The text was updated successfully, but these errors were encountered: